Paul Tremberth
|
7be773e14a
|
Add SPIDER_LOADER_WARN_ONLY to toggle between spiderloader failure and warning
|
2017-03-07 17:40:40 +01:00 |
|
Mikhail Korobov
|
d3f8f3d38a
|
Merge pull request #2612 from redapple/dupe-spider-name-tests
[WIP] Add warning on duplicate spider name
|
2017-03-07 20:08:42 +05:00 |
|
Paul Tremberth
|
0b9a18e1a1
|
Warn only once for all spiders
|
2017-03-07 15:41:17 +01:00 |
|
Paul Tremberth
|
978306a223
|
Fix dupe spider name warning string tests
|
2017-03-07 14:48:16 +01:00 |
|
Paul Tremberth
|
a017f7b932
|
Warn about modules where duplicate spider names were found
|
2017-03-07 14:44:27 +01:00 |
|
Mikhail Korobov
|
802ed30e8e
|
Merge pull request #2391 from redapple/always-gunzip-content-enc-gzip
[MRG] Always decompress Content-Encoding: gzip at HttpCompression stage
|
2017-03-07 16:56:54 +05:00 |
|
Paul Tremberth
|
b6378c7ef6
|
Revert to using self.assert methods
|
2017-03-07 12:28:24 +01:00 |
|
Paul Tremberth
|
4caceccd59
|
Use 3-bytes for gzip archive type sniffing
|
2017-03-07 11:02:46 +01:00 |
|
Paul Tremberth
|
b174744b80
|
Do not silently fail on gzip unzipping
|
2017-03-07 11:02:46 +01:00 |
|
Paul Tremberth
|
11cdf58abe
|
Always decompress Content-Encoding: gzip at HttpCompression stage
Let SitemapSpider handle decoding of .xml.gz files if necessary
|
2017-03-07 11:02:46 +01:00 |
|
Mikhail Korobov
|
6320b743cf
|
Merge pull request #2393 from redapple/response-replace-encoding
[MRG] Use body to choose response type after decompression content
|
2017-03-07 13:23:35 +05:00 |
|
Paul Tremberth
|
e42b846a9f
|
Use body to chose response type after decompression content
|
2017-03-06 23:10:38 +01:00 |
|
Mikhail Korobov
|
451f147468
|
Merge pull request #2628 from redapple/brotli-docs
DOC Mention brotli support in HttpCompressionMiddleware section
|
2017-03-06 22:16:21 +05:00 |
|
Mikhail Korobov
|
20c4f9f43a
|
Merge pull request #2627 from redapple/referrer-empty-string-header
Fix referrer policy from response headers and support explicit empty string
|
2017-03-06 21:44:08 +05:00 |
|
Paul Tremberth
|
2a7d391e0b
|
DOC Mention brotli support in HttpCompressionMiddleware section
|
2017-03-06 17:30:32 +01:00 |
|
Paul Tremberth
|
768f3155e5
|
Fix referrer policy from response headers and support explicit empty string
|
2017-03-06 16:20:37 +01:00 |
|
Paul Tremberth
|
d7b26edf6b
|
Merge pull request #2625 from redapple/release-notes-packaging-fix
Update release notes for 1.0.7, 1.1.4 and 1.2.3
|
2017-03-06 14:47:01 +01:00 |
|
Paul Tremberth
|
a8b47d6c68
|
Update release notes for 1.0.7, 1.1.4 and 1.2.3
|
2017-03-06 14:25:52 +01:00 |
|
Paul Tremberth
|
0a17f9b55c
|
Merge pull request #2334 from ArturGaspar/data_uri
[MRG+1] data URI download handler.
|
2017-03-06 13:50:46 +01:00 |
|
Mikhail Korobov
|
2c5fa0c130
|
Merge pull request #2617 from redapple/engine-dupe-statement
Remove redundant slot.add_request() call in ExecutionEngine
|
2017-03-03 21:44:07 +05:00 |
|
Paul Tremberth
|
30d812eea2
|
Remove redundant slot.add_request() call in ExecutionEngine
|
2017-03-03 17:15:37 +01:00 |
|
Mikhail Korobov
|
7e8453cf1e
|
Merge pull request #2306 from redapple/referrer-policy
[MRG] Referrer policies in RefererMiddleware
|
2017-03-03 04:05:13 +05:00 |
|
Paul Tremberth
|
fad499ab60
|
Rename arguments (bis)
|
2017-03-02 23:06:04 +01:00 |
|
Paul Tremberth
|
db176f872b
|
Remove Legacy Policy which is equivalent to UnsafeUrl Policy
|
2017-03-02 22:56:23 +01:00 |
|
Mikhail Korobov
|
93024c242b
|
Merge pull request #2537 from scrapy/no-canonicalize
[MRG+1] Set canonicalize=False for LinkExtractor
|
2017-03-03 02:53:36 +05:00 |
|
Paul Tremberth
|
c2c503192f
|
Rename arguments
|
2017-03-02 22:53:27 +01:00 |
|
Artur Gaspar
|
b50d0370f4
|
Test response attributes in data URI download handler.
|
2017-03-02 14:46:33 -03:00 |
|
Paul Tremberth
|
12a8ddecab
|
Fix tests
|
2017-03-02 13:03:18 +01:00 |
|
Paul Tremberth
|
e71803c833
|
Add tests for duplicate spider name warnings
|
2017-03-02 12:48:47 +01:00 |
|
Paul Tremberth
|
5be5ef57f3
|
Remove extra blank line
|
2017-03-02 12:44:40 +01:00 |
|
Erick
|
6abd9ba843
|
Fix warning to duplicated spider. Issue 2181
|
2017-03-02 12:44:40 +01:00 |
|
MrMenezes
|
f3b75c940d
|
Fix warning to duplicated spider. Issue 2181
|
2017-03-02 12:44:40 +01:00 |
|
Paul Tremberth
|
2d55d838ca
|
Fix strip_url() tests
|
2017-03-01 20:59:52 +01:00 |
|
Paul Tremberth
|
efa50039ec
|
Add tests for policy fallback on unknown policies from meta and headers
|
2017-03-01 17:51:23 +01:00 |
|
Paul Tremberth
|
6916dd6240
|
Warn or fail with exception on unknown policies
|
2017-03-01 17:51:23 +01:00 |
|
Paul Tremberth
|
8226e77010
|
Add test for Referer header on HTTP redirections
|
2017-03-01 17:51:23 +01:00 |
|
Paul Tremberth
|
d2aa51c0fb
|
Update tests
|
2017-03-01 17:51:23 +01:00 |
|
Paul Tremberth
|
04e4d08612
|
Pass URLs around instead of Request/Responses
|
2017-03-01 17:51:23 +01:00 |
|
Paul Tremberth
|
bc200d1155
|
Rename setting to REFERRER_POLICY (with 2 Rs)
|
2017-03-01 17:51:23 +01:00 |
|
Paul Tremberth
|
537683f945
|
Add autoclass directives to document built-in policies
|
2017-03-01 17:51:23 +01:00 |
|
Paul Tremberth
|
3dc09eeceb
|
Use table for referrer policy options
|
2017-03-01 17:51:23 +01:00 |
|
Paul Tremberth
|
605935f015
|
Edit text
|
2017-03-01 17:51:23 +01:00 |
|
Paul Tremberth
|
eb07285a63
|
Reword warning on no-referrer-when-downgrade policy
|
2017-03-01 17:51:23 +01:00 |
|
Paul Tremberth
|
03ff19d188
|
Update docs for new "referrer_policy" Request.meta key
|
2017-03-01 17:51:23 +01:00 |
|
Paul Tremberth
|
e249abc32b
|
Update docs
|
2017-03-01 17:50:39 +01:00 |
|
Paul Tremberth
|
c86f568b9c
|
Update docs with "strict-..." policies
|
2017-03-01 17:50:39 +01:00 |
|
Paul Tremberth
|
b6c761d2b4
|
Fix tests
|
2017-03-01 17:50:39 +01:00 |
|
Paul Tremberth
|
ebcacd3f54
|
Update StrictOriginPolicy
|
2017-03-01 17:50:39 +01:00 |
|
Paul Tremberth
|
deb8567116
|
Update NoReferrerWhenDowngradePolicy
|
2017-03-01 17:50:39 +01:00 |
|
Paul Tremberth
|
77aec5a796
|
Fix implementation
|
2017-03-01 17:50:39 +01:00 |
|