1
0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-24 13:23:59 +00:00

6229 Commits

Author SHA1 Message Date
Paul Tremberth
c68f99eed8 Refactor settings tests 2017-03-03 17:03:25 +01:00
Paul Tremberth
c3b6feca0e Fix setting lookup for MEDIA_ALLOWED_REDIRECTS 2017-03-03 16:29:07 +01:00
Paul Tremberth
f7e11b198e Cleanup 2017-03-03 16:00:59 +01:00
Paul Tremberth
ecde166ee1 Refactor without MEDIA_HTTPSTATUS_LIST setting 2017-03-03 15:52:05 +01:00
Paul Tremberth
72fbb687d7 Revert "expose allowed_status tuple for media pipeline"
This reverts commit 052809c73ed20b9a728a8fd7df3de5f45f2dad8d.
2017-03-03 15:52:05 +01:00
Paul Tremberth
c64ebee062 Refactor (WIP) 2017-03-03 15:52:05 +01:00
Bernardas
11b31c9fbd fix redirect change 2017-03-03 15:52:05 +01:00
Bernardas
3cef1cd451 adjust variable wording and redirect logic 2017-03-03 15:52:05 +01:00
Bernardas
f0b4077f81 expose allowed_status tuple for media pipeline 2017-03-03 15:52:05 +01:00
Bernardas
2e052c8615 fix error when settings are not provided 2017-03-03 15:52:05 +01:00
Bernardas
8542780854 typo and clarify handling 2017-03-03 15:52:05 +01:00
Bernardas
6a42214716 add tests for media pipeline MEDIA_ALLOW_REDIRECTS and MEDIA_HTTPSTATUS_LIST settings 2017-03-03 15:52:05 +01:00
Bernardas
25ed491219 add description for media pipeline MEDIA_ALLOW_REDIRECTS and MEDIA_HTTPSTATUS_LIST settings 2017-03-03 15:52:05 +01:00
Bernardas
5d0058492c add media pipeline settings to enable redirection and handling of certain http statuses 2017-03-03 15:52:05 +01:00
Mikhail Korobov
7e8453cf1e Merge pull request #2306 from redapple/referrer-policy
[MRG] Referrer policies in RefererMiddleware
2017-03-03 04:05:13 +05:00
Paul Tremberth
fad499ab60 Rename arguments (bis) 2017-03-02 23:06:04 +01:00
Paul Tremberth
db176f872b Remove Legacy Policy which is equivalent to UnsafeUrl Policy 2017-03-02 22:56:23 +01:00
Mikhail Korobov
93024c242b Merge pull request #2537 from scrapy/no-canonicalize
[MRG+1] Set canonicalize=False for LinkExtractor
2017-03-03 02:53:36 +05:00
Paul Tremberth
c2c503192f Rename arguments 2017-03-02 22:53:27 +01:00
Paul Tremberth
2d55d838ca Fix strip_url() tests 2017-03-01 20:59:52 +01:00
Paul Tremberth
efa50039ec Add tests for policy fallback on unknown policies from meta and headers 2017-03-01 17:51:23 +01:00
Paul Tremberth
6916dd6240 Warn or fail with exception on unknown policies 2017-03-01 17:51:23 +01:00
Paul Tremberth
8226e77010 Add test for Referer header on HTTP redirections 2017-03-01 17:51:23 +01:00
Paul Tremberth
d2aa51c0fb Update tests 2017-03-01 17:51:23 +01:00
Paul Tremberth
04e4d08612 Pass URLs around instead of Request/Responses 2017-03-01 17:51:23 +01:00
Paul Tremberth
bc200d1155 Rename setting to REFERRER_POLICY (with 2 Rs) 2017-03-01 17:51:23 +01:00
Paul Tremberth
537683f945 Add autoclass directives to document built-in policies 2017-03-01 17:51:23 +01:00
Paul Tremberth
3dc09eeceb Use table for referrer policy options 2017-03-01 17:51:23 +01:00
Paul Tremberth
605935f015 Edit text 2017-03-01 17:51:23 +01:00
Paul Tremberth
eb07285a63 Reword warning on no-referrer-when-downgrade policy 2017-03-01 17:51:23 +01:00
Paul Tremberth
03ff19d188 Update docs for new "referrer_policy" Request.meta key 2017-03-01 17:51:23 +01:00
Paul Tremberth
e249abc32b Update docs 2017-03-01 17:50:39 +01:00
Paul Tremberth
c86f568b9c Update docs with "strict-..." policies 2017-03-01 17:50:39 +01:00
Paul Tremberth
b6c761d2b4 Fix tests 2017-03-01 17:50:39 +01:00
Paul Tremberth
ebcacd3f54 Update StrictOriginPolicy 2017-03-01 17:50:39 +01:00
Paul Tremberth
deb8567116 Update NoReferrerWhenDowngradePolicy 2017-03-01 17:50:39 +01:00
Paul Tremberth
77aec5a796 Fix implementation 2017-03-01 17:50:39 +01:00
Paul Tremberth
5cef67ae75 Update Referrer tests for "strict-" policies 2017-03-01 17:50:39 +01:00
Paul Tremberth
c808a97c74 Add new "strict-" policies 2017-03-01 17:50:39 +01:00
Paul Tremberth
0a0b60a59f Add tests for stripping userinfo with percent-encoded delimiters 2017-03-01 17:50:39 +01:00
Paul Tremberth
8864d0e8c1 Rename helper function to strip_url() + add more tests 2017-03-01 17:50:39 +01:00
Paul Tremberth
5dd7311cd4 Move URL credentials stripping to a helper function 2017-03-01 17:50:39 +01:00
Paul Tremberth
c9c59db489 Update documentation about REFERER_POLICY setting 2017-03-01 17:50:39 +01:00
Paul Tremberth
285d5bc03a Patch "Referer" header on HTTP redirects if necessary 2017-03-01 17:50:39 +01:00
Paul Tremberth
d3d4d66ce8 Add tests for referrer-policy set in response HTTP headers 2017-03-01 17:50:39 +01:00
Paul Tremberth
e50e670eff Add test for custom referrer policy via settings 2017-03-01 17:50:39 +01:00
Paul Tremberth
ec8b4c1a9b Change __init__ default "settings" arg handling 2017-03-01 17:50:39 +01:00
Paul Tremberth
0344f57fef Support case-insensitive policy names in settings 2017-03-01 17:50:39 +01:00
Paul Tremberth
e72b6e3361 Add tests for referrer policy via settings and via Request meta 2017-03-01 17:50:39 +01:00
Paul Tremberth
842ce131aa Make default referrer policy customizable via settings 2017-03-01 17:50:39 +01:00