BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-24 03:43:56 +00:00
Code Releases Wiki Activity
6,327 Commits 39 Branches 119 Tags
Commit Graph

6327 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Mikhail Korobov
93024c242b Merge pull request #2537 from scrapy/no-canonicalize 
[MRG+1] Set canonicalize=False for LinkExtractor
 2017-03-03 02:53:36 +05:00
Paul Tremberth
c2c503192f Rename arguments 2017-03-02 22:53:27 +01:00
Artur Gaspar
b50d0370f4 Test response attributes in data URI download handler. 2017-03-02 14:46:33 -03:00
jorenham
f96490df2c Move cache storage logging to the individual storage classes 2017-03-02 16:17:51 +01:00
Paul Tremberth
12a8ddecab Fix tests 2017-03-02 13:03:18 +01:00
Paul Tremberth
e71803c833 Add tests for duplicate spider name warnings 2017-03-02 12:48:47 +01:00
Paul Tremberth
5be5ef57f3 Remove extra blank line 2017-03-02 12:44:40 +01:00
Erick
6abd9ba843 Fix warning to duplicated spider. Issue 2181 2017-03-02 12:44:40 +01:00
MrMenezes
f3b75c940d Fix warning to duplicated spider. Issue 2181 2017-03-02 12:44:40 +01:00
jorenham
97d84d920b Logging the cache directory at HttpCacheMiddleware instantiation #2604 2017-03-02 11:04:16 +01:00
Paul Tremberth
2d55d838ca Fix strip_url() tests 2017-03-01 20:59:52 +01:00
Paul Tremberth
efa50039ec Add tests for policy fallback on unknown policies from meta and headers 2017-03-01 17:51:23 +01:00
Paul Tremberth
6916dd6240 Warn or fail with exception on unknown policies 2017-03-01 17:51:23 +01:00
Paul Tremberth
8226e77010 Add test for Referer header on HTTP redirections 2017-03-01 17:51:23 +01:00
Paul Tremberth
d2aa51c0fb Update tests 2017-03-01 17:51:23 +01:00
Paul Tremberth
04e4d08612 Pass URLs around instead of Request/Responses 2017-03-01 17:51:23 +01:00
Paul Tremberth
bc200d1155 Rename setting to REFERRER_POLICY (with 2 Rs) 2017-03-01 17:51:23 +01:00
Paul Tremberth
537683f945 Add autoclass directives to document built-in policies 2017-03-01 17:51:23 +01:00
Paul Tremberth
3dc09eeceb Use table for referrer policy options 2017-03-01 17:51:23 +01:00
Paul Tremberth
605935f015 Edit text 2017-03-01 17:51:23 +01:00
Paul Tremberth
eb07285a63 Reword warning on no-referrer-when-downgrade policy 2017-03-01 17:51:23 +01:00
Paul Tremberth
03ff19d188 Update docs for new "referrer_policy" Request.meta key 2017-03-01 17:51:23 +01:00
Paul Tremberth
e249abc32b Update docs 2017-03-01 17:50:39 +01:00
Paul Tremberth
c86f568b9c Update docs with "strict-..." policies 2017-03-01 17:50:39 +01:00
Paul Tremberth
b6c761d2b4 Fix tests 2017-03-01 17:50:39 +01:00
Paul Tremberth
ebcacd3f54 Update StrictOriginPolicy 2017-03-01 17:50:39 +01:00
Paul Tremberth
deb8567116 Update NoReferrerWhenDowngradePolicy 2017-03-01 17:50:39 +01:00
Paul Tremberth
77aec5a796 Fix implementation 2017-03-01 17:50:39 +01:00
Paul Tremberth
5cef67ae75 Update Referrer tests for "strict-" policies 2017-03-01 17:50:39 +01:00
Paul Tremberth
c808a97c74 Add new "strict-" policies 2017-03-01 17:50:39 +01:00
Paul Tremberth
0a0b60a59f Add tests for stripping userinfo with percent-encoded delimiters 2017-03-01 17:50:39 +01:00
Paul Tremberth
8864d0e8c1 Rename helper function to strip_url() + add more tests 2017-03-01 17:50:39 +01:00
Paul Tremberth
5dd7311cd4 Move URL credentials stripping to a helper function 2017-03-01 17:50:39 +01:00
Paul Tremberth
c9c59db489 Update documentation about REFERER_POLICY setting 2017-03-01 17:50:39 +01:00
Paul Tremberth
285d5bc03a Patch "Referer" header on HTTP redirects if necessary 2017-03-01 17:50:39 +01:00
Paul Tremberth
d3d4d66ce8 Add tests for referrer-policy set in response HTTP headers 2017-03-01 17:50:39 +01:00
Paul Tremberth
e50e670eff Add test for custom referrer policy via settings 2017-03-01 17:50:39 +01:00
Paul Tremberth
ec8b4c1a9b Change __init__ default "settings" arg handling 2017-03-01 17:50:39 +01:00
Paul Tremberth
0344f57fef Support case-insensitive policy names in settings 2017-03-01 17:50:39 +01:00
Paul Tremberth
e72b6e3361 Add tests for referrer policy via settings and via Request meta 2017-03-01 17:50:39 +01:00
Paul Tremberth
842ce131aa Make default referrer policy customizable via settings 2017-03-01 17:50:39 +01:00
Paul Tremberth
f6205778f3 Refactor ReferrerPolicy methods 2017-03-01 17:50:39 +01:00
Paul Tremberth
f6a800fde6 Remove all non-cached urlparsing references 2017-03-01 17:50:39 +01:00
Paul Tremberth
59cb884ace Use urlparse_cached() for OriginWhenCrossOriginPolicy 2017-03-01 17:50:39 +01:00
Paul Tremberth
f2ee6be3bb Use urlparse_cached() for OriginPolicy 2017-03-01 17:50:39 +01:00
Paul Tremberth
3af88a2877 Use urlparse_cached() on request and responses 2017-03-01 17:50:39 +01:00
Paul Tremberth
7ec1b5f6c3 Add tests for the different referrer policies 2017-03-01 17:50:38 +01:00
Paul Tremberth
baed7c436f WIP Add Referrer policies 2017-03-01 17:50:38 +01:00
Mikhail Korobov
7b49b9c0f5 Merge pull request #2590 from rolando-contrib/handle-data-loss-gracefully 
[MRG+2] Handle data loss gracefully.
 2017-03-01 20:23:19 +05:00
Rolando Espinoza
f01ae6ffcd Handle data loss gracefully. 
Websites that return a wrong ``Content-Length`` header may cause a data
loss error. Also when a chunked response is not finished properly.

This change adds a new setting ``DOWNLOAD_FAIL_ON_DATALOSS`` (default:
``True``) and request.meta key ``download_fail_on_dataloss``.
 2017-03-01 11:43:53 -03:00