BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-24 10:43:48 +00:00
Code Releases Wiki Activity
6,376 Commits 39 Branches 119 Tags
Commit Graph

696 Commits

Author SHA1 Message Date
Mikhail Korobov
160da6abab fixed tests in Python 2 2017-02-15 04:41:53 +05:00
Mikhail Korobov
2674f317df Response.follow 2017-02-15 04:39:47 +05:00
Mikhail Korobov
608c3f0c45 handle whitespace in response.follow; add tests 2017-02-15 04:17:41 +05:00
Mikhail Korobov
e285b1d6c2 retry stats 2017-02-14 20:27:49 +05:00
Mikhail Korobov
afac3fd2c2 Merge pull request #2530 from elacuesta/proxy_credentials 
[MRG+1] Use credentials from request.meta['proxy']
 2017-02-14 20:58:39 +06:00
Eugenio Lacuesta
ae0ea31abd Add HTTPPROXY_ENABLED setting (default True) 2017-02-14 11:33:01 -03:00
Paul Tremberth
f32a229e3d Merge pull request #2510 from elacuesta/reqser_request_class 
[MRG+1] Preserve request class when converting to/from dicts (utils.reqser)
 2017-02-08 18:30:07 +01:00
Eugenio Lacuesta
9c0aae724e Use credentials from request.meta['proxy'] if present 2017-02-08 13:22:30 -03:00
Paul Tremberth
a8893190a1 Merge pull request #2464 from elacuesta/component_order_integer 
[MRG+2] Validate numeric values (components order)
 2017-02-08 13:01:02 +01:00
Paul Tremberth
3becb861af Merge pull request #2544 from scrapy/twisted-version-cleanup 
[MRG+1] Remove code required to support ancient twisted versions.
 2017-02-08 12:59:45 +01:00
Rolando Espinoza
4e765acaed BUG: Fix __classcell__ propagation. 
Python 3.6 added simpler customization of class creation but this
requires to propagate correctly the __classcell__ attribute in custom
__new__ methods.

See https://docs.python.org/3.6/whatsnew/3.6.html#pep-487-simpler-
customization-of-class-creation
 2017-02-08 02:12:52 -04:00
Mikhail Korobov
04b2f79e7a Remove code required to support ancient twisted versions. See GH-1887. 2017-02-07 22:30:58 +05:00
Eugenio Lacuesta
24e82bfe75 Validate values for components order 2017-02-07 09:13:57 -03:00
Pawel Miech
fb4ef21a1d [httpcompression] minor style edits 2017-02-07 10:22:42 +01:00
Pawel Miech
af802bad14 [httpcompression] add brotlipy for python 3 2017-02-06 15:45:21 +01:00
Pawel Miech
3daf473686 [httpcompression] skip test if no brotli 2017-02-06 13:49:41 +01:00
Pawel Miech
f8f8bbe080 [httpcompression] import brotli when available 2017-02-06 12:19:00 +01:00
Pawel Miech
16c4b4e184 [httpcompression] add support for br - brotli content encoding 2017-02-06 11:41:08 +01:00
Mikhail Korobov
814ce37dff Merge pull request #2528 from redapple/dns-retry-test 
Make DNS retry test compatible with Twisted 17+
 2017-02-03 02:43:46 +05:00
Paul Tremberth
02e1d2b1fd Add trailing dot 2017-02-02 22:28:37 +01:00
Paul Tremberth
e604c0f3ab Remove unused imports 2017-02-02 18:26:41 +01:00
Paul Tremberth
3358254c5c Make DNS retry test compatible with Twisted 17+ 2017-02-02 17:53:28 +01:00
Paul Tremberth
803d8c4b57 Add tests for passing kwargs on response .xpath() shortcut 2017-02-02 17:44:57 +01:00
Paul Tremberth
ae6d8d728e Support 'True' and 'False' strings as boolean settings values 2017-01-30 16:33:08 +01:00
Eugenio Lacuesta
53757e51e5 Preserve request class when converting to/from dicts (utils.reqser) 2017-01-24 17:13:21 -03:00
Paul Tremberth
07f9985a94 TST: Randomize FILES_EXPIRES above 90 days 2016-12-21 17:03:11 +01:00
Elias Dorneles
d09ec3db68 Merge pull request #2410 from redapple/fetch-transparent-redirect 
[MRG+1] Transparently handle redirections in fetch and shell
 2016-12-21 09:49:15 -02:00
Mikhail Korobov
d19c4c1f80 Merge pull request #2433 from redapple/wrong-spidermodules-warning 
[MRG+1] Warn user instead of failing for wrong SPIDER_MODULES setting
 2016-12-19 21:46:56 +05:00
Paul Tremberth
ed1e4d8df3 Merge pull request #1731 from scrapy/disable-toplevel-2 
[MRG+1] LOG_SHORT_NAMES option to disable TopLevelFormatter
 2016-12-19 16:02:55 +01:00
Elias Dorneles
97e82107b1 Merge pull request #2270 from redapple/httperror-log-info 
[MRG+1] Raise log level for HttpErrorMiddleware to INFO (from DEBUG)
 2016-12-14 12:18:04 -02:00
Paul Tremberth
f7e4081414 Add tests for SequenceExclude container 2016-12-12 22:37:53 +01:00
Mikhail Korobov
05b4555f39 TST tests for LOG_SHORT_NAMES 2016-12-09 02:19:51 +05:00
Mikhail Korobov
e46572d6f2 TST end-to-end test for LOG_LEVEL option 
there were no end-to-end tests for this option
 2016-12-09 02:19:33 +05:00
Mikhail Korobov
6eab59cbac TST cleanup runspider tests 2016-12-09 02:14:12 +05:00
Paul Tremberth
948e3cd003 Warn user instead of failing for wrong SPIDER_MODULES setting 2016-12-08 12:50:26 +01:00
Paul Tremberth
2cd579a774 Add test for fetch(url) within shell with and without redirect 2016-12-07 19:07:32 +01:00
Paul Tremberth
7e54de2455 Add tests for shell command with and without --no-redirect 2016-12-07 18:41:24 +01:00
Elias Dorneles
cce631abec Merge pull request #1887 from nyov/twisted11 
[MRG+2] Bump Twisted dependency to 13.1.0
 2016-12-07 15:00:01 -02:00
Paul Tremberth
778bed07bf Let framework handle only HTTP redirects by default for fetch and shell commands 2016-12-07 17:56:13 +01:00
Mikhail Korobov
ff3aec6613 Merge pull request #2331 from moisesguimaraes/fixes-2272 
[MRG+1] Fixes issue #2272 using arg_to_iter() to wrap single values and list() to…
 2016-12-07 20:08:18 +05:00
Elias Dorneles
a9c69458ff Merge pull request #2422 from rolando-contrib/nested-spiders-modules 
[MRG+1] DOC State explicitly that spiders are loaded recursively.
 2016-12-07 11:21:15 -02:00
Paul Tremberth
5efd65255c TST: Randomize IMAGES_EXPIRES above 90 days 2016-12-06 18:49:53 +01:00
nyov
534772f6ea Import xlib.tx code from twisted proper 2016-12-02 21:21:51 +00:00
Elias Dorneles
c4e67c0696 Merge pull request #2421 from rolando-contrib/tests-bug 
[MRG+1] TST: Fix duplicated test name.
 2016-12-01 18:43:48 -02:00
Elias Dorneles
f3a4420750 Merge pull request #2388 from redapple/robotparser-native-str 
[MRG+1] Parse robots.txt content as native str
 2016-12-01 18:43:23 -02:00
Rolando Espinoza
923b974f0a TST Include nested a nested spider in spider loader test. 2016-12-01 13:26:19 -03:00
Rolando Espinoza
d9f43e21ba TST: Fix duplicated test name. 2016-12-01 11:56:33 -03:00
Eugenio Lacuesta
5ff64ad015 handle relative sitemap urls in robots.txt 2016-12-01 09:53:40 -03:00
Paul Tremberth
9aefc0a886 Add test for fetch command with redirections disabled 2016-11-24 13:41:51 +01:00
Paul Tremberth
01142e2ae5 Print more dependencies versions in "scrapy version" verbose output 2016-11-22 14:48:33 +01:00