BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-21 08:12:59 +00:00
Code Releases Wiki Activity
6,451 Commits 39 Branches 119 Tags
Commit Graph

6451 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Chomba Ng'ang'a
4ca61a2051 Update deprecated test aliases 
- change ``failIf`` to ``assertFalse``
- change ``asertEquals`` to ``assertEqual``
- change ``assert_`` to ``assertTrue``

https://docs.python.org/2/library/unittest.html#deprecated-aliases
 2017-08-09 12:52:10 +02:00
Paul Tremberth
a68a8f8fdc Merge remote-tracking branch 'origin/1.4' 2017-08-05 11:45:50 +02:00
david watson
71d5b7d75a fix typo (#2867) 2017-08-01 14:49:22 -03:00
Andrei Petre
6e6b5cc29f Use getfullargspec under the scenes for py3 to stop DeprecationWarning (#2864) 
Use getfullargspec under the scenes for py3 to stop DeprecationWarning. 

Closes #2862
 2017-08-01 11:14:43 -03:00
Daniel Graña
19382c5675 Merge pull request #2755 from redapple/downloader-mdw-template 
[MRG+1] Add template for a downloader middleware
 2017-07-31 11:37:26 -03:00
simik-ru
a65fec050a Small fix in description of startproject arguments 2017-07-31 13:03:49 +02:00
Paul Tremberth
aaaa4da7a4 Add template for a downloader middleware 2017-07-28 13:48:23 +02:00
Mikhail Korobov
9d34b2fe16 Merge pull request #2857 from redapple/scrapy-components-logs-startup 
Log versions information at startup
 2017-07-28 00:02:06 +05:00
Paul Tremberth
bf7ef3e4c3 Move methods to a new scrapy.utils.versions 2017-07-27 20:08:30 +02:00
Paul Tremberth
219c8aa0b6 Log versions information at startup 2017-07-27 17:30:30 +02:00
Daniel Graña
5d9bac789d Merge pull request #2849 from cclauss/patch-2 
[MRG+1] xrange() --> range() for Python 3
 2017-07-26 15:59:37 -03:00
Daniel Graña
bbc56e6960 Merge pull request #2852 from starrify/new-http-codes-to-retry-2851 
Added: HTTP status code 522/524 to retry.
 2017-07-26 15:46:35 -03:00
Daniel Graña
881a5e3a57 Merge pull request #2847 from redapple/redirect-308 
Handle HTTP 308 Permanent Redirect
 2017-07-26 15:29:23 -03:00
Paul Tremberth
15a5c533fa Add tests for HTTP 307 permanent redirects 2017-07-26 19:07:57 +02:00
Paul Tremberth
1fdc10684f HTTP Cache: treat 308 as 301 2017-07-26 19:01:48 +02:00
Paul Tremberth
5dc9a88c34 Handle HTTP 308 Permanent Redirect 2017-07-26 19:01:48 +02:00
Pengyu Chen
11a1f970b7
Added: HTTP status code 522/524 to retry. 2017-07-26 16:11:13 +08:00
cclauss
33dfac5018 xrange() --> range() for Python 3 
Either this PR or #2845.
 2017-07-24 22:06:17 +02:00
Mikhail Korobov
17bbd71433 Merge pull request #2812 from elacuesta/inspect_response_populate_spider 
Populate spider variable when using shell.inspect_response
 2017-07-25 00:28:14 +05:00
Daniel Graña
b6d036ede9 Merge pull request #2837 from dguo/patch-2 
Fix a typo in the Items documentation
 2017-07-24 15:55:30 -03:00
Daniel Graña
6de106e7a8 Merge pull request #2763 from scrapy/dataloss-typo 
DOC fixed rst syntax in DOWNLOAD_FAIL_ON_DATALOSS docs
 2017-07-24 15:48:03 -03:00
Daniel Graña
12971a7b61 Merge pull request #2764 from scrapy/readme-cleanup 
DOC change "releases" section content
 2017-07-24 15:47:39 -03:00
Daniel Graña
c01c1ef2ae Merge pull request #2655 from Granitosaurus/ptpython_support 
[MRG+1] add support for embeded ptpython shell
 2017-07-24 15:42:00 -03:00
Daniel Graña
2371a2a0df Merge pull request #2789 from starrify/add-response-follow-tag-link 
[MRG+1] Added: Now supporting <link> tags in Response.follow
 2017-07-24 15:41:30 -03:00
Daniel Graña
d1e948cef8 Merge pull request #2791 from starrify/doc-DontCloseSpider 
[MRG+1] Added doc for `scrapy.exceptions.DontCloseSpider`. Also fixes inaccurate doc for `scrapy.signals.spider_idle`.
 2017-07-24 15:29:38 -03:00
Daniel Graña
8e0b640d2b Merge pull request #2826 from dguo/patch-1 
[MRG+1] Tweak the CSVFeedSpider documentation
 2017-07-24 15:28:15 -03:00
Daniel Graña
45b81693b4 Merge pull request #2769 from stummjr/issue-2766 
[MRG+1] Add verification to check if Request callback is callable
 2017-07-24 15:14:33 -03:00
Daniel Graña
50b2567d0d Merge pull request #2848 from redapple/tox-jessie-ssl 
[WIP] Jessie toxenv: Add cryptography as per https://packages.debian.org/jessie/python-cryptography
 2017-07-24 14:44:41 -03:00
Paul Tremberth
1a18587d41 Jessi toxenv: Add cryptography as per https://packages.debian.org/jessie/python-cryptography 2017-07-24 19:30:08 +02:00
Valdir Stumm Junior
bb0bd691d9 Improve error message when callback is not callable 2017-07-24 11:12:09 -03:00
Danny Guo
26c488970c Fix a typo in the Items documentation 2017-07-18 19:56:51 -05:00
Mikhail Korobov
17fe6d2553 Merge pull request #2828 from cconrad/patch-1 
Spelling mistake
 2017-07-15 12:09:45 +02:00
Claus Conrad
18b96dd82a Spelling mistake 2017-07-15 11:31:09 +02:00
Danny Guo
dedc4a8b8f Tweak the CSVFeedSpider documentation 2017-07-13 22:58:10 -05:00
Mikhail Korobov
15abc0b9e3 Merge pull request #2816 from redapple/dns-cache-disabled 
Fix DNS resolver when DNSCACHE_ENABLED=False
 2017-07-06 15:53:33 +05:00
Paul Tremberth
f0ded6b775 Do not cache DNS responses when cache size is 0 2017-07-04 23:18:15 +02:00
Paul Tremberth
1f08d9a648 Add test for DNS cache disabling 2017-07-04 23:10:19 +02:00
Eugenio Lacuesta
793b2376f8 Populate spider variable when using shell.inspect_response 2017-07-03 11:28:04 -03:00
Mikhail Korobov
dd5ab6c5fe Merge pull request #2793 from lopuhin/pypy2 
Fix PyPy test failures
 2017-07-03 18:36:54 +05:00
Konstantin Lopuhin
b0a9236357 Use environment markers for custom PyPy requirements 2017-06-19 19:16:50 +03:00
Konstantin Lopuhin
5ba8e5adc0 Remove duplicate PyPy toxenv from Travis config 
Thanks for the catch @redapple
 2017-06-19 17:45:28 +03:00
Konstantin Lopuhin
271b3a485c Require pypy build to pass 2017-06-19 16:46:16 +03:00
Konstantin Lopuhin
ea08b95280 Remove Jython gc branch: it's not supported 2017-06-19 16:45:29 +03:00
Konstantin Lopuhin
a8df090071 Fix httpcache leveldb tests: gc.collect after del 
LevelDB does not have "official" close method, so we have
to rely on garbage collection to close it.
 2017-06-15 14:29:28 +03:00
Konstantin Lopuhin
b4eb60e527 Install PyPyDispatcher for PyPy tests 
Using https://github.com/lopuhin/pydispatcher, pypy branch.
This is executed as a separate step to avoid changing
default requirements.txt and setup.py. If just added to "deps"
in tox, this install command will be executed as one command
and PyPyDispatcher will not override PyDispatcher.
 2017-06-15 13:34:58 +03:00
Konstantin Lopuhin
19ca986aa1 Move garbage_collect to scrapy.utils.python 2017-06-15 13:14:31 +03:00
Konstantin Lopuhin
7c67047e77 Fix get_func_args tests under PyPy 
On CPython get_func_args does not work correctly for built-in
methods.
 2017-06-15 13:07:59 +03:00
Konstantin Lopuhin
5abb70c8d7 Fix test_weakkeycache on PyPy: run gc.collect() 
One gc.collect() seems to be enough, but it's more reliable
to run it several times (at most 100), until all objects are collected.
 2017-06-15 13:06:30 +03:00
Konstantin Lopuhin
c3d17659b3 Fix queue serialization test on PyPy 
It is not affected by Twisted bug #7989 and is more permissive
with pickling (especially with protocol=2).
 2017-06-15 13:05:45 +03:00
Konstantin Lopuhin
6014856df5 Fix test_output_processor_error undere PyPy 
For float(u'$10') PyPy includes "u'" in the error message,
and it's more fair to check error message on input we are really
passing.
 2017-06-15 12:57:32 +03:00