Konstantin Lopuhin
722e1afcdb
Update ancient pytest on python 3
...
2.9 gives collection errors on python 3.7 due to PEP 479.
2018-07-09 12:21:19 +03:00
Konstantin Lopuhin
1d2e2735fc
Merge branch 'master' of https://github.com/patiences/scrapy into patiences-master
2018-07-09 11:59:22 +03:00
Daniel Graña
666e23714a
Merge pull request #3318 from scrapy/delete-relocation-shims
...
[MRG+1] Delete relocation shims
2018-07-06 13:49:59 -03:00
Mikhail Korobov
f531b66822
SpiderManager shim is removed
2018-07-06 03:28:01 +05:00
Mikhail Korobov
36453348fa
remove ancient modules kept only for error messages
2018-07-06 03:23:37 +05:00
Mikhail Korobov
d4c7cc848b
remove backwards compatibility shims for relocated modules
2018-07-06 03:19:43 +05:00
Oz T
6f5c39d65f
Fix for CSV export unnecessary blank lines problem on Windows ( #3039 )
2018-07-03 18:22:24 -03:00
chainly
74ce156154
add item_error to be catchable ( #3256 )
2018-07-03 16:00:59 -03:00
Grammy Jiang
d05c8677c5
[MRG+1] change the bad smell code ( #3304 )
...
Change the bad smell code
2018-07-03 15:58:43 -03:00
Daniel Graña
8d93691a8a
Merge pull request #3311 from scrapy/rename-requirements
...
[MRG+1] TST make it clear which requirements are Python 2-only
2018-07-03 10:33:26 -03:00
Mikhail Korobov
f11d65f7d6
TST make it clear which requirements are Python 2-only
...
* rename requirements.txt to requirements-py2.txt, to make it clear they are Python 2-only
* make requirements-py3.txt consistent with requirements-py2.txt
2018-06-29 18:34:11 +05:00
Daniel Graña
64f48efa26
Merge pull request #3308 from scrapy/fix-build
...
TST exclude lxml==4.2.2 from tests
2018-06-27 23:42:32 -03:00
Mikhail Korobov
8782901fc8
[MRG+1] TST test agains latest pypy ( #3309 )
...
pypy3 is not upgraded, as tests segfault with pypy3 6.0 for some reason
2018-06-28 01:11:15 +05:00
Mikhail Korobov
45f67eb64d
TST exclude lxml==4.2.2 from tests, as it doesn't play well with Pillow
2018-06-28 01:02:30 +05:00
Mikhail Korobov
fac1b2f351
TST remove workaround for old Pillow versions which don't support BytesIO
2018-06-28 01:01:27 +05:00
Daniel Graña
bc4dbd2766
Merge pull request #3305 from grammy-jiang/patch-1
...
Make the version of ipython less than 6.0 in python 2.7
2018-06-27 11:19:32 -03:00
Daniel Graña
594a441eb5
Merge pull request #3294 from rennerocha/feed-exporter-docs
...
[MRG+1] Improve docs of S3 Storage to make Python version more explicit
2018-06-27 11:16:07 -03:00
Grammy Jiang
9ad3af9d88
Update requirements.txt
...
make the version of ipython less than 6.0 in python 2.7
2018-06-23 17:31:54 +08:00
Mikhail Korobov
1fd1702a11
Merge pull request #3299 from grammy-jiang/dev
...
[MRG+1] fix the test case name of HttpProxyMiddleware
2018-06-22 04:17:18 +05:00
Grammy Jiang
88bd067912
fix the test case name of HttpProxyMiddleware
2018-06-20 16:56:46 +08:00
Konstantin Lopuhin
991e9b889f
Merge pull request #3298 from leo8a/master
...
fix typo in news.rst: extractred --> extracted
2018-06-19 18:10:14 +03:00
Leo
7a601d76de
fix typo
...
extractred --> extracted
2018-06-19 10:51:55 +02:00
Vostretsov Nikita
72d0899bce
Return non-zero exit code from scrapy commands in case of spider bootstrap errors
...
* method to detect spider creation in crawler
* correct method name
* method to know if crawlers has spiders
* we do not need to issue requests
* set exit code accordingly to spiders in crawlers
* more portable way to check ofr exceptions
* more clear way
* test cases for several spiders per crawler
* grammatically correct name for method
* method is private
* grammatically correct name for method
* method is private
* remove unused import
* correct order of imports
* changes mechanism of obtaining spider status from method to object member
* rename tests
2018-06-14 19:58:48 +05:00
Mikhail Korobov
667eb7151b
Merge pull request #3284 from mugayoshi/issue3282
...
[MRG+1] Update debugging memory leaks section in the docs
2018-06-14 19:56:52 +05:00
Renne Rocha
e2bb218e9a
Include Python version indication to each required library used in S3 storage
2018-06-13 18:11:43 -03:00
mugayoshi
d4511667fb
Update debugging memory leaks section in the docs
...
Add Python3 tools description.
2018-06-09 18:17:11 +09:00
Daniel Graña
c6030ce8c6
Merge pull request #3231 from starrify/updating-argument-of-cookiejar-clear
...
[MRG+2] Added: Allowing optional arguments for `scrapy.http.cookies.CookieJar.clear`
2018-06-01 21:52:32 -03:00
Colton Herinckx
596f39600d
reversed earlier change that seemed to cause Travis CI build failure
2018-06-01 21:48:43 -03:00
Colton Herinckx
12d10eec2c
changed Twisted >= 17.9.0 to Twisted>=17.9.0
2018-06-01 21:48:43 -03:00
Colton Herinckx
9bd5444a42
added oxford commas to LICENSE
2018-06-01 21:48:43 -03:00
Colton Herinckx
98d9093dc7
minor grammatical fixes in CODE_OF_CONDUCT.md
2018-06-01 21:48:43 -03:00
Mikhail Korobov
3cf871c61f
Merge pull request #3281 from fbergen/gunzipperf
...
[MRG+2] Improve gunzip performance for big files on Python 3
2018-06-02 04:13:26 +05:00
Mikhail Korobov
847b50ce9c
Merge pull request #3201 from grammy-jiang/master
...
[MRG+1] fix a mistake in topic spider-middleware.rst
2018-06-02 01:09:59 +05:00
Fredrik Bergenlid
6a2d2c3b77
Improve gunzip performance for big files
2018-06-01 21:38:07 +02:00
Mikhail Korobov
13b15dc9a8
Merge pull request #3279 from lewoudar/patch-1
...
Update spiders.rst
2018-06-01 22:20:41 +05:00
Mikhail Korobov
f1d87ee09a
Merge pull request #3280 from cms-/patch-1
...
Minor edits to contributing.rst
2018-06-01 20:54:39 +05:00
Chris Slothouber
ecdd888ff4
Minor edits to contributing.rst
...
Corrected minor grammatical issues and increased clarity of instructions.
2018-06-01 09:25:34 -04:00
Kevin Tewouda
ffa7bede17
Update spiders.rst
...
I changed URLs to :class:`~scrapy.http.Request` in start_urls explanation of the default spider
2018-05-30 06:33:18 +02:00
Vostretsov Nikita
b364d27247
[MRG+1] Automatic port selection for servicies in unit tests ( #3210 )
...
* ability to pass port as a parameter
* try to find free ports
* use environment variables to pass mock server address
* get mock server address from environment variables
* ability to select ports for proxy in runtime
* use common method for URLs from mock server
* https support
* get mock server address
* get mock address
* replace hand-written mechanism by kernel-based one
* use ephemeral ports in mockserver
* strip EOL from addresses
* use ephemeral port in proxy
* no need to restore environment as it is restored in tearDown
* decode bytes
* use mockserver address as a variable
* ability to pass address as variable
* per test-case mockserver
* use base class
* remove obsolete environment manipulation
* return usage of proxy for http cases
* common method for broking proxy auth credentials
* python version-independent url methods
2018-05-23 21:25:50 +03:00
Ryan P Kilby
6a182c9552
Depth stats are not optional
2018-05-17 11:00:21 -04:00
Ryan P Kilby
2dfc5d128b
Update DEPTH_STATS refs to DEPTH_STATS_VERBOSE
2018-05-17 11:00:21 -04:00
Konstantin Lopuhin
bac1e2d47d
Merge pull request #3263 from whalebot-helmsman/no_twisted_18_4_0
...
[MRG+1] Blacklist twisted version with regression
2018-05-17 13:21:42 +03:00
Vostretsov Nikita
c5ddfddb7e
blacklist twisted version with regression in constraints file
2018-05-17 08:53:42 +00:00
Vostretsov Nikita
0d015e5c0f
blacklist twisted version with regression
2018-05-16 09:36:07 +00:00
Konstantin Lopuhin
c4f096d3a5
Merge pull request #3224 from lucywang000/better-processors-doc
...
[MRG+1] Improve document about functions as processors
2018-04-26 09:36:39 +03:00
Konstantin Lopuhin
f36e1b53d4
Merge pull request #3199 from rhoboro/gcs_acl
...
[MRG+1] FilesPipeline supports ACL for Google Cloud Storage
2018-04-25 19:42:19 +03:00
Pengyu Chen
e75f721c04
Added: Allowing optional arguments for scrapy.http.cookies.CookieJar.clear
2018-04-23 22:08:28 +08:00
Lucy Wang
57b0e6b695
improve document about functions as processors
2018-04-19 13:35:46 +08:00
rhoboro
6ef6585b5a
update docs
2018-04-13 19:06:29 +09:00
rhoboro
560ee623fd
set defalut value "" to FILES_STORE_GCS_ACL
2018-04-13 19:00:27 +09:00
