BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-24 03:43:56 +00:00
Code Releases Wiki Activity
7,310 Commits 39 Branches 119 Tags
Commit Graph

7310 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Andrey Rakhmatullin
98689b27a8 Improve the DOWNLOADER_CLIENTCONTEXTFACTORY doc. 2019-07-11 14:02:35 +05:00
Andrey Rakhmatullin
0de6ffc8e1 Fix super() call. 2019-07-11 13:12:56 +05:00
Robin
5cdf277017 Update broad-crawls.rst 
Added section on how to treat memory consumption problems of broad crawls.
 2019-07-11 08:49:24 +02:00
Luiz Francisco Rodrigues da Silva
df68c4b9b1 Add a setting to use active mode in FTPFeedStorage (#3829) 
Add a setting to use active mode in FTPFeedStorage

FTP servers can be configured in active and passive mode, by default
Python and Scrapy use passive mode and there was no way to use active.

This commit adds a setting FEED_STORAGE_FTP_ACTIVE to allow a feed
to be exported to a FTP server configured in active mode.
 2019-07-10 14:25:52 +02:00
Mikhail Korobov
050a62a955
Merge pull request #3861 from Gallaecio/documentation-coverage 
Increase the documentation coverage of exceptions
 2019-07-10 10:18:34 +05:00
Adrián Chaves
68d898d493
Merge pull request #3712 from elacuesta/docs_crawlspider_link_text 
[Docs] CrawlSpider: update Rule docs
 2019-07-10 00:18:51 +02:00
Eugenio Lacuesta
9fb0f8454e
Extend docs about Crawling Rules 2019-07-09 15:30:22 -03:00
Eugenio Lacuesta
d04e84c11d Merge remote-tracking branch 'upstream/master' into docs_crawlspider_link_text 2019-07-09 15:24:31 -03:00
Adrián Chaves
a0bbbe8ffa Add the API documentation of ContractFail to the contracts topic 2019-07-09 12:33:57 +02:00
Adrián Chaves
578bccf3bb Skip scrapy.exceptions.UsageError in the documentation coverage report 2019-07-09 12:33:07 +02:00
Mikhail Korobov
a97a9b57bf
Merge pull request #3621 from Gallaecio/bfo-for-few-requests 
Document that the crawl order is BFO for small numbers of start requests
 2019-07-08 21:26:04 +05:00
Mikhail Korobov
dda12b6537
Merge pull request #3672 from Gallaecio/split-items 
Add a FAQ entry about splitting items in item pipelines
 2019-07-08 20:15:23 +05:00
Mikhail Korobov
488156b15f
Merge pull request #3860 from Gallaecio/documentation-coverage 
Skip scrapy.downloadermiddlewares private APIs in the documentation c…
 2019-07-08 20:13:55 +05:00
Adrián Chaves
a0b09e0193 Skip scrapy.downloadermiddlewares private APIs in the documentation coverage report 2019-07-08 14:45:17 +02:00
Andrey Rakhmatullin
0b9dce3a6c Add DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGING setting. 2019-07-08 17:40:56 +05:00
Adrián Chaves
783de8a8d8 Document how request concurrency settings impact request order 2019-07-08 14:23:21 +02:00
Adrián Chaves
6ea62cacc7 Add a FAQ entry about splitting items in item pipelines 2019-07-08 13:58:55 +02:00
Mikhail Korobov
3dff65ac2c
Merge pull request #3857 from mikolaje/master 
PEP8 in cmdline.py
 2019-07-08 13:54:35 +05:00
Adrián Chaves
9cab3f247d
Merge pull request #3859 from scrapy/more-deprecations 
More deprecations
 2019-07-08 09:46:48 +02:00
Mikhail Korobov
4bef6f438d
Merge pull request #3578 from nyov/deprecations 
Deprecation removals for Scrapy 1.7
 2019-07-08 11:32:07 +05:00
Mikhail Korobov
cb4477db3e deprecate scrapy.utils.markup 
it was an import-only shim for w3lib.html
 2019-07-08 11:18:40 +05:00
Mikhail Korobov
64ff3cd6aa deprecate scrapy.utils.multipart 
It was a shim for w3lib.form, but w3lib.form is deprecated as well.
 2019-07-08 11:18:15 +05:00
Mikhail Korobov
1130711cc1 deprecate scrapy.utils.http 
ChunkedTransferMiddleware is deprecated, so decode_chunked_transfer
can be deprecated as well.
 2019-07-08 11:17:30 +05:00
Andrey Rakhmatullin
67a4000928 Work around older pyOpenSSL not having get_cipher_name or get_protocol_version_name. 2019-07-08 10:31:52 +05:00
nyov
5442c2d3c3 Updating S3FeedStorage instancing without AWS key. 2019-07-06 13:41:56 +00:00
nyov
7a398b7086 Deprecation removals for Scrapy 1.7 
Removing deprecations of 2015 and prior (pre-1.1)
 2019-07-06 13:41:56 +00:00
mikolaje
8f52dada55 PEP8 in cmdline.py 2019-07-06 19:55:09 +08:00
Daniel Graña
3d8f075b0a
Merge pull request #3840 from mabelvj/itemloader-errors 
[MRG+1] Itemloader errors #3836
 2019-07-05 08:55:43 -03:00
Mikhail Korobov
4d4bd0e823
Merge pull request #3609 from Gallaecio/2253 
Document FilesPipeline.file_path and ImagesPipeline.file_path
 2019-07-04 22:05:46 +05:00
Andrey Rakhmatullin
69b1d5d3d7 Log cipher, certificate and temp key info on establishing an SSL connection. 2019-07-04 12:13:39 +05:00
Mikhail Korobov
4c755f1ddc
Merge pull request #3692 from anubhavp28/doc-patch 
[MRG+1] doc for creating custom cache storage backend.
 2019-07-02 20:18:14 +05:00
Mikhail Korobov
9aec7856b0
Merge pull request #3794 from csalazar/whitelist-form-methods-in-fromresponse 
[MRG+1] Fix form methods in FormRequest.from_response (#3777)
 2019-07-02 20:08:14 +05:00
Claudio Salazar
2e4dc20393 Add backward compability when method=None in FormRequest 2019-06-26 21:36:28 +02:00
Daniel Graña
3adf09bc4f
Merge pull request #3563 from elacuesta/callback_kwargs 
[MRG+1] Callback kwargs
 2019-06-26 14:01:29 -03:00
Eugenio Lacuesta
312e57382c Request.cb_kwargs: update in spider middleware 2019-06-26 12:48:00 -03:00
Eugenio Lacuesta
d4d68cf321 Request.cb_kwargs: update in downloader middleware 2019-06-26 12:31:41 -03:00
Eugenio Lacuesta
1f9f41b850 Move request.cb_kwargs tests to their own test file 2019-06-26 12:31:12 -03:00
Eugenio Lacuesta
428309ba1a Merge remote-tracking branch 'origin/master' into callback_kwargs 2019-06-26 11:03:31 -03:00
Mabel Villalba
268a37cb7b [itemloader-errors] updated Compose and MapCompose function names 
#3836
 2019-06-26 13:23:10 +02:00
Mikhail Korobov
c81d120bde
Merge pull request #3768 from Gallaecio/meta-ignore-tags 
Implement the METAREFRESH_IGNORE_TAGS setting
 2019-06-25 23:25:58 +03:00
Mikhail Korobov
e5f12faf5f
Merge pull request #3703 from Gallaecio/ajax-docs 
Add a topic about reaching data that selectors cannot reach
 2019-06-25 23:11:59 +03:00
Mikhail Korobov
bc137dd215
Merge pull request #3842 from Gallaecio/documentation-coverage 
Document scrapy.crawler.Crawler.stop()
 2019-06-25 22:28:55 +03:00
Adrián Chaves
d7795484dc Document scrapy.crawler.Crawler.stop() 2019-06-25 17:29:49 +02:00
Mabel Villalba
b1e348b281 [itemloader-errors] updated Compose and MapCompose messages and added tests 
#3836
 2019-06-25 17:11:14 +02:00
Mabel Villalba
a753ea7e16 [itemloader-errors] added test for Compose and MapCompose 
#3836
 2019-06-25 14:35:49 +02:00
Mabel Villalba
99eb03a810 [itemloader-errors] adapted compose test for the new error being returned by compose 
#3836
 2019-06-25 14:24:27 +02:00
Mabel Villalba
ef56e34a4a [itemloader-errors] fixed typo 
#3836
 2019-06-25 13:56:53 +02:00
Mabel Villalba
ad8c980010 [itemloader-errors] undo _proc 
#3836
 2019-06-25 13:54:47 +02:00
Mabel Villalba
f134b1daf3 [itemloader-errors] reordered method 
#3836
 2019-06-25 13:46:44 +02:00
Mabel Villalba
e5d17b4efe [itemloader-errors] reordered method 
#3836
 2019-06-25 13:44:20 +02:00