BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-19 12:58:08 +00:00
Code Releases Wiki Activity
7,850 Commits 39 Branches 119 Tags
Commit Graph

7850 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Eugenio Lacuesta
2e405d2d5c
Merge branch 'master' into response_follow_all 2020-01-05 00:33:19 -03:00
Mikhail Korobov
ce618fb6f2
Merge pull request #4259 from scrapy/asyncio-mw 
Asyncio support in downloader middlewares
 2020-01-03 22:28:41 +05:00
Andrey Rakhmatullin
b2dd379bc2 Remove the py35-asyncio env for 3.5 from Travis. 2020-01-03 21:38:05 +05:00
Andrey Rakhmatullin
2b9254c2bd Add a test function that uses asyncio.Queue(). 2019-12-31 17:54:41 +05:00
Andrey Rakhmatullin
e3b8ba6188 Run py35-asyncio also on 3.5.2 to test Xenial. 2019-12-31 17:54:01 +05:00
Andrey Rakhmatullin
16787f5bf4 Merge middleware tests back as we don't need to set the setting anymore. 2019-12-30 12:02:19 +05:00
Andrey Rakhmatullin
50aa6ef22c Add deferred_from_coro. 2019-12-30 11:46:45 +05:00
Andrey Rakhmatullin
5cf1ac0005 Move the asyncio downloader mw test to a separate class. 2019-12-30 11:46:45 +05:00
Andrey Rakhmatullin
3603644552 Add a non-asyncio async def middleware test. 2019-12-30 11:46:45 +05:00
Andrey Rakhmatullin
21f50c795a Add async def support to downloader middlewares. 2019-12-30 11:46:45 +05:00
1um0s
14d4428e70 Rephrasing documentation for image and file pipelines (#4252) 
* scrapy#4034 Clarify documentation for image and file pipelines

* scrapy#4034 Clarify documentation for file pipeline

* scrapy#4034 Simplify documentation for pipeline

* scrapy#4034 Simplify documentation for pipeline

* scrapy#4034 Clarify documentation for image and file pipelines

* scrapy#4034 Clarify documentation for file pipeline

* scrapy#4034 Simplify documentation for pipeline

* scrapy#4034 Simplify documentation for pipeline

* scrapy#4034 Revert image, file pipeline docs. Enhance custom media pipeline docs.

* scrapy#4034 rebase master

* scrapy#4034 Clarify documentation for image and file pipelines

* scrapy#4034 Clarify documentation for file pipeline

* scrapy#4034 Simplify documentation for pipeline

* scrapy#4034 Simplify documentation for pipeline

* scrapy#4034 Clarify documentation for image and file pipelines

* scrapy#4034 Clarify documentation for file pipeline

* scrapy#4034 Simplify documentation for pipeline

* scrapy#4034 Simplify documentation for pipeline

* scrapy#4034 Revert image, file pipeline docs. Enhance custom media pipeline docs.

* scrapy#4034 rebase master

* Rebase master

* Add class to media pipeline docs

Co-Authored-By: elacuesta <elacuesta@users.noreply.github.com>

Co-authored-by: elacuesta <elacuesta@users.noreply.github.com>
 2019-12-30 00:56:22 +05:00
Mikhail Korobov
f0ae673452
Merge pull request #4258 from atul-g/patch-1 
Edited the link provided to homepage of lxml's website
 2019-12-30 00:55:15 +05:00
Mikhail Korobov
bb991cd303
Merge pull request #4010 from scrapy/asyncio-base 
Base support for asyncio
 2019-12-30 00:51:28 +05:00
Atul Gopinathan
82861c73c8
Edited the link of the homepage of lxml website 
The link "https://lxml.de" is redirecting to a completely different and unintended website. I changed the link to the index page of lxml's official website. I thought of changing it to the PyPi page of lxml, but even they are providing the same "https://lxml.de" link which doesn't seem to be working now.
 2019-12-27 22:57:58 +05:30
Andrey Rakhmatullin
dc1ee09481 Rename ASYNCIO_ENABLED to ASYNCIO_REACTOR, change the logic accordingly. 2019-12-27 21:56:28 +05:00
Andrey Rakhmatullin
f75ccc997a FIx a typo in the only_asyncio fixture. 2019-12-27 19:48:54 +05:00
Andrey Rakhmatullin
30ebd05a5f Simplify the tox asyncio entries. 2019-12-27 00:05:14 +05:00
Andrey Rakhmatullin
37ac47ff80 Fix a deprecation warning. 2019-12-26 20:46:54 +05:00
Andrey Rakhmatullin
87ece066ca Remove conditional asyncio imports. 2019-12-26 20:41:06 +05:00
Eugenio Lacuesta
ab54e0d33e
Keyword-only args for S3DownloadHandler 2019-12-23 20:37:18 -03:00
Eugenio Lacuesta
982a66f9fb
[test] Download handler: avoid passing settings if not necessary 2019-12-23 20:28:17 -03:00
Eugenio Lacuesta
9a75b46fb8
Explicit argument names 2019-12-23 20:26:58 -03:00
Eugenio Lacuesta
2fb160e3ba
Use settings instead of crawler 2019-12-23 20:24:16 -03:00
Mikhail Korobov
4d594b8c2b
Merge pull request #4193 from Gallaecio/lgtm 
Use super().__init__ in BaseItemExporter subclasses
 2019-12-23 23:52:43 +05:00
Eugenio Lacuesta
e2e15d6651
Downloader handlers: sort imports 2019-12-23 10:48:19 -03:00
Eugenio Lacuesta
a6ec89251e
Downloader handlers: crawler=None in __init__ 2019-12-23 10:47:08 -03:00
Eugenio Lacuesta
8a567e98bb
Remove unnecessary __init__ methods in downloader handler tests 2019-12-23 10:36:46 -03:00
Eugenio Lacuesta
7e6387de40
Use create_instance in FTPDownloadHandler/DataURIDownloadHandler tests 2019-12-23 10:02:58 -03:00
Eugenio Lacuesta
fa21d8687a
Use create_instance in S3DownloadHandler tests 2019-12-23 10:00:25 -03:00
Eugenio Lacuesta
9e5d945ef2
Use create_instance in downloader handler tests 2019-12-23 09:55:47 -03:00
Eugenio Lacuesta
342bf3cd35
Explicit keyword arguments 2019-12-23 09:52:55 -03:00
Eugenio Lacuesta
931b7e68d3
Update FileDownloadHandler test 2019-12-23 09:50:28 -03:00
Eugenio Lacuesta
310cc081d0
Merge branch 'master' into from_crawler_downloader_handlers 2019-12-21 17:24:45 -03:00
Andrey Rakhmatullin
8de80f59db Raise an exception if ASYNCIO_ENABLED but the reactor is wrong. 2019-12-21 13:08:29 +05:00
Mikhail Korobov
5577d4d2be
Merge pull request #3869 from elacuesta/detect_return_in_generator_callbacks 
[MRG+1][Py3] Check for 'return' with arguments inside generators
 2019-12-20 22:46:46 +05:00
Andrey Rakhmatullin
e342de5038 Remove a stray newline. 2019-12-20 19:37:50 +05:00
Andrey Rakhmatullin
40697dcbfa Remove deferred_from_coro from this PR. 2019-12-20 19:33:44 +05:00
Andrey Rakhmatullin
20289be810 Merge remote-tracking branch 'origin/master' into asyncio-base 2019-12-20 19:31:20 +05:00
Mikhail Korobov
fb3fb17c28
Merge pull request #4045 from Gallaecio/documentation-coverage 
Deprecate FilteringLinkExtractor direct usage; add API docs for scrapy.linkextractors
 2019-12-20 02:01:45 +05:00
Mikhail Korobov
6a32a36e09
Merge pull request #4143 from Gallaecio/undent-rst-doctests 
Do not indent doctests from the documentation unnecessarily
 2019-12-20 01:59:07 +05:00
Mikhail Korobov
1213c0e5f0
Merge pull request #4247 from noviluni/fix_typos 
fix typos
 2019-12-20 01:56:36 +05:00
Mikhail Korobov
a611a7c0f2
Merge pull request #4249 from Gallaecio/docs-py37 
Use Python 3.7 to build the documentation
 2019-12-20 01:54:26 +05:00
Adrián Chaves
b4a2d985a7
Merge branch 'master' into undent-rst-doctests 2019-12-19 12:39:17 +01:00
Adrián Chaves
e22c0c27d9 Revert "Improve FilteringLinkExtractor.__new__" 
This reverts commit ee9881d2704798c9cd61b6da503bb0694227c58c.
 2019-12-19 12:15:54 +01:00
Adrián Chaves
f6bc1940a3 Use Python 3.7 to build the documentation 2019-12-19 12:06:15 +01:00
Andrey Rahmatullin
c841a1f3e7
Merge pull request #4140 from Gallaecio/docs-py38 
Use the latest Python version to build the documentation
 2019-12-19 14:09:02 +05:00
Marc Hernandez Cabot
16b363de31 Merge branch 'master' into fix_typos 2019-12-19 10:02:21 +01:00
Marc Hernandez Cabot
23a67cec27 fix first letter capitalization for Raring and Scrapy 2019-12-19 09:57:17 +01:00
Mikhail Korobov
3bcb14fe4d
Merge pull request #4164 from Gallaecio/speed-up-tests-2 
Improve the performance of the DOWNLOAD_DELAY test
 2019-12-18 23:56:56 +05:00
Mikhail Korobov
8f7faaa63d
Merge pull request #4190 from Gallaecio/doctest 
Make developer-tools doctests pass
 2019-12-18 23:40:21 +05:00