BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-19 13:58:19 +00:00
Code Releases Wiki Activity
7,850 Commits 39 Branches 119 Tags
Commit Graph

7850 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Andrey Rahmatullin
7079d12c5e
Merge pull request #4212 from grammy-jiang/fix-imports 
Convert the relative imports to absolute imports
 2019-12-05 12:44:22 +05:00
Andrey Rahmatullin
aaf94affee
Merge pull request #4213 from dqwerter/patch-1 
Update overview.rst | Fix an inconsistency
 2019-12-05 12:43:24 +05:00
Wang Qin
af624ef414
Update overview.rst | Fix an inconsistency 
There exists an inconsistency between the code (line 37 - 38) and the output 'quotes.json' (line 56 - 68). 

Note that even though according to line 53 - 54  'quotes.json' is "reformatted here for better readability", it cannot explain why the "author" field precedes the "text" field. 

Intended output for the code BEFORE change: 
    [{
        "text": "\u201cThe person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.\u201d",
        "author": "Jane Austen"
    },
    {
        "text": "\u201cOutside of a dog, a book is man's best friend. Inside of a dog it's too dark to read.\u201d",
        "author": "Groucho Marx"
    },
    {
        "text": "\u201cA day without sunshine is like, you know, night.\u201d",
        "author": "Steve Martin"
    },
    ...]

Intended output for the code After change (the inconsistency is fixed): 
    [{
        "author": "Jane Austen",
        "text": "\u201cThe person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.\u201d"
    },
    {
        "author": "Groucho Marx",
        "text": "\u201cOutside of a dog, a book is man's best friend. Inside of a dog it's too dark to read.\u201d"
    },
    {
        "author": "Steve Martin",
        "text": "\u201cA day without sunshine is like, you know, night.\u201d"
    },
    ...]
 2019-12-05 09:29:12 +08:00
Grammy Jiang
9b4b43f8ac
Convert the relative imports to absolute imports 
This commits converts the relative imports to absolute imports in the
entire package
 2019-12-05 11:25:19 +11:00
Eugenio Lacuesta
2c010152c3
Merge remote-tracking branch 'upstream/master' into remove-six-code 2019-12-04 15:43:02 -03:00
Andrey Rakhmatullin
0b9f29215f Update .travis.yml. 2019-12-04 22:06:35 +05:00
Andrey Rakhmatullin
97fb61cec8 Move an import to postpone another "import twisted.internet.reactor". 2019-12-04 21:53:07 +05:00
Andrey Rakhmatullin
ed34ce14c0 Add the ASYNCIO_SUPPORT setting, reshuffle other logic accordingly. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin
c079d5002b Run tests without asyncio support by default, add py35-asyncio and py38-asyncio envs. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin
794cf71806 Fix or ignore flake8 problems. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin
3ba25ccbd3 Don't use asyncio.iscoroutine, as it is True for generators. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin
f41c2f3874 Add py38-no-asyncio to Travis. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin
cc19ab5439 Add tests that check asyncio support. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin
2fbe7d49dc Log asyncio support on spider start. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin
b04b541372 Install the asyncio reactor only in scrapy.cmdline. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin
8d8fbddbde Switch to the released version of pytest-twisted. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin
63c3c62305 Add utils.deferred_from_coro. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin
9777639533 Run tests using the asyncio reactor. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin
1b437bbe9f Install the asyncio reactor on "import scrapy". 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin
1b35260625 Add a test for downloader middlewares using Deferreds. 2019-12-04 21:32:56 +05:00
Grammy Jiang
74627033c4 Remove the used import and re-arrange the imports (#4208) 
This commit removes unused import and re-arrange the imports in cookies
module
 2019-12-04 14:24:14 +01:00
Grammy Jiang
702333478d Re-arrange the imports in httpcache module (#4209) 
This commit re-arrange the imports in httpcache module to follow pep8
 2019-12-04 14:23:28 +01:00
Grammy Jiang
5d8d4bb7d7 Re-arrange the imports in the httpproxy module (#4210) 
This commit re-arranges the imports in the httpproxy module to follow
pep8
 2019-12-04 14:22:10 +01:00
Eugenio Lacuesta
62778cf23f
Request: remove restriction about errback without callback 2019-12-04 08:53:38 -03:00
Eugenio Lacuesta
2a9f5a0aef
Skip invalid links when passing SelectorLists to Response.follow_all 2019-12-03 16:01:42 -03:00
Mikhail Korobov
9b7452211a
Merge pull request #4099 from BurnzZ/itemloader-docs 
update docs of scrapy.loader.ItemLoader.item
 2019-12-03 13:14:45 +05:00
Eugenio Lacuesta
e43f37fff3
Pass args/kwargs in S3DownloadHandler.from_crawler, update tests 2019-12-02 17:36:31 -03:00
Eugenio Lacuesta
3d77f74e40
Download handlers: from_crawler factory method, take crawler instead of settings in __init__ 2019-12-02 17:36:31 -03:00
Adrián Chaves
5980b0f284 Don’t use follow_all where a single item is expected (#4) 2019-12-02 12:47:44 -03:00
Grammy Jiang
d1cdfb4701 Use pprint.pformat on overridden settings (#4199) 
Keeps consistency with scrapy.middleware
 2019-11-29 09:13:57 +01:00
Mikhail Korobov
3a5b86220f
Merge pull request #4194 from Gallaecio/intersphinx 
Use InterSphinx for coverage links
 2019-11-28 17:36:19 +05:00
Eugenio Lacuesta
048cd74ae5
Add separate mapping for Twisted API docs 2019-11-27 19:16:18 -03:00
Eugenio Lacuesta
17e6481823
[Docs] Fix Twisted links 2019-11-27 18:42:42 -03:00
Andrey Rakhmatullin
63546cbf3e Deprecate the HTTPS proxy noconnect mode. 2019-11-27 22:42:52 +05:00
Adrián Chaves
b73fc99b60 Use InterSphinx for coverage links 2019-11-26 10:31:55 +01:00
Adrián Chaves
dd12f5fdcd Use Response.follow_all in the documentation where appropiate 2019-11-25 11:59:59 -03:00
Adrián Chaves
ed1e577610 Use super().__init__ in BaseItemExporter subclasses 2019-11-25 13:38:10 +01:00
Adrián Chaves
54b056c4be Make developer-tools doctests pass 2019-11-25 12:32:35 +01:00
Adrián Chaves
6d9ed6146d
Merge branch 'master' into remove-six-code 2019-11-25 10:34:21 +01:00
Andrey Rahmatullin
8a1c99676e
Merge pull request #3899 from elacuesta/py3_single_argument_processors 
[Py3] Item loaders: allow single argument functions as processors
 2019-11-25 13:47:58 +05:00
Eugenio Lacuesta
40b5cfc0a4
Item loaders: allow single-argument processors (unbound methods) 2019-11-22 20:47:22 -03:00
Eugenio Lacuesta
6fa8f06b64
Merge remote-tracking branch 'upstream/master' into response_follow_all 2019-11-22 13:05:06 -03:00
Eugenio Lacuesta
55cc5c9068
Skip pickle in bandit check 2019-11-22 12:41:31 -03:00
Eugenio Lacuesta
5bab3c0261
Merge remote-tracking branch 'upstream/master' into remove-six-code 2019-11-22 12:12:29 -03:00
Mikhail Korobov
16e0636dcf
Merge pull request #4186 from Gallaecio/lgtm 
Remove unused imports
 2019-11-22 12:28:29 +05:00
Adrián Chaves
9b5053c564 Undo unintended tox.ini changes 2019-11-21 22:00:34 +01:00
Mikhail Korobov
0d416c6191
Merge pull request #4185 from Gallaecio/intersphinx 
Use InterSphinx for links to the pytest and tox documentation
 2019-11-21 23:27:57 +05:00
Mabel Villalba
070b3a4e84
Merge branch 'master' into 4133-handle-start_url 2019-11-21 17:10:31 +01:00
Mabel Villalba
1718e450ef [start_url] Fixes #4133: Raise AttributeError error when empty 'start_urls' and 'start_url' found. Added test. 2019-11-21 17:06:47 +01:00
Mikhail Korobov
82c01c261f
Merge pull request #4184 from Gallaecio/doctest 
Make debug doctests pass
 2019-11-21 20:53:56 +05:00