BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-27 04:44:26 +00:00
Code Releases Wiki Activity
5,464 Commits 39 Branches 119 Tags
Commit Graph

5464 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
stummjr
27758f60ad Changes fallback for CSVItemExporter, avoiding to call to_native_str(str()). 2016-01-27 16:28:01 -02:00
Paul Tremberth
27fb200e45 Merge pull request #1727 from redapple/priority-adjust 
[MRG+1] Clarify priority adjust settings docs
 2016-01-27 18:52:39 +01:00
stummjr
c55ff110a3 Fix CSV exporter for non string Python types. 2016-01-27 15:43:17 -02:00
stummjr
f1d971a5c0 fix PythonItemExporter for non-string types 2016-01-27 14:34:46 -02:00
Artur Gaspar
b2beb3e85d Fix handling of already failed deferreds when downloading page in robots.txt middleware. 2016-01-27 13:09:08 -02:00
Artur Gaspar
dc8701ea42 Add test for already failed deferreds when downloading page in robots.txt middleware. 2016-01-27 12:56:42 -02:00
Mikhail Korobov
7d24df3738 Merge pull request #1681 from redapple/code-of-conduct 
[MRG+1] Add Code of Conduct Version 1.3.0 from http://contributor-covenant.org/
 2016-01-27 17:56:11 +05:00
Mikhail Korobov
7ca9ae1976 DOC typo fix 2016-01-27 17:54:28 +05:00
Elias Dorneles
77951095e0 Merge pull request #1730 from scrapy/response-text 
[MRG+1] response.text
 2016-01-27 10:45:28 -02:00
Paul Tremberth
e0f48c486e Add link to CoC mardown file on Github 2016-01-27 13:04:08 +01:00
Paul Tremberth
d999e3f7a7 More explicit description of DEPTH_PRIORITY 2016-01-27 12:57:03 +01:00
Paul Tremberth
6ed08d2332 Add note for DEPTH_PRIORITY 2016-01-27 11:53:29 +01:00
Mikhail Korobov
4bcbb77bcc response.text. Fixes GH-1729. 2016-01-27 01:28:11 +05:00
Paul Tremberth
1c83108893 Clarify priority adjust settings docs 
Fixes #1593
 2016-01-26 19:24:11 +01:00
Daniel Graña
85f0596c43 Merge pull request #1725 from redapple/spiderstate-notconfigured 
[MRG+1] Disable SpiderState extension if no JOBDIR set
 2016-01-26 14:32:17 -03:00
Mikhail Korobov
6660175de9 Merge pull request #1726 from redapple/mwlist-logging-long 
Use long classes names for enabled middlewares in startup logs
 2016-01-26 21:29:44 +05:00
Paul Tremberth
bb1f4013a3 Rewrite warning about shell with local files as note 2016-01-26 17:23:28 +01:00
Paul Tremberth
c22a4e3bb8 Use long classes names for enabled middlewares in startup logs 2016-01-26 16:41:16 +01:00
Paul Tremberth
29695375d1 Add test for raised exception with SpiderState extension when no JOBDIR used 2016-01-26 16:33:24 +01:00
Paul Tremberth
0349bbf9d3 Disable SpiderState extension if no JOBDIR set 2016-01-26 15:25:15 +01:00
Paul Tremberth
0d368c5d6f Merge pull request #1724 from scrapy/robotstxt-default 
[MRG+1] Enable robots.txt handling by default for new projects.
 2016-01-26 14:19:15 +01:00
Mikhail Korobov
f30758c246 Enable robots.txt handling by default for new projects. Fixes GH-1668. 
For backwards compatibility reasons the default value is not changed.
 2016-01-26 17:47:46 +05:00
Elias Dorneles
2246280bb6 Merge pull request #1723 from redapple/closespider-notconfigured 
[MRG+1] Disable CloseSpider extension if no CLOSPIDER_* setting set
 2016-01-26 10:35:59 -02:00
Paul Tremberth
52daa1b953 Merge pull request #1720 from scrapy/deprecations 
[MRG+1] deprecated unused and untested code in scrapy.utils.datatypes
 2016-01-26 13:25:21 +01:00
Mikhail Korobov
1e48574576 Merge pull request #1722 from redapple/mwlist-logging 
[MRG+1] Fix logging of enabled middlewares
 2016-01-26 17:21:08 +05:00
Paul Tremberth
6ee8d8650a Disable CloseSpider extension if no CLOSPIDER_* setting set 2016-01-26 13:08:42 +01:00
Paul Tremberth
7608da8868 Fix logging of enabled middlewares 
Wrong middlewares list was being pretty-printed
(introduced in #1263)
 2016-01-26 13:01:12 +01:00
Elias Dorneles
de22b6f3ac Merge pull request #1721 from Digenis/offsitemw_subdomains 
[MRG+1] tests+doc for subdomains in offsite middleware
 2016-01-26 09:37:18 -02:00
Νικόλαος-Διγενής Καραγιάννης
1cffa99e0d tests+doc for subdomains in offsite middleware 2016-01-26 12:49:43 +02:00
Paul Tremberth
713e1eee9b Update docs about local files support for "scrapy shell" 2016-01-26 10:44:38 +01:00
Mikhail Korobov
9c2aa50ea2 deprecate unused and untested scrapy.utils.datatypes.MultiValueDict 2016-01-26 13:58:56 +05:00
Mikhail Korobov
7070dae48d deprecate unused and untested scrapy.utils.datatypes.SiteNode 2016-01-26 13:56:16 +05:00
Daniel Graña
a7b86137d0 Merge pull request #1499 from scrapy/py3-port-exporters 
[MRG+1] PY3 exporters
 2016-01-26 01:01:30 -03:00
Daniel Graña
d0eacfe0f9 Add test case for marshal item exporter 2016-01-26 00:26:27 -03:00
Elias Dorneles
2dfdde3c79 fallback to repr when can't convert to native string 2016-01-25 22:24:35 -02:00
Elias Dorneles
23b3336c1f add test for invalid option 2016-01-25 22:11:04 -02:00
Daniel Graña
b20463183c Merge pull request #1716 from lopuhin/py3-test-pipeline-files-images 
[MRG+1] py3: test_pipeline_files and test_pipeline_images
 2016-01-25 18:24:12 -03:00
Paul Tremberth
fb8ab2427b Move urlparsing statement in add_http_if_no_scheme() 2016-01-25 13:13:35 +01:00
Elias Dorneles
0c44fac2b5 added tests for feed export marshal and pickle 2016-01-24 19:17:42 -02:00
Elias Dorneles
4fef2c7628 Merge pull request #1717 from lopuhin/py3-test-spidermiddleware-httperror 
[MRG+1] py3: reviewed passing test_spidermiddleware_httperror.py
 2016-01-24 18:49:22 -02:00
Konstantin Lopuhin
1be90323c2 py3: properly skip s3 tests on py3 2016-01-24 23:44:56 +03:00
Konstantin Lopuhin
4233b3cda4 py3: reviewed passing test_spidermiddleware_httperror.py 2016-01-24 23:10:03 +03:00
Konstantin Lopuhin
097082cffa reviewed py3 compat in pipelines/images.py and pipelines/files.py 2016-01-24 23:05:23 +03:00
Konstantin Lopuhin
333d4c91fb py3: add boto to py3 test requirements, test_pipeline_files and test_pipeline_images passing now 2016-01-24 22:55:21 +03:00
Mikhail Korobov
85ae273230 Merge pull request #1715 from lopuhin/py3-test-mail 
[MRG+1] py3: fix test_mail
 2016-01-24 17:47:57 +05:00
Konstantin Lopuhin
860353b0c0 py3: unskip test_mail and scrapy/mail.py 2016-01-24 13:27:41 +03:00
Konstantin Lopuhin
9704226ee4 py3: fix test_mail - get_payload returns bytes when decode is True 2016-01-24 13:25:14 +03:00
Elias Dorneles
9fbe6f3e81 added feedexport test for xml output 2016-01-23 17:17:40 -02:00
Elias Dorneles
935b1da8c2 uses ScrapyDeprecationWarning instead of silenced PendingDeprecationWarning 2016-01-23 16:13:42 -02:00
Elias Dorneles
c75f1fe46a restore bytes instead of text, for easier reviewing 2016-01-23 16:09:57 -02:00