BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-25 04:04:21 +00:00
Code Releases Wiki Activity
5,384 Commits 39 Branches 119 Tags
Commit Graph

5384 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Mikhail Korobov
7ca9ae1976 DOC typo fix 2016-01-27 17:54:28 +05:00
Elias Dorneles
77951095e0 Merge pull request #1730 from scrapy/response-text 
[MRG+1] response.text
 2016-01-27 10:45:28 -02:00
Mikhail Korobov
4bcbb77bcc response.text. Fixes GH-1729. 2016-01-27 01:28:11 +05:00
Daniel Graña
85f0596c43 Merge pull request #1725 from redapple/spiderstate-notconfigured 
[MRG+1] Disable SpiderState extension if no JOBDIR set
 2016-01-26 14:32:17 -03:00
Mikhail Korobov
6660175de9 Merge pull request #1726 from redapple/mwlist-logging-long 
Use long classes names for enabled middlewares in startup logs
 2016-01-26 21:29:44 +05:00
Paul Tremberth
c22a4e3bb8 Use long classes names for enabled middlewares in startup logs 2016-01-26 16:41:16 +01:00
Paul Tremberth
29695375d1 Add test for raised exception with SpiderState extension when no JOBDIR used 2016-01-26 16:33:24 +01:00
Paul Tremberth
0349bbf9d3 Disable SpiderState extension if no JOBDIR set 2016-01-26 15:25:15 +01:00
Paul Tremberth
0d368c5d6f Merge pull request #1724 from scrapy/robotstxt-default 
[MRG+1] Enable robots.txt handling by default for new projects.
 2016-01-26 14:19:15 +01:00
Mikhail Korobov
f30758c246 Enable robots.txt handling by default for new projects. Fixes GH-1668. 
For backwards compatibility reasons the default value is not changed.
 2016-01-26 17:47:46 +05:00
Elias Dorneles
2246280bb6 Merge pull request #1723 from redapple/closespider-notconfigured 
[MRG+1] Disable CloseSpider extension if no CLOSPIDER_* setting set
 2016-01-26 10:35:59 -02:00
Paul Tremberth
52daa1b953 Merge pull request #1720 from scrapy/deprecations 
[MRG+1] deprecated unused and untested code in scrapy.utils.datatypes
 2016-01-26 13:25:21 +01:00
Mikhail Korobov
1e48574576 Merge pull request #1722 from redapple/mwlist-logging 
[MRG+1] Fix logging of enabled middlewares
 2016-01-26 17:21:08 +05:00
Paul Tremberth
6ee8d8650a Disable CloseSpider extension if no CLOSPIDER_* setting set 2016-01-26 13:08:42 +01:00
Paul Tremberth
7608da8868 Fix logging of enabled middlewares 
Wrong middlewares list was being pretty-printed
(introduced in #1263)
 2016-01-26 13:01:12 +01:00
Elias Dorneles
de22b6f3ac Merge pull request #1721 from Digenis/offsitemw_subdomains 
[MRG+1] tests+doc for subdomains in offsite middleware
 2016-01-26 09:37:18 -02:00
Νικόλαος-Διγενής Καραγιάννης
1cffa99e0d tests+doc for subdomains in offsite middleware 2016-01-26 12:49:43 +02:00
Mikhail Korobov
9c2aa50ea2 deprecate unused and untested scrapy.utils.datatypes.MultiValueDict 2016-01-26 13:58:56 +05:00
Mikhail Korobov
7070dae48d deprecate unused and untested scrapy.utils.datatypes.SiteNode 2016-01-26 13:56:16 +05:00
Daniel Graña
a7b86137d0 Merge pull request #1499 from scrapy/py3-port-exporters 
[MRG+1] PY3 exporters
 2016-01-26 01:01:30 -03:00
Daniel Graña
d0eacfe0f9 Add test case for marshal item exporter 2016-01-26 00:26:27 -03:00
Elias Dorneles
2dfdde3c79 fallback to repr when can't convert to native string 2016-01-25 22:24:35 -02:00
Elias Dorneles
23b3336c1f add test for invalid option 2016-01-25 22:11:04 -02:00
Daniel Graña
b20463183c Merge pull request #1716 from lopuhin/py3-test-pipeline-files-images 
[MRG+1] py3: test_pipeline_files and test_pipeline_images
 2016-01-25 18:24:12 -03:00
Elias Dorneles
0c44fac2b5 added tests for feed export marshal and pickle 2016-01-24 19:17:42 -02:00
Elias Dorneles
4fef2c7628 Merge pull request #1717 from lopuhin/py3-test-spidermiddleware-httperror 
[MRG+1] py3: reviewed passing test_spidermiddleware_httperror.py
 2016-01-24 18:49:22 -02:00
Konstantin Lopuhin
1be90323c2 py3: properly skip s3 tests on py3 2016-01-24 23:44:56 +03:00
Konstantin Lopuhin
4233b3cda4 py3: reviewed passing test_spidermiddleware_httperror.py 2016-01-24 23:10:03 +03:00
Konstantin Lopuhin
097082cffa reviewed py3 compat in pipelines/images.py and pipelines/files.py 2016-01-24 23:05:23 +03:00
Konstantin Lopuhin
333d4c91fb py3: add boto to py3 test requirements, test_pipeline_files and test_pipeline_images passing now 2016-01-24 22:55:21 +03:00
Mikhail Korobov
85ae273230 Merge pull request #1715 from lopuhin/py3-test-mail 
[MRG+1] py3: fix test_mail
 2016-01-24 17:47:57 +05:00
Konstantin Lopuhin
860353b0c0 py3: unskip test_mail and scrapy/mail.py 2016-01-24 13:27:41 +03:00
Konstantin Lopuhin
9704226ee4 py3: fix test_mail - get_payload returns bytes when decode is True 2016-01-24 13:25:14 +03:00
Elias Dorneles
9fbe6f3e81 added feedexport test for xml output 2016-01-23 17:17:40 -02:00
Elias Dorneles
935b1da8c2 uses ScrapyDeprecationWarning instead of silenced PendingDeprecationWarning 2016-01-23 16:13:42 -02:00
Elias Dorneles
c75f1fe46a restore bytes instead of text, for easier reviewing 2016-01-23 16:09:57 -02:00
Elias Dorneles
d0955fd083 add back test for latin-1 encoding 2016-01-22 10:07:55 -02:00
Mikhail Korobov
c75f7ed20b Merge pull request #1711 from scrapy/py3-parse-command-tests 
PY3 enable tests for scrapy parse command
 2016-01-22 15:21:26 +05:00
Mikhail Korobov
35ada10729 PY3 enable tests for scrapy parse command 
scrapy parse command is already ported
 2016-01-22 13:39:27 +05:00
Elias Dorneles
e938752973 add test for PythonItemExporter binary mode 2016-01-21 21:51:59 -02:00
Elias Dorneles
2514973242 re-enable skipped feed export tests 2016-01-21 21:22:12 -02:00
Elias Dorneles
b746d85f4c PY3 port csv exporter 2016-01-21 21:12:43 -02:00
Elias Dorneles
9f35c28643 fix indentation 2016-01-21 18:43:36 -02:00
Elias Dorneles
fed7c8b4fc fix: use is_listlike 2016-01-21 18:39:59 -02:00
Elias Dorneles
c76190d491 PY3: ported json(lines), xml exporters 2016-01-21 18:24:06 -02:00
Elias Dorneles
b6ef1f19fd make BaseItemExporter export unicode, pushed down previous behavior for classes that need it 2016-01-21 15:36:14 -02:00
Elias Dorneles
a76ecd4ef0 remove test_exporters from py3 ignores 2016-01-21 15:36:14 -02:00
Elias Dorneles
e0074d7f4d Merge pull request #1692 from lopuhin/py3-test-crawl 
[MRG+1] Py3: port test crawl
 2016-01-21 13:05:20 -02:00
Konstantin Lopuhin
5813de8838 py3: unskip test_closespider - it passes after fixing mockserver.Follow resouce on py3 2016-01-21 17:53:08 +03:00
Konstantin Lopuhin
4607f2843e py3: unskip test_crawl 2016-01-21 17:53:08 +03:00