BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-27 07:24:32 +00:00
Code Releases Wiki Activity
5,427 Commits 39 Branches 119 Tags
Commit Graph

5427 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Paul Tremberth
268e912273 Add pretty-printting of settings as dict if using IPython shell 
Suggested by @digenis
see http://ipython.readthedocs.org/en/stable/api/generated/IPython.lib.pretty.html?#extending
 2016-02-01 12:43:27 +01:00
Paul Tremberth
d843a0aae8 Amend "settings" command to output JSON for dict settings 2016-01-29 21:12:03 +01:00
Paul Tremberth
aa78758bc7 Update tests for settings copy_to_dict() 2016-01-29 18:59:12 +01:00
Paul Tremberth
a1ebff83d3 Remove __str__ and __repr__ from settings, introduce copy_to_dict() instead 
Settings instances as dict's are easier to print or pretty print in the shell

Fixes #1732
 2016-01-29 18:39:34 +01:00
Mikhail Korobov
a35aec71e9 Merge pull request #1710 from redapple/1550-shell_file-cont 
[MRG+1] shell command's ability to open local files + tests
 2016-01-29 21:03:40 +05:00
Paul Tremberth
78f00401cd Remove unused import in tests 2016-01-29 16:56:05 +01:00
Mikhail Korobov
505e87258c Merge pull request #1742 from stummjr/add-tests-to-exporters 
[MRG+1] Include tests for non-string items to Exporters
 2016-01-29 03:27:38 +05:00
stummjr
cf2ebb0687 Include tests for exporters: JSON, JSON-Lines, Pickle and Marshal. 2016-01-28 19:43:53 -02:00
Paul Tremberth
e9f6b98816 Amend guess_scheme() docstring 2016-01-28 16:26:45 +01:00
Paul Tremberth
c6f374f2eb Merge remote-tracking branch 'origin/master' into 1550-shell_file-cont 2016-01-28 14:02:48 +01:00
Paul Tremberth
481e251775 Move guess_scheme() tests to relevant test module 2016-01-28 13:51:50 +01:00
Paul Tremberth
cae268402d Move guess_scheme() to scrapy.utils.url 2016-01-28 13:42:04 +01:00
Mikhail Korobov
4c9a6ef3ce Merge pull request #1740 from scrapy/response-text-micro-optimize 
[MRG+1] micro-optimize response.text
 2016-01-28 01:39:48 +05:00
Mikhail Korobov
3e080c3c52 call .text from .body_as_unicode() and not the other way around 2016-01-28 00:59:27 +05:00
Mikhail Korobov
394b991e3a Merge pull request #1737 from stummjr/fix-py3-exporters-multiple-types 
[MRG+1] Fix PythonItemExporter and CSVExporter for non-string item types
 2016-01-28 00:46:00 +05:00
stummjr
27758f60ad Changes fallback for CSVItemExporter, avoiding to call to_native_str(str()). 2016-01-27 16:28:01 -02:00
Paul Tremberth
27fb200e45 Merge pull request #1727 from redapple/priority-adjust 
[MRG+1] Clarify priority adjust settings docs
 2016-01-27 18:52:39 +01:00
stummjr
c55ff110a3 Fix CSV exporter for non string Python types. 2016-01-27 15:43:17 -02:00
stummjr
f1d971a5c0 fix PythonItemExporter for non-string types 2016-01-27 14:34:46 -02:00
Mikhail Korobov
7d24df3738 Merge pull request #1681 from redapple/code-of-conduct 
[MRG+1] Add Code of Conduct Version 1.3.0 from http://contributor-covenant.org/
 2016-01-27 17:56:11 +05:00
Mikhail Korobov
7ca9ae1976 DOC typo fix 2016-01-27 17:54:28 +05:00
Elias Dorneles
77951095e0 Merge pull request #1730 from scrapy/response-text 
[MRG+1] response.text
 2016-01-27 10:45:28 -02:00
Paul Tremberth
e0f48c486e Add link to CoC mardown file on Github 2016-01-27 13:04:08 +01:00
Paul Tremberth
d999e3f7a7 More explicit description of DEPTH_PRIORITY 2016-01-27 12:57:03 +01:00
Paul Tremberth
6ed08d2332 Add note for DEPTH_PRIORITY 2016-01-27 11:53:29 +01:00
Mikhail Korobov
4bcbb77bcc response.text. Fixes GH-1729. 2016-01-27 01:28:11 +05:00
Paul Tremberth
1c83108893 Clarify priority adjust settings docs 
Fixes #1593
 2016-01-26 19:24:11 +01:00
Daniel Graña
85f0596c43 Merge pull request #1725 from redapple/spiderstate-notconfigured 
[MRG+1] Disable SpiderState extension if no JOBDIR set
 2016-01-26 14:32:17 -03:00
Mikhail Korobov
6660175de9 Merge pull request #1726 from redapple/mwlist-logging-long 
Use long classes names for enabled middlewares in startup logs
 2016-01-26 21:29:44 +05:00
Paul Tremberth
bb1f4013a3 Rewrite warning about shell with local files as note 2016-01-26 17:23:28 +01:00
Paul Tremberth
c22a4e3bb8 Use long classes names for enabled middlewares in startup logs 2016-01-26 16:41:16 +01:00
Paul Tremberth
29695375d1 Add test for raised exception with SpiderState extension when no JOBDIR used 2016-01-26 16:33:24 +01:00
Paul Tremberth
0349bbf9d3 Disable SpiderState extension if no JOBDIR set 2016-01-26 15:25:15 +01:00
Paul Tremberth
0d368c5d6f Merge pull request #1724 from scrapy/robotstxt-default 
[MRG+1] Enable robots.txt handling by default for new projects.
 2016-01-26 14:19:15 +01:00
Mikhail Korobov
f30758c246 Enable robots.txt handling by default for new projects. Fixes GH-1668. 
For backwards compatibility reasons the default value is not changed.
 2016-01-26 17:47:46 +05:00
Elias Dorneles
2246280bb6 Merge pull request #1723 from redapple/closespider-notconfigured 
[MRG+1] Disable CloseSpider extension if no CLOSPIDER_* setting set
 2016-01-26 10:35:59 -02:00
Paul Tremberth
52daa1b953 Merge pull request #1720 from scrapy/deprecations 
[MRG+1] deprecated unused and untested code in scrapy.utils.datatypes
 2016-01-26 13:25:21 +01:00
Mikhail Korobov
1e48574576 Merge pull request #1722 from redapple/mwlist-logging 
[MRG+1] Fix logging of enabled middlewares
 2016-01-26 17:21:08 +05:00
Paul Tremberth
6ee8d8650a Disable CloseSpider extension if no CLOSPIDER_* setting set 2016-01-26 13:08:42 +01:00
Paul Tremberth
7608da8868 Fix logging of enabled middlewares 
Wrong middlewares list was being pretty-printed
(introduced in #1263)
 2016-01-26 13:01:12 +01:00
Elias Dorneles
de22b6f3ac Merge pull request #1721 from Digenis/offsitemw_subdomains 
[MRG+1] tests+doc for subdomains in offsite middleware
 2016-01-26 09:37:18 -02:00
Νικόλαος-Διγενής Καραγιάννης
1cffa99e0d tests+doc for subdomains in offsite middleware 2016-01-26 12:49:43 +02:00
Paul Tremberth
713e1eee9b Update docs about local files support for "scrapy shell" 2016-01-26 10:44:38 +01:00
Mikhail Korobov
9c2aa50ea2 deprecate unused and untested scrapy.utils.datatypes.MultiValueDict 2016-01-26 13:58:56 +05:00
Mikhail Korobov
7070dae48d deprecate unused and untested scrapy.utils.datatypes.SiteNode 2016-01-26 13:56:16 +05:00
Daniel Graña
a7b86137d0 Merge pull request #1499 from scrapy/py3-port-exporters 
[MRG+1] PY3 exporters
 2016-01-26 01:01:30 -03:00
Daniel Graña
d0eacfe0f9 Add test case for marshal item exporter 2016-01-26 00:26:27 -03:00
Elias Dorneles
2dfdde3c79 fallback to repr when can't convert to native string 2016-01-25 22:24:35 -02:00
Elias Dorneles
23b3336c1f add test for invalid option 2016-01-25 22:11:04 -02:00
Daniel Graña
b20463183c Merge pull request #1716 from lopuhin/py3-test-pipeline-files-images 
[MRG+1] py3: test_pipeline_files and test_pipeline_images
 2016-01-25 18:24:12 -03:00