Jordi Llonch
5b118ff4ab
added documentation (experimental feature)
2013-03-06 06:36:23 +11:00
Jordi Llonch
d9261f6c54
pluggable sub-commands for scrapy comand-line
2013-03-06 05:25:38 +11:00
Pablo Hoffman
19d0942c74
Merge branch 'shell' of git://github.com/stav/scrapy into stav-shell
2013-03-04 02:14:01 -02:00
Pablo Hoffman
3c8eef99cb
docs/contributing: added note explaining what Scrapy contrib is
2013-03-04 01:35:17 -02:00
Pablo Hoffman
7dd360f39f
Merge pull request #257 from stav/cleanups
...
doc: fix typo in spider middleware
2013-03-03 19:17:46 -08:00
Steven Almeroth
81111dd39a
fetch command should catch IgnoreRequest exception
2013-03-02 20:13:14 -06:00
Steven Almeroth
f62b6660d4
doc: fix typo in spider middleware
2013-03-02 19:46:31 -06:00
Pablo Hoffman
d5d944fa44
Log overriden Scrapy settings when Scrapy starts.
...
This is useful for debugging, to quickly find out which settings where
used for a specific spider run. dict settings are omitted for brevity.
DEBUG level is used for consistency with "Enabled
extensions/middlewares" lines (which share a similar purpose).
2013-02-28 11:35:11 -02:00
Pablo Hoffman
8f3a509d44
remove debugging code
2013-02-27 03:52:55 -02:00
Pablo Hoffman
3a6b80259d
arg_to_iter: replaced double call to isinstance with a single one. refs #248
2013-02-27 03:51:23 -02:00
Pablo Hoffman
2bbd92742b
arg_to_iter: treat items the same way as dicts (ie. non iterables). fixes #248
2013-02-27 02:39:31 -02:00
Pablo Hoffman
7400ceb1ed
added 502 to RETRY_HTTP_CODES
2013-02-22 19:12:59 -02:00
Pablo Hoffman
e3f50b97b2
made scrapy/item.py pep8 compliant
2013-02-20 11:38:50 -02:00
Pablo Hoffman
a038f46859
doc: fixed rst title
2013-02-14 11:11:17 -02:00
Pablo Hoffman
22edc44c6c
doc: remove links to diveintopython.org, which is no longer available. closes #246
2013-02-14 11:09:40 -02:00
Pablo Hoffman
aeb7fbe221
Merge pull request #240 from llonchj/spider_log
...
spider.log to pass keyword arguments into twisted log.msg
2013-02-12 18:13:26 -08:00
Pablo Hoffman
07669947c7
Merge pull request #244 from morty/master
...
scrapy genspider allows you to create a spider module with the same name as the project
2013-02-12 18:11:49 -08:00
Tom Mortimer-Jones
2efd859525
Added check so that genspider cannot create a spider with the same name as the project.
2013-02-12 12:26:38 +00:00
Pablo Hoffman
1ff8b4f831
updated release notes with previous commit
2013-02-12 00:59:25 -02:00
Rolando Espinoza La fuente
5971333a93
added --pdb command option to enable pdb debugger on failure.
2013-02-11 12:34:29 -04:00
Pablo Hoffman
d9043ffa6e
reverted previous changed as it broke tests
2013-02-11 11:43:25 -02:00
Pablo Hoffman
2df010a972
Merge pull request #241 from zuhao/master
...
Fix url_has_any_extension bug
2013-02-11 05:29:14 -08:00
Zuhao Wan
27ca25472f
Fix url_has_any_extension bug
2013-02-11 17:19:31 +08:00
Jordi Llonch
6d4b764f29
spider.log to pass keyword arguments into twisted log.msg
2013-02-11 05:00:06 +11:00
Daniel Graña
c0a7040f16
add PHONYs so build does not match build/ path
2013-02-08 15:07:00 -02:00
Daniel Graña
910effd145
get scrapy version from package data
2013-02-06 11:44:26 -02:00
Daniel Graña
79cb0318d0
update travis-ci conf
2013-02-06 10:36:08 -02:00
Daniel Graña
5db45b3825
remove scrapyd, it was migrated to its own repository
2013-02-06 05:24:07 +00:00
Daniel Graña
33ca295129
Merge pull request #235 from whodatninja/patch-1
...
Fix typo labeling attrs type bool instead of list
2013-02-05 17:27:10 -08:00
whodatninja
8e3b5baac5
Fix typo labeling attrs type bool instead of list
2013-02-05 15:10:41 -05:00
Daniel Graña
f293d08e7a
Merge pull request #234 from darkrho/httpcache-dbm-module-fullpath
...
allow to use full path in HTTPCACHE_DBM_MODULE setting
2013-02-01 09:01:27 -08:00
Rolando Espinoza La fuente
d00e6cf1d7
tests: dummydbm.error should be KeyError.
2013-02-01 12:52:59 -04:00
Rolando Espinoza La fuente
6ed44a3a67
httpcache: added tests for custom dbm module.
...
And removed the break in the line to improve readability as there
is no strict 80-chars line width convention.
2013-02-01 12:41:52 -04:00
Rolando Espinoza La fuente
a2602df90b
httpcache: allow to import submodules within packages as HTTPCACHE_DBM_MODULE setting.
2013-01-31 23:49:36 -04:00
Daniel Graña
e5edb8ec3e
Merge branch 'dangra/issue-24'
2013-01-30 16:55:53 -02:00
Daniel Graña
3af240f5f9
Merge branch 'dangra/issue-12'
2013-01-30 16:55:16 -02:00
Daniel Graña
0d3e4b4c43
do not unquote slash and question mark in url paths. fix #24
2013-01-30 11:22:03 -02:00
Daniel Graña
872a22df68
pep8ize sgml link extractors
2013-01-29 18:08:32 -02:00
Daniel Graña
cc69b3aa4c
Fix #199 encoding error concatenating link text
2013-01-29 17:56:34 -02:00
Daniel Graña
95fde0a498
register namespaces when XMLFeedSpider uses iternodes mode. fixes #12
2013-01-29 12:00:13 -02:00
Daniel Graña
8e77f27897
Find form nodes in invalid html5 documents
...
lxml fails to parse invalid html5 documents
This error was reported in scrapy/loginform#3
2013-01-24 17:47:53 -02:00
Daniel Graña
ff04480675
fix exporting nested items as xml. fixes #66
2013-01-24 14:57:53 -02:00
Pablo Hoffman
71d7df9273
simplify InitSpider implementation, fixing one bug ( closes #228 ) and adding support generators return from start_requests() (which the previous version didn't)
2013-01-23 16:28:41 -02:00
Daniel Graña
3cf7f4975b
Add 0.16.4 to release notes
...
Conflicts:
docs/news.rst
2013-01-23 11:29:38 -02:00
Daniel Graña
c40f947dc1
Merge pull request #229 from christilden/master
...
fixes spelling errors in documentation
2013-01-22 20:27:40 -08:00
Chris Tilden
aae6aed4fb
fixes spelling errors in documentation
2013-01-22 14:52:18 -08:00
Pablo Hoffman
27583922a7
remove unused imports
2013-01-21 14:01:12 -02:00
Pablo Hoffman
65258e3621
Merge pull request #227 from tonal/correct-old-cache
...
Correct init bag for load FilesystemCacheStorage from old location
2013-01-21 04:57:19 -08:00
Alexandr N Zamaraev (aka tonal)
2c51266a40
Correct init bag for load old scrapy.contrib.httpcache.FilesystemCacheStorage
2013-01-21 13:54:36 +07:00
Pablo Hoffman
6ab8afb992
improve documentation about removing namespaces
2013-01-18 12:35:30 -02:00
