Nicolás Ramírez
f043bba049
Override request method in FormRequest
...
Changes proposed in the comments
2013-03-14 10:39:17 -03:00
Pablo Hoffman
a2e9d031f9
removed duplicated test
2013-03-14 10:32:19 -03:00
Pablo Hoffman
8391b36251
minor updates to contributing doc
2013-03-13 03:24:25 -03:00
Pablo Hoffman
51c301b3a2
added link to python binary libs, for windows installation
2013-03-13 03:18:33 -03:00
Pablo Hoffman
8e72730792
Merge pull request #261 from stav/allowed_domains
...
allow spider allowed_domains to be set/tuple, #259
2013-03-12 20:44:51 -07:00
Pablo Hoffman
296db1dc09
log (just once) when duplicate requests are filtered out. closes #105 , #249
2013-03-12 19:28:43 -03:00
Pablo Hoffman
a4507f62c3
sep-019: fixed typo
2013-03-12 18:44:09 -03:00
Steven Almeroth
650eda68da
doc: add comment about commit history cleanliness
2013-03-10 18:51:04 -06:00
Steven Almeroth
5828179c5f
remove over-testing and dict testing for test_utils_url.py
2013-03-10 18:30:49 -06:00
Steven Almeroth
1514b3b5db
pylint clean-ups for test_utils_url.py
2013-03-10 16:47:24 -06:00
Steven Almeroth
a613e15154
add tests for url_is_from_spider() with allowed_domains
2013-03-10 16:41:27 -06:00
Pablo Hoffman
b2b256dc37
added nicolas to sep-019 authors, changed main title formatting for sep-019 and sep-020
2013-03-10 14:22:08 -03:00
Pablo Hoffman
b11a1326b9
Merge pull request #264 from stav/sep
...
add new sep-20 and update sep-19 with table at the top
2013-03-08 11:23:15 -08:00
Steven Almeroth
e66db6fa56
add new sep-20 and update sep-19 with table at the top
2013-03-08 13:04:11 -06:00
Pablo Hoffman
eeb69d2f70
added #260 to release notes
2013-03-08 11:59:38 -02:00
Pablo Hoffman
42499c72b1
updated AUTHORS
2013-03-08 11:57:54 -02:00
Pablo Hoffman
b5b37197bf
Merge pull request #260 from llonchj/entry_point
...
Commands module
2013-03-08 05:49:08 -08:00
Pablo Hoffman
b560f4d1c1
sep-019: remove TODO section (settled down with class method)
2013-03-08 09:57:12 -02:00
Pablo Hoffman
a44757179b
sep-019: other minor fixes
2013-03-07 19:38:12 -02:00
Pablo Hoffman
c7add1e5bd
sep-019: fix typos
2013-03-07 19:24:44 -02:00
Pablo Hoffman
d806acdf50
sep-019: minor tide up
2013-03-07 12:39:40 -02:00
Pablo Hoffman
6f24b46115
added SEP-019 (per-spider settings) - first draft
2013-03-07 12:34:17 -02:00
Pablo Hoffman
a027da0a2c
Some changes to scrapy deploy command:
...
- use branch names when constructing HG/GIT versions
- added -d flag for debugging (keeps build directory)
- improved command output (redirected setup.py stderr)
2013-03-06 11:41:34 -02:00
Steven Almeroth
b48ec1dce4
allow spider allowed_domains to be set/tuple, #259
2013-03-06 00:19:47 -06:00
Jordi Llonch
5b118ff4ab
added documentation (experimental feature)
2013-03-06 06:36:23 +11:00
Jordi Llonch
d9261f6c54
pluggable sub-commands for scrapy comand-line
2013-03-06 05:25:38 +11:00
Pablo Hoffman
19d0942c74
Merge branch 'shell' of git://github.com/stav/scrapy into stav-shell
2013-03-04 02:14:01 -02:00
Pablo Hoffman
3c8eef99cb
docs/contributing: added note explaining what Scrapy contrib is
2013-03-04 01:35:17 -02:00
Pablo Hoffman
7dd360f39f
Merge pull request #257 from stav/cleanups
...
doc: fix typo in spider middleware
2013-03-03 19:17:46 -08:00
Steven Almeroth
81111dd39a
fetch command should catch IgnoreRequest exception
2013-03-02 20:13:14 -06:00
Steven Almeroth
f62b6660d4
doc: fix typo in spider middleware
2013-03-02 19:46:31 -06:00
Pablo Hoffman
d5d944fa44
Log overriden Scrapy settings when Scrapy starts.
...
This is useful for debugging, to quickly find out which settings where
used for a specific spider run. dict settings are omitted for brevity.
DEBUG level is used for consistency with "Enabled
extensions/middlewares" lines (which share a similar purpose).
2013-02-28 11:35:11 -02:00
Pablo Hoffman
8f3a509d44
remove debugging code
2013-02-27 03:52:55 -02:00
Pablo Hoffman
3a6b80259d
arg_to_iter: replaced double call to isinstance with a single one. refs #248
2013-02-27 03:51:23 -02:00
Pablo Hoffman
2bbd92742b
arg_to_iter: treat items the same way as dicts (ie. non iterables). fixes #248
2013-02-27 02:39:31 -02:00
Pablo Hoffman
7400ceb1ed
added 502 to RETRY_HTTP_CODES
2013-02-22 19:12:59 -02:00
Pablo Hoffman
e3f50b97b2
made scrapy/item.py pep8 compliant
2013-02-20 11:38:50 -02:00
Pablo Hoffman
a038f46859
doc: fixed rst title
2013-02-14 11:11:17 -02:00
Pablo Hoffman
22edc44c6c
doc: remove links to diveintopython.org, which is no longer available. closes #246
2013-02-14 11:09:40 -02:00
Pablo Hoffman
aeb7fbe221
Merge pull request #240 from llonchj/spider_log
...
spider.log to pass keyword arguments into twisted log.msg
2013-02-12 18:13:26 -08:00
Pablo Hoffman
07669947c7
Merge pull request #244 from morty/master
...
scrapy genspider allows you to create a spider module with the same name as the project
2013-02-12 18:11:49 -08:00
Tom Mortimer-Jones
2efd859525
Added check so that genspider cannot create a spider with the same name as the project.
2013-02-12 12:26:38 +00:00
Pablo Hoffman
1ff8b4f831
updated release notes with previous commit
2013-02-12 00:59:25 -02:00
Rolando Espinoza La fuente
5971333a93
added --pdb command option to enable pdb debugger on failure.
2013-02-11 12:34:29 -04:00
Pablo Hoffman
d9043ffa6e
reverted previous changed as it broke tests
2013-02-11 11:43:25 -02:00
Pablo Hoffman
2df010a972
Merge pull request #241 from zuhao/master
...
Fix url_has_any_extension bug
2013-02-11 05:29:14 -08:00
Zuhao Wan
27ca25472f
Fix url_has_any_extension bug
2013-02-11 17:19:31 +08:00
Jordi Llonch
6d4b764f29
spider.log to pass keyword arguments into twisted log.msg
2013-02-11 05:00:06 +11:00
Daniel Graña
c0a7040f16
add PHONYs so build does not match build/ path
2013-02-08 15:07:00 -02:00
Daniel Graña
910effd145
get scrapy version from package data
2013-02-06 11:44:26 -02:00
