Pablo Hoffman
99467d4e6e
Changed (unstable) scheduler middleware API to receive spider (instead of domain) in enqueue_request method
2009-09-13 20:51:43 -03:00
Pablo Hoffman
921fc4f3bf
Big Scrapy core refactoring to pass around spider references instead of domains.
...
This is to avoid accessing the scrapy.spider.spiders singleton for "resolving"
spiders, which is considered an "evil" practice because it ties us to the
singleton model for the spider resolver, which is a bad thing.
This change will also work as the foundation for the API cleaning that we'll
perform for 0.8. We decided to introduce this change now to have a more common
basecode between 0.7 and 0.8, which will allow us to better support 0.7 until
0.8 is released.
However, this change doesn't modify the stable/documented API, nor does it
change the core logic. Those changes will land on the 0.8 branch, after 0.7 is
released.
--HG--
rename : scrapy/contrib/domainsch.py => scrapy/contrib/spiderscheduler.py
2009-09-12 14:34:18 -03:00
Pablo Hoffman
8d49dc2fb5
changed IMAGES_THUMBS setting to a dict instead of a list of tuples, and more improvements to images pipeline doc
2009-09-11 17:36:00 -03:00
Pablo Hoffman
e20f766792
fixed some typos
2009-09-11 16:55:37 -03:00
Pablo Hoffman
c2fe350f72
more changes to images pipeline doc
2009-09-11 16:53:36 -03:00
Ismael Carnales
ada46a2dbb
styled imagesp doc
2009-09-11 15:30:46 -03:00
Pablo Hoffman
be0f2beef0
more cleanup to scheduler middelware doc, and permanentely moved to experimental doc
2009-09-11 13:27:31 -03:00
Pablo Hoffman
0af052b68f
removed confusing title
2009-09-11 12:19:18 -03:00
Pablo Hoffman
f3240748cb
changed link to scheduler middleware doc, now in experimental
2009-09-11 12:03:23 -03:00
Ismael Carnales
3998a0cb58
added more scheduler middleware documentation, and moved it to experimental
...
--HG--
rename : docs/topics/scheduler-middleware.rst => docs/experimental/scheduler-middleware.rst
2009-09-11 11:58:53 -03:00
Pablo Hoffman
d242a20573
updated images pipeline doc
2009-09-11 11:47:12 -03:00
Pablo Hoffman
f1bb8dc2a3
first cleanup of spider manager api
...
- removed asdict() and reload() methods
- added list() method
- removed default spider
2009-09-10 19:06:46 -03:00
Pablo Hoffman
f85813cd94
added FAQ entry about scrapy recipes and community spiders
2009-09-10 18:32:50 -03:00
Pablo Hoffman
269724a2b7
added Debugger extension, removed StackTraceDump from extensions available by default
2009-09-08 22:32:17 -03:00
Ismael Carnales
4ddfa9a2a3
stlyed downloaded middleware doc
2009-09-07 12:18:57 -03:00
Ismael Carnales
e3df11e5bb
added module directive to spidermw documentation
2009-09-07 12:03:24 -03:00
Pablo Hoffman
827aa19c6e
removed obsolete scrapy.utils.db module
2009-09-04 17:38:14 -03:00
Pablo Hoffman
861a803cc3
removed obsolete RestrictMiddleware
2009-09-04 17:22:56 -03:00
Ismael Carnales
7e2587169b
added missing middleware docs
2009-09-04 12:39:02 -03:00
Pablo Hoffman
aefb94063a
more updates to spider middleware doc
2009-09-04 13:46:04 -03:00
Pablo Hoffman
d04640be5c
some improvements to spider middleware doc
2009-09-04 13:29:16 -03:00
Pablo Hoffman
96bb223c13
removed (pretty useless) DebugMiddleware
2009-09-04 12:59:58 -03:00
Pablo Hoffman
8a715701ec
fixed another doc typo
2009-09-03 14:31:00 -03:00
Ismael Carnales
3c1bb7bc40
fixed typo in djangoitems doc (thanks anibal)
2009-09-03 11:23:25 -03:00
Daniel Grana
0e7b2a6da5
write header line by default when using csv exporter
...
--HG--
extra : rebase_source : 2d2d7153dde5e3f77e682e16d2e4408f732f234e
2009-09-03 13:58:39 -03:00
Pablo Hoffman
596d2c4479
moved CoreStats extension to scrapy.contrib.corestats
...
--HG--
rename : scrapy/stats/corestats.py => scrapy/contrib/corestats.py
2009-09-01 23:00:49 -03:00
Pablo Hoffman
6a50af05d7
removed useless SpiderReloader extension
2009-09-01 22:49:15 -03:00
Pablo Hoffman
79851aefa6
moved SpiderProfiler extension to scrapy.contrib_exp and removed references from documentation
...
--HG--
rename : scrapy/contrib/spider/profiler.py => scrapy/contrib_exp/spiderprofiler.py
2009-09-01 22:38:37 -03:00
Pablo Hoffman
d3c51fd6f2
improved images pipeline documentation
2009-09-01 21:07:47 -03:00
Pablo Hoffman
18fd635124
another doc typo
2009-09-01 12:52:40 -03:00
Pablo Hoffman
538cc9803a
fixed doc typo
2009-09-01 12:47:53 -03:00
Pablo Hoffman
df0e1f005f
exporters doc: fixed example and some typos
2009-09-01 08:56:54 -03:00
Pablo Hoffman
ac8f46ce9e
added File Export Pipeline reference to Exporters doc
2009-08-31 21:01:35 -03:00
Pablo Hoffman
8d006e9ea1
moved item exporters doc to stable doc
...
--HG--
rename : docs/experimental/exporters.rst => docs/topics/exporters.rst
2009-08-31 20:47:12 -03:00
Pablo Hoffman
0b152c99b5
added File Export Pipeline, a wrapper to use Item Exporters as Item Pipelines
2009-08-31 20:40:41 -03:00
Pablo Hoffman
8fab524978
moved engine.getstatus() method to scrapy.utils.engine function, to leave reporting logic out of engine code. added est() shortcut to telnet console
2009-08-31 12:44:32 -03:00
Pablo Hoffman
884f0c878f
Stats collectin: fixed race condition between stats persistance and population of stats on domain close
2009-08-29 19:44:13 -03:00
Pablo Hoffman
895c70e036
doc: fixed some links to scrapy-ctl topic
2009-08-29 18:23:55 -03:00
Pablo Hoffman
64572124e0
added doc about SCRAPY_SETTINGS_MODULE
2009-08-29 18:20:13 -03:00
Pablo Hoffman
9dc7a3a373
some minor adjustments to logging doc
2009-08-29 18:10:28 -03:00
Pablo Hoffman
60cbf24c89
more cleanups to startproject and project templates
...
--HG--
rename : scrapy/templates/project/root/scrapy-ctl.py => scrapy/templates/project/scrapy-ctl.py
2009-08-29 04:29:47 -03:00
Pablo Hoffman
c87216f02b
doc: added missing :synopsis: to some modules
2009-08-29 03:46:58 -03:00
Pablo Hoffman
2f55d7932b
replaced :ref: by :doc: links in doc index
2009-08-29 03:37:59 -03:00
Pablo Hoffman
924219dd5a
- added reference documentation about scrapy-ctl.py script
...
- yet another refactor to cmdline module
- removed help command (use -h to help now)
--HG--
rename : docs/experimental/scripts.rst => docs/topics/scrapy-ctl.rst
2009-08-28 20:32:55 -03:00
Pablo Hoffman
609aed4553
added inspect_response() function for inspecting responses from spiders
2009-08-27 19:33:33 -03:00
Pablo Hoffman
ea4f16bae6
refactored scrapy shell implementation, dropping IPython dependency, and adding a new 'view' shortcut
2009-08-27 18:24:08 -03:00
Ismael Carnales
fb39bca24f
added djangoitem doc
2009-08-26 08:44:22 -03:00
Pablo Hoffman
65ab611fb0
more updates to HttpErrorMiddleware doc
2009-08-26 08:30:18 -03:00
Pablo Hoffman
44783a3a06
minor improvements to FAQ entry
2009-08-26 00:18:58 -03:00
Pablo Hoffman
4dba75a73f
updated HttpErrorMiddleware doc
2009-08-25 20:13:58 -03:00
