Daniel Grana
73e7788ba2
dont try to guess if spider output is iterable for Items and Requests objects
2009-08-24 01:25:14 -03:00
Daniel Grana
6e2bd49a81
Host header must include port number when port used for connecting is not default protocol port
2009-08-24 01:16:49 -03:00
Pablo Hoffman
46aaea9f32
doc: improved documentation about debugging leaks
2009-08-23 20:36:00 -03:00
Pablo Hoffman
0ed849248f
some improvements to item exporters
...
- passed previous class attributes to instances attributes
- better handling of constructor arguments
- better coverage on unittets (including encoding)
- updated documentation with new changes
2009-08-23 05:48:35 -03:00
Pablo Hoffman
de89909195
send_catch_log: pass through results from sendRobust
2009-08-22 16:38:25 -03:00
Pablo Hoffman
499dc6677c
utils.signal: made send_catch_log function more robust (by using pydispatch.robust.sendRobust) and added unittests
2009-08-22 16:22:31 -03:00
Pablo Hoffman
f204da82d1
disconnecting signal handlers after using them in stats unittests
2009-08-22 16:22:02 -03:00
Pablo Hoffman
825cec51ff
fixed bug recently introduced in stats collector closing logic, and added unittests
2009-08-21 23:23:46 -03:00
Pablo Hoffman
7461710ef1
added some missing dots
2009-08-21 21:54:10 -03:00
Pablo Hoffman
9635a7839c
rearranged documentation into a better organization
...
--HG--
rename : docs/topics/index.rst => docs/index.rst
2009-08-21 21:49:54 -03:00
Pablo Hoffman
4761e0c88d
minor doc correction
2009-08-21 19:11:59 -03:00
Pablo Hoffman
012fd1cc2f
moved api-stability.rst doc to root and updated it
...
--HG--
rename : docs/misc/api-stability.rst => docs/api-stability.rst
2009-08-21 16:29:23 -03:00
Pablo Hoffman
07c1f3b8d8
updated ugly argument name
2009-08-21 16:13:25 -03:00
Ismael Carnales
5050787380
removed spider templates from project, addeded sumcommands to manage templates in genspider
2009-08-21 16:10:45 -03:00
Pablo Hoffman
ef6f04eb06
moved doc about debugging memory leaks to its own topic and added doc about trackref module
2009-08-21 16:07:16 -03:00
Pablo Hoffman
4980eb4968
added titles to signals doc
2009-08-21 15:07:52 -03:00
Pablo Hoffman
daea3f8a0c
sphinx docs: replaced custom :exception: xref by standard :exc:
2009-08-21 15:05:06 -03:00
Ismael Carnales
e2b8817c33
updated project templates to new item
2009-08-21 14:21:22 -03:00
Ismael Carnales
c08d3aa9cc
updated tutorial to use new items api
2009-08-21 14:16:27 -03:00
Pablo Hoffman
37760eb13f
improved consistency of logging settings to use LOG_*
2009-08-21 08:54:12 -03:00
Ismael Carnales
f0689a9ee8
fixed error in link extractors doc, thanks tarasm
2009-08-21 08:34:53 -03:00
Daniel Grana
bdc0ec6863
remove undefined variable from image pipeline
2009-08-20 20:30:08 -03:00
Pablo Hoffman
e7e1cba4a8
updated some docstings
2009-08-20 18:39:04 -03:00
Pablo Hoffman
41f1972e47
removed unused TRACE log level and improved logging documentation
2009-08-20 18:17:48 -03:00
Pablo Hoffman
297a8ac76b
moved caching resolver to an extension in contrib.resolver
2009-08-20 17:37:46 -03:00
Pablo Hoffman
a767b7044e
removed old blocking caching DNS resolver and replaced by a non-blocking one installed as the default reactor resolver
...
--HG--
rename : scrapy/core/downloader/dnscache.py => scrapy/core/downloader/resolver.py
2009-08-20 17:11:12 -03:00
Pablo Hoffman
cbed0f1ea6
moved send_catch_log to new scrapy.utils.signal module
2009-08-20 16:02:40 -03:00
Pablo Hoffman
8fad0a5744
fixed bug with defer_fail rename
2009-08-20 15:33:35 -03:00
Pablo Hoffman
eeac040fce
minor docstring update
2009-08-20 14:40:37 -03:00
Pablo Hoffman
119bb11090
removed unused chain_deferred function, renamed defer_fail to defer_failed
2009-08-20 14:37:38 -03:00
Pablo Hoffman
b763350340
removed unused module: scrapy.contrib_exp.history
2009-08-20 14:29:39 -03:00
Pablo Hoffman
b65c87cb67
removed unused module: scrapy.utils.c14n
2009-08-20 14:23:49 -03:00
Pablo Hoffman
5de409a6a0
removed unused module: scrapy.tests.serialization
2009-08-20 14:20:45 -03:00
Pablo Hoffman
20e82335e2
rename some exporter methods and complete exporter tests refactoring
2009-08-20 14:09:24 -03:00
Ismael Carnales
6b7162f3ca
updated JsonLinesItemExporter to new exporters API
2009-08-20 12:58:10 -03:00
Pablo Hoffman
3152e66fec
deprecate domain_open signal and handle stats domain open/close directly from the engine
2009-08-20 10:54:58 -03:00
Pablo Hoffman
5522560d36
updated example project to use new selectors module
2009-08-20 10:25:43 -03:00
Pablo Hoffman
06db0bc124
updated some documentation references in source code
2009-08-19 22:41:26 -03:00
Pablo Hoffman
33b53c59d5
moved scrapy.xpath to scrapy.selector
...
--HG--
rename : scrapy/xpath/__init__.py => scrapy/selector/__init__.py
rename : scrapy/xpath/document.py => scrapy/selector/document.py
rename : scrapy/xpath/factories.py => scrapy/selector/factories.py
2009-08-19 21:50:52 -03:00
Pablo Hoffman
dcc90fc196
declared loaders api stable and updated example project to use them
...
--HG--
rename : docs/experimental/loaders.rst => docs/topics/loaders.rst
2009-08-19 21:39:59 -03:00
Pablo Hoffman
e8504a054c
moved scrapy.newitem to scrapy.item and declared newitem api officially stable. updated docs and example project. deprecated old ScrapedItem
2009-08-19 21:39:58 -03:00
Ismael Carnales
314e8dea43
added new item exporter tests, introduced some api changes
2009-08-19 19:05:33 -03:00
Pablo Hoffman
907fad6da8
make sure input processors always receive iterables as input
2009-08-19 16:49:49 -03:00
Pablo Hoffman
1d351c2866
minor change to offsite middleware regex, for clarity (doesn't change behaviour)
2009-08-19 16:16:34 -03:00
Pablo Hoffman
f3c6d83ad3
added check to CsvItemExporter
2009-08-19 15:20:29 -03:00
Pablo Hoffman
741a6d7863
item exporters refactoring
...
--HG--
rename : scrapy/contrib/exporter/jsonexporter.py => scrapy/contrib/exporter/jsonlines.py
2009-08-19 13:09:39 -03:00
Pablo Hoffman
9c5dad3f89
renamed scrapy.utils.ref module to scrapy.utils.trackref, and improved docstring
...
--HG--
rename : scrapy/utils/ref.py => scrapy/utils/trackref.py
2009-08-19 11:19:36 -03:00
Pablo Hoffman
e947e1d45b
added some unittests to make sure certain objects are using __slots__ and are also weak-referenceable
2009-08-18 20:40:04 -03:00
Pablo Hoffman
240e06813d
scarpy.xpath: added __weakref__ to __slots__, removed unused XPathSelector.response attribute, moved deprecated methods to the bottom
2009-08-18 20:00:54 -03:00
Pablo Hoffman
00527fc93c
added __slots__ to XPathSelector and Libxml2Document classes
2009-08-18 19:57:05 -03:00
