Pablo Hoffman
|
f0aea4aa4a
|
moved scrapy.newitem.exporters to scrapy.contrib.exporter
--HG--
rename : scrapy/newitem/exporters/__init__.py => scrapy/contrib/exporter/__init__.py
rename : scrapy/newitem/exporters/jsonexporter.py => scrapy/contrib/exporter/jsonexporter.py
rename : scrapy/tests/test_itemexporters.py => scrapy/tests/test_contrib_exporter.py
|
2009-08-12 21:52:15 -03:00 |
|
Pablo Hoffman
|
c99e572fdb
|
changed some variable names to avoid confusion
|
2009-08-12 21:51:29 -03:00 |
|
Pablo Hoffman
|
5aeab5b291
|
converted scrapy.item package to module
--HG--
rename : scrapy/item/models.py => scrapy/item.py
|
2009-08-12 21:31:50 -03:00 |
|
Pablo Hoffman
|
991afdf6b3
|
some minor fixes to loaders doc
|
2009-08-12 19:23:04 -03:00 |
|
Pablo Hoffman
|
6452e1934c
|
removed obsolete adaptors code
|
2009-08-12 19:09:02 -03:00 |
|
Pablo Hoffman
|
e6dd4d0955
|
renamed ApplyConcat processor to MapCompose
|
2009-08-12 18:43:08 -03:00 |
|
Pablo Hoffman
|
4e74f32401
|
renamed Pipe processor to Compose and documented it
|
2009-08-12 18:09:36 -03:00 |
|
Pablo Hoffman
|
958c3d9940
|
fixed some links to item loaders doc
|
2009-08-12 17:42:01 -03:00 |
|
Pablo Hoffman
|
d94b633565
|
renamed ItemLoader method populate_item() to load_item()
|
2009-08-12 17:40:27 -03:00 |
|
Ismael Carnales
|
77670a6b13
|
added Pipe parser
|
2009-08-12 17:23:53 -03:00 |
|
Pablo Hoffman
|
aace51f336
|
merge with ismael branch
|
2009-08-12 17:37:43 -03:00 |
|
Pablo Hoffman
|
cf566d6238
|
fixed bug with html meta refresh in multiple lines (thanks Molvo for the patch)
|
2009-08-12 16:56:30 -03:00 |
|
Pablo Hoffman
|
1dc592882b
|
Moved Item Loader to its final location in scrapy.contrib.loader, and updated doc/tests
--HG--
rename : docs/experimental/itemparser.rst => docs/experimental/loaders.rst
rename : scrapy/contrib/itemparser/__init__.py => scrapy/contrib/loader/__init__.py
rename : scrapy/contrib/itemparser/common.py => scrapy/contrib/loader/common.py
rename : scrapy/contrib/itemparser/parsers.py => scrapy/contrib/loader/processor.py
rename : scrapy/tests/test_itemparser.py => scrapy/tests/test_contrib_loader.py
|
2009-08-12 16:49:07 -03:00 |
|
Pablo Hoffman
|
7cbbc3ffb0
|
Renamed Loader to ItemParser (SEP-8 proposal). Documentation and unittests also updated.
--HG--
rename : docs/experimental/loaders.rst => docs/experimental/itemparser.rst
rename : scrapy/newitem/loader/__init__.py => scrapy/contrib/itemparser/__init__.py
rename : scrapy/newitem/loader/expanders.py => scrapy/contrib/itemparser/parsers.py
rename : scrapy/tests/test_itemloader.py => scrapy/tests/test_itemparser.py
|
2009-08-12 16:49:05 -03:00 |
|
Pablo Hoffman
|
d4aa72d2bb
|
removed obsolete documentation about Robust Scraped Item and Adaptors
|
2009-08-12 13:50:15 -03:00 |
|
Ismael Carnales
|
c83fb1c7d1
|
added newitem exporter tests and fixed exporter errors
|
2009-08-12 10:17:06 -03:00 |
|
Pablo Hoffman
|
bc4ec80db7
|
restored stats tests, and added some more for max_value/min_value attribute
|
2009-08-11 17:10:37 -03:00 |
|
Pablo Hoffman
|
a938e3c7e4
|
merge with ismael repo
|
2009-08-11 16:59:30 -03:00 |
|
Ismael Carnales
|
4dbeaf75e2
|
updated item exporters to new version of item, added JSONItemExporter
|
2009-08-11 16:55:15 -03:00 |
|
Pablo Hoffman
|
e69311ef38
|
added missing text to new stats collector methods
|
2009-08-11 16:37:52 -03:00 |
|
Daniel Grana
|
45cbe1333f
|
fix typo in stats docs
|
2009-08-11 16:30:04 -03:00 |
|
Daniel Grana
|
8f16e7f9fa
|
remove default parameter from max_value/min_value stats methods, update docs, and add new methods to dummy collector
|
2009-08-11 16:23:23 -03:00 |
|
Daniel Grana
|
9b620652fc
|
Automated merge with ssh://hg.scrapy.org/scrapy
|
2009-08-11 15:54:22 -03:00 |
|
Daniel Grana
|
02cb1e3432
|
colllect max itemproc_size and active_size in scraper per domain
|
2009-08-11 15:47:26 -03:00 |
|
Daniel Grana
|
cd8eed77c5
|
stats collector gains two new methods to store values only if greater/lower than previous value
|
2009-08-11 15:46:44 -03:00 |
|
Ismael Carnales
|
5fac909670
|
try to import json from python 2.6 or fallback to simplejson
|
2009-08-11 15:23:38 -03:00 |
|
Daniel Grana
|
9a531b552e
|
remove compiled pys before running tests
|
2009-08-11 15:11:31 -03:00 |
|
Daniel Grana
|
edb7150b27
|
Automated merge with ssh://hg.scrapy.org/scrapy
|
2009-08-11 12:39:05 -03:00 |
|
Daniel Grana
|
743d326184
|
returning None from process_response is not allowed, ignore the request using exception instead
|
2009-08-11 12:39:02 -03:00 |
|
Ismael Carnales
|
3614998053
|
fixed error in doc
|
2009-08-11 09:23:45 -03:00 |
|
Pablo Hoffman
|
10a527daca
|
removed unused scrapy.contrib.codecs module
|
2009-08-10 21:02:46 -03:00 |
|
Pablo Hoffman
|
7ac7597950
|
removed obsolete scrapy.contrib.cluster
|
2009-08-10 21:02:46 -03:00 |
|
Pablo Hoffman
|
a0e2086b1b
|
moved deprecated scrapy.item.adaptors to scrapy.contrib.item, and added deprecation warning
--HG--
rename : scrapy/item/adaptors.py => scrapy/contrib/item/adaptors.py
|
2009-08-10 21:02:46 -03:00 |
|
Pablo Hoffman
|
e96622166e
|
removed backwards compatibility support for importing link extractors from scrapy.link.extractors
--HG--
rename : scrapy/link/__init__.py => scrapy/link.py
|
2009-08-10 21:02:46 -03:00 |
|
Pablo Hoffman
|
c0532712f9
|
removed unnecesary response ResponseSoup extension, and replaced by a utils function with cache support
|
2009-08-10 21:02:24 -03:00 |
|
Pablo Hoffman
|
d334c035c0
|
removed unnecesary ResponseLibxml2 extension and moved libxml2 document caching functionality to Libxml2Document using weak references
|
2009-08-10 20:52:43 -03:00 |
|
Daniel Grana
|
21f2bb6797
|
Automated merge with ssh://hg.scrapy.org/scrapy
|
2009-08-10 20:28:37 -03:00 |
|
Daniel Grana
|
7078cdc336
|
remove unmantained web server code
|
2009-08-10 20:28:32 -03:00 |
|
Pablo Hoffman
|
49bab4777b
|
XPathSelector: added 're' argument to add_xpath method, exposed selector attribute
|
2009-08-10 19:42:20 -03:00 |
|
Pablo Hoffman
|
dfa4a4846c
|
removed unused module: scrapy.xpath.types
|
2009-08-10 10:23:42 -03:00 |
|
Pablo Hoffman
|
50fcc67278
|
improved reducers examples
|
2009-08-10 10:13:30 -03:00 |
|
Pablo Hoffman
|
5be0cee577
|
added TreeExpander example
|
2009-08-09 20:54:22 -03:00 |
|
Pablo Hoffman
|
38f82e3993
|
loaders doc: added information about expanders/reducers declaration precendece, and other minor improvements
|
2009-08-09 18:06:12 -03:00 |
|
Pablo Hoffman
|
b296d4169e
|
minor doc update for making it more windows-friendly
|
2009-08-09 17:08:42 -03:00 |
|
Pablo Hoffman
|
80b96fc195
|
minor changes to referer logging when crawling
|
2009-08-08 16:07:10 -03:00 |
|
Pablo Hoffman
|
b8eb08f444
|
additional cleanup to scrapy.xpath module
|
2009-08-08 15:29:50 -03:00 |
|
Pablo Hoffman
|
467dbef20e
|
fixed bug when no project module setting is defined
|
2009-08-08 15:12:40 -03:00 |
|
Pablo Hoffman
|
91eea82eef
|
added XPathLoader for working with XPath Selectors more conveniently
|
2009-08-08 07:26:20 -03:00 |
|
Pablo Hoffman
|
10cdc70fc4
|
some cleanup to scrapy.xpath module
--HG--
rename : scrapy/xpath/constructors.py => scrapy/xpath/factories.py
|
2009-08-08 06:03:46 -03:00 |
|
Pablo Hoffman
|
661aeb5cda
|
moved ItemPipelineManager from scrapy.item.pipeline to scrapy.contrib.pipeline
|
2009-08-08 05:01:18 -03:00 |
|