BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-26 15:04:37 +00:00
Code Releases Wiki Activity
1,460 Commits 39 Branches 119 Tags
Commit Graph

1460 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Pablo Hoffman
23d0c08c65 added missing module in previous commit 2009-08-13 22:29:13 -03:00
Pablo Hoffman
d6f4e382ae moved scrapy.utils.db module to scrapy.utils.mysql 
--HG--
rename : scrapy/utils/db.py => scrapy/utils/mysql.py
 2009-08-13 22:14:55 -03:00
Pablo Hoffman
8be62e2508 commented out line until we find a proper fix 2009-08-13 22:11:30 -03:00
Pablo Hoffman
d8722c8c34 cleaned up scrapy.utils.db module 2009-08-13 22:09:19 -03:00
Pablo Hoffman
bebc8b2027 removed obsolete scrapy.contrib.item module (RobustScrapedItem model) 2009-08-13 21:50:41 -03:00
Pablo Hoffman
1624981443 added tests for builtin loader processors 2009-08-13 15:33:20 -03:00
Ismael Carnales
066fd9fa84 renamed internal names of Item Loader 2009-08-13 13:32:02 -03:00
Ismael Carnales
ecca7f6478 fixes to Item Loader doc 2009-08-13 13:30:50 -03:00
Pablo Hoffman
8131d674d4 fixed outdated documentation (refs #97) 2009-08-13 09:24:41 -03:00
Pablo Hoffman
5d29c3427c converted scrapy.newitem package to module 
--HG--
rename : scrapy/newitem/__init__.py => scrapy/newitem.py
 2009-08-12 21:53:34 -03:00
Pablo Hoffman
f0aea4aa4a moved scrapy.newitem.exporters to scrapy.contrib.exporter 
--HG--
rename : scrapy/newitem/exporters/__init__.py => scrapy/contrib/exporter/__init__.py
rename : scrapy/newitem/exporters/jsonexporter.py => scrapy/contrib/exporter/jsonexporter.py
rename : scrapy/tests/test_itemexporters.py => scrapy/tests/test_contrib_exporter.py
 2009-08-12 21:52:15 -03:00
Pablo Hoffman
c99e572fdb changed some variable names to avoid confusion 2009-08-12 21:51:29 -03:00
Pablo Hoffman
5aeab5b291 converted scrapy.item package to module 
--HG--
rename : scrapy/item/models.py => scrapy/item.py
 2009-08-12 21:31:50 -03:00
Pablo Hoffman
991afdf6b3 some minor fixes to loaders doc 2009-08-12 19:23:04 -03:00
Pablo Hoffman
6452e1934c removed obsolete adaptors code 2009-08-12 19:09:02 -03:00
Pablo Hoffman
e6dd4d0955 renamed ApplyConcat processor to MapCompose 2009-08-12 18:43:08 -03:00
Pablo Hoffman
4e74f32401 renamed Pipe processor to Compose and documented it 2009-08-12 18:09:36 -03:00
Pablo Hoffman
958c3d9940 fixed some links to item loaders doc 2009-08-12 17:42:01 -03:00
Pablo Hoffman
d94b633565 renamed ItemLoader method populate_item() to load_item() 2009-08-12 17:40:27 -03:00
Ismael Carnales
77670a6b13 added Pipe parser 2009-08-12 17:23:53 -03:00
Pablo Hoffman
aace51f336 merge with ismael branch 2009-08-12 17:37:43 -03:00
Pablo Hoffman
cf566d6238 fixed bug with html meta refresh in multiple lines (thanks Molvo for the patch) 2009-08-12 16:56:30 -03:00
Pablo Hoffman
1dc592882b Moved Item Loader to its final location in scrapy.contrib.loader, and updated doc/tests 
--HG--
rename : docs/experimental/itemparser.rst => docs/experimental/loaders.rst
rename : scrapy/contrib/itemparser/__init__.py => scrapy/contrib/loader/__init__.py
rename : scrapy/contrib/itemparser/common.py => scrapy/contrib/loader/common.py
rename : scrapy/contrib/itemparser/parsers.py => scrapy/contrib/loader/processor.py
rename : scrapy/tests/test_itemparser.py => scrapy/tests/test_contrib_loader.py
 2009-08-12 16:49:07 -03:00
Pablo Hoffman
7cbbc3ffb0 Renamed Loader to ItemParser (SEP-8 proposal). Documentation and unittests also updated. 
--HG--
rename : docs/experimental/loaders.rst => docs/experimental/itemparser.rst
rename : scrapy/newitem/loader/__init__.py => scrapy/contrib/itemparser/__init__.py
rename : scrapy/newitem/loader/expanders.py => scrapy/contrib/itemparser/parsers.py
rename : scrapy/tests/test_itemloader.py => scrapy/tests/test_itemparser.py
 2009-08-12 16:49:05 -03:00
Pablo Hoffman
d4aa72d2bb removed obsolete documentation about Robust Scraped Item and Adaptors 2009-08-12 13:50:15 -03:00
Ismael Carnales
c83fb1c7d1 added newitem exporter tests and fixed exporter errors 2009-08-12 10:17:06 -03:00
Pablo Hoffman
bc4ec80db7 restored stats tests, and added some more for max_value/min_value attribute 2009-08-11 17:10:37 -03:00
Pablo Hoffman
a938e3c7e4 merge with ismael repo 2009-08-11 16:59:30 -03:00
Ismael Carnales
4dbeaf75e2 updated item exporters to new version of item, added JSONItemExporter 2009-08-11 16:55:15 -03:00
Pablo Hoffman
e69311ef38 added missing text to new stats collector methods 2009-08-11 16:37:52 -03:00
Daniel Grana
45cbe1333f fix typo in stats docs 2009-08-11 16:30:04 -03:00
Daniel Grana
8f16e7f9fa remove default parameter from max_value/min_value stats methods, update docs, and add new methods to dummy collector 2009-08-11 16:23:23 -03:00
Daniel Grana
9b620652fc Automated merge with ssh://hg.scrapy.org/scrapy 2009-08-11 15:54:22 -03:00
Daniel Grana
02cb1e3432 colllect max itemproc_size and active_size in scraper per domain 2009-08-11 15:47:26 -03:00
Daniel Grana
cd8eed77c5 stats collector gains two new methods to store values only if greater/lower than previous value 2009-08-11 15:46:44 -03:00
Ismael Carnales
5fac909670 try to import json from python 2.6 or fallback to simplejson 2009-08-11 15:23:38 -03:00
Daniel Grana
9a531b552e remove compiled pys before running tests 2009-08-11 15:11:31 -03:00
Daniel Grana
edb7150b27 Automated merge with ssh://hg.scrapy.org/scrapy 2009-08-11 12:39:05 -03:00
Daniel Grana
743d326184 returning None from process_response is not allowed, ignore the request using exception instead 2009-08-11 12:39:02 -03:00
Ismael Carnales
3614998053 fixed error in doc 2009-08-11 09:23:45 -03:00
Pablo Hoffman
10a527daca removed unused scrapy.contrib.codecs module 2009-08-10 21:02:46 -03:00
Pablo Hoffman
7ac7597950 removed obsolete scrapy.contrib.cluster 2009-08-10 21:02:46 -03:00
Pablo Hoffman
a0e2086b1b moved deprecated scrapy.item.adaptors to scrapy.contrib.item, and added deprecation warning 
--HG--
rename : scrapy/item/adaptors.py => scrapy/contrib/item/adaptors.py
 2009-08-10 21:02:46 -03:00
Pablo Hoffman
e96622166e removed backwards compatibility support for importing link extractors from scrapy.link.extractors 
--HG--
rename : scrapy/link/__init__.py => scrapy/link.py
 2009-08-10 21:02:46 -03:00
Pablo Hoffman
c0532712f9 removed unnecesary response ResponseSoup extension, and replaced by a utils function with cache support 2009-08-10 21:02:24 -03:00
Pablo Hoffman
d334c035c0 removed unnecesary ResponseLibxml2 extension and moved libxml2 document caching functionality to Libxml2Document using weak references 2009-08-10 20:52:43 -03:00
Daniel Grana
21f2bb6797 Automated merge with ssh://hg.scrapy.org/scrapy 2009-08-10 20:28:37 -03:00
Daniel Grana
7078cdc336 remove unmantained web server code 2009-08-10 20:28:32 -03:00
Pablo Hoffman
49bab4777b XPathSelector: added 're' argument to add_xpath method, exposed selector attribute 2009-08-10 19:42:20 -03:00
Pablo Hoffman
dfa4a4846c removed unused module: scrapy.xpath.types 2009-08-10 10:23:42 -03:00