BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-26 10:23:41 +00:00
Code Releases Wiki Activity
1,524 Commits 39 Branches 119 Tags
Commit Graph

1524 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Pablo Hoffman
db8b29500f removed duplicated code from memdebug extension (already present in memusage extension) 2009-08-17 14:25:38 -03:00
Pablo Hoffman
b20837a028 added __slots__ to Request/Response/Headers objects, to reduce memory footprint 2009-08-17 13:01:33 -03:00
Pablo Hoffman
45ed662ee5 some cleanup to memusage and memdebug extensions 2009-08-17 09:41:02 -03:00
Pablo Hoffman
f1980a3d9f updated some RFC numbers 2009-08-15 20:54:34 -03:00
Pablo Hoffman
08424ee093 OffsiteMiddleware: isolate policy of urls belonging to spiders into a separate method 2009-08-15 20:44:11 -03:00
Pablo Hoffman
bc87350133 removed legacy comment, and wrapped some lines to 80 columns 2009-08-15 20:34:40 -03:00
Pablo Hoffman
08194186b4 improved docstring a encoding parameter of safe_url_string function. also added some unittests 2009-08-15 19:44:37 -03:00
Pablo Hoffman
2463842acb HttpErrorMiddleware: performance improvement and added support for 'handle_httpstatus_list' request meta key 2009-08-15 17:14:35 -03:00
Ismael Carnales
8c2f62ba9c added Item Exporters documentation 2009-08-14 09:16:29 -03:00
Daniel Grana
7fca2f26f6 Automated merge with ssh://hg.scrapy.org/scrapy 2009-08-14 01:34:35 -03:00
Daniel Grana
e0ccbed40d imported patch improve_download_troughput.patch 2009-08-14 01:33:36 -03:00
Pablo Hoffman
02b01c3d2b loaders doc: fixed outdated line 2009-08-13 23:24:24 -03:00
Daniel Grana
6a74b51371 remove dupes words in loaders doc, and unused import in example 
--HG--
extra : rebase_source : ea7886725af7a54bd1031cb28f92efbdfe921d9e
 2009-08-13 23:23:08 -03:00
Pablo Hoffman
5494237cbb upgraded bundled beautifulsoup to 3.0.7a 2009-08-13 22:30:07 -03:00
Pablo Hoffman
23d0c08c65 added missing module in previous commit 2009-08-13 22:29:13 -03:00
Pablo Hoffman
d6f4e382ae moved scrapy.utils.db module to scrapy.utils.mysql 
--HG--
rename : scrapy/utils/db.py => scrapy/utils/mysql.py
 2009-08-13 22:14:55 -03:00
Pablo Hoffman
8be62e2508 commented out line until we find a proper fix 2009-08-13 22:11:30 -03:00
Pablo Hoffman
d8722c8c34 cleaned up scrapy.utils.db module 2009-08-13 22:09:19 -03:00
Pablo Hoffman
bebc8b2027 removed obsolete scrapy.contrib.item module (RobustScrapedItem model) 2009-08-13 21:50:41 -03:00
Pablo Hoffman
1624981443 added tests for builtin loader processors 2009-08-13 15:33:20 -03:00
Ismael Carnales
066fd9fa84 renamed internal names of Item Loader 2009-08-13 13:32:02 -03:00
Ismael Carnales
ecca7f6478 fixes to Item Loader doc 2009-08-13 13:30:50 -03:00
Pablo Hoffman
8131d674d4 fixed outdated documentation (refs #97) 2009-08-13 09:24:41 -03:00
Pablo Hoffman
5d29c3427c converted scrapy.newitem package to module 
--HG--
rename : scrapy/newitem/__init__.py => scrapy/newitem.py
 2009-08-12 21:53:34 -03:00
Pablo Hoffman
f0aea4aa4a moved scrapy.newitem.exporters to scrapy.contrib.exporter 
--HG--
rename : scrapy/newitem/exporters/__init__.py => scrapy/contrib/exporter/__init__.py
rename : scrapy/newitem/exporters/jsonexporter.py => scrapy/contrib/exporter/jsonexporter.py
rename : scrapy/tests/test_itemexporters.py => scrapy/tests/test_contrib_exporter.py
 2009-08-12 21:52:15 -03:00
Pablo Hoffman
c99e572fdb changed some variable names to avoid confusion 2009-08-12 21:51:29 -03:00
Pablo Hoffman
5aeab5b291 converted scrapy.item package to module 
--HG--
rename : scrapy/item/models.py => scrapy/item.py
 2009-08-12 21:31:50 -03:00
Pablo Hoffman
991afdf6b3 some minor fixes to loaders doc 2009-08-12 19:23:04 -03:00
Pablo Hoffman
6452e1934c removed obsolete adaptors code 2009-08-12 19:09:02 -03:00
Pablo Hoffman
e6dd4d0955 renamed ApplyConcat processor to MapCompose 2009-08-12 18:43:08 -03:00
Pablo Hoffman
4e74f32401 renamed Pipe processor to Compose and documented it 2009-08-12 18:09:36 -03:00
Pablo Hoffman
958c3d9940 fixed some links to item loaders doc 2009-08-12 17:42:01 -03:00
Pablo Hoffman
d94b633565 renamed ItemLoader method populate_item() to load_item() 2009-08-12 17:40:27 -03:00
Ismael Carnales
77670a6b13 added Pipe parser 2009-08-12 17:23:53 -03:00
Pablo Hoffman
aace51f336 merge with ismael branch 2009-08-12 17:37:43 -03:00
Pablo Hoffman
cf566d6238 fixed bug with html meta refresh in multiple lines (thanks Molvo for the patch) 2009-08-12 16:56:30 -03:00
Pablo Hoffman
1dc592882b Moved Item Loader to its final location in scrapy.contrib.loader, and updated doc/tests 
--HG--
rename : docs/experimental/itemparser.rst => docs/experimental/loaders.rst
rename : scrapy/contrib/itemparser/__init__.py => scrapy/contrib/loader/__init__.py
rename : scrapy/contrib/itemparser/common.py => scrapy/contrib/loader/common.py
rename : scrapy/contrib/itemparser/parsers.py => scrapy/contrib/loader/processor.py
rename : scrapy/tests/test_itemparser.py => scrapy/tests/test_contrib_loader.py
 2009-08-12 16:49:07 -03:00
Pablo Hoffman
7cbbc3ffb0 Renamed Loader to ItemParser (SEP-8 proposal). Documentation and unittests also updated. 
--HG--
rename : docs/experimental/loaders.rst => docs/experimental/itemparser.rst
rename : scrapy/newitem/loader/__init__.py => scrapy/contrib/itemparser/__init__.py
rename : scrapy/newitem/loader/expanders.py => scrapy/contrib/itemparser/parsers.py
rename : scrapy/tests/test_itemloader.py => scrapy/tests/test_itemparser.py
 2009-08-12 16:49:05 -03:00
Pablo Hoffman
d4aa72d2bb removed obsolete documentation about Robust Scraped Item and Adaptors 2009-08-12 13:50:15 -03:00
Ismael Carnales
c83fb1c7d1 added newitem exporter tests and fixed exporter errors 2009-08-12 10:17:06 -03:00
Pablo Hoffman
bc4ec80db7 restored stats tests, and added some more for max_value/min_value attribute 2009-08-11 17:10:37 -03:00
Pablo Hoffman
a938e3c7e4 merge with ismael repo 2009-08-11 16:59:30 -03:00
Ismael Carnales
4dbeaf75e2 updated item exporters to new version of item, added JSONItemExporter 2009-08-11 16:55:15 -03:00
Pablo Hoffman
e69311ef38 added missing text to new stats collector methods 2009-08-11 16:37:52 -03:00
Daniel Grana
45cbe1333f fix typo in stats docs 2009-08-11 16:30:04 -03:00
Daniel Grana
8f16e7f9fa remove default parameter from max_value/min_value stats methods, update docs, and add new methods to dummy collector 2009-08-11 16:23:23 -03:00
Daniel Grana
9b620652fc Automated merge with ssh://hg.scrapy.org/scrapy 2009-08-11 15:54:22 -03:00
Daniel Grana
02cb1e3432 colllect max itemproc_size and active_size in scraper per domain 2009-08-11 15:47:26 -03:00
Daniel Grana
cd8eed77c5 stats collector gains two new methods to store values only if greater/lower than previous value 2009-08-11 15:46:44 -03:00
Ismael Carnales
5fac909670 try to import json from python 2.6 or fallback to simplejson 2009-08-11 15:23:38 -03:00