Daniel Grana
|
49cc1b1e7e
|
add missing future import for python 2.5
|
2009-08-17 21:42:37 -03:00 |
|
Pablo Hoffman
|
16c51353f2
|
updated select() method in crawl spider template
|
2009-08-17 21:22:05 -03:00 |
|
Pablo Hoffman
|
9ae3a1946d
|
remove Url class and use str instead for Request and Response urls. Also added urlparse_cached function for achieving the same caching functionality provided by old Url class
|
2009-08-17 21:16:55 -03:00 |
|
Pablo Hoffman
|
29710461f6
|
some refactoring to robotstxt downloader middleware
|
2009-08-17 19:11:43 -03:00 |
|
Pablo Hoffman
|
d6da7eb04f
|
removed backwards compatibility alias: load_class
|
2009-08-17 18:32:24 -03:00 |
|
Pablo Hoffman
|
e71764e707
|
removed unused functions: memoize, gzip_file
|
2009-08-17 18:30:40 -03:00 |
|
Pablo Hoffman
|
50de128e0d
|
removed unused items_to_csv function
|
2009-08-17 18:21:21 -03:00 |
|
Pablo Hoffman
|
6dfda357ef
|
removed unused dict_updatedefault function
|
2009-08-17 18:19:59 -03:00 |
|
Pablo Hoffman
|
c81987f624
|
removed unused hash_values function
|
2009-08-17 18:16:44 -03:00 |
|
Pablo Hoffman
|
4f06f6ef41
|
applied fix to deprecated decorator to warn only once (thanks Dan)
|
2009-08-17 18:13:45 -03:00 |
|
Pablo Hoffman
|
41036af643
|
some refactoring to genspider command
|
2009-08-17 17:59:34 -03:00 |
|
Ismael Carnales
|
48b40bd620
|
renamed x method of selectors to select
|
2009-08-17 15:58:06 -03:00 |
|
Pablo Hoffman
|
59e0a83ad4
|
removed more obsolete adaptors code
|
2009-08-17 14:48:11 -03:00 |
|
Pablo Hoffman
|
1bf5682144
|
removed unused modules
|
2009-08-17 14:41:06 -03:00 |
|
Pablo Hoffman
|
62c29e4d69
|
removed unused module
|
2009-08-17 14:39:54 -03:00 |
|
Pablo Hoffman
|
db8b29500f
|
removed duplicated code from memdebug extension (already present in memusage extension)
|
2009-08-17 14:25:38 -03:00 |
|
Pablo Hoffman
|
b20837a028
|
added __slots__ to Request/Response/Headers objects, to reduce memory footprint
|
2009-08-17 13:01:33 -03:00 |
|
Pablo Hoffman
|
45ed662ee5
|
some cleanup to memusage and memdebug extensions
|
2009-08-17 09:41:02 -03:00 |
|
Pablo Hoffman
|
f1980a3d9f
|
updated some RFC numbers
|
2009-08-15 20:54:34 -03:00 |
|
Pablo Hoffman
|
08424ee093
|
OffsiteMiddleware: isolate policy of urls belonging to spiders into a separate method
|
2009-08-15 20:44:11 -03:00 |
|
Pablo Hoffman
|
bc87350133
|
removed legacy comment, and wrapped some lines to 80 columns
|
2009-08-15 20:34:40 -03:00 |
|
Pablo Hoffman
|
08194186b4
|
improved docstring a encoding parameter of safe_url_string function. also added some unittests
|
2009-08-15 19:44:37 -03:00 |
|
Pablo Hoffman
|
2463842acb
|
HttpErrorMiddleware: performance improvement and added support for 'handle_httpstatus_list' request meta key
|
2009-08-15 17:14:35 -03:00 |
|
Ismael Carnales
|
8c2f62ba9c
|
added Item Exporters documentation
|
2009-08-14 09:16:29 -03:00 |
|
Daniel Grana
|
7fca2f26f6
|
Automated merge with ssh://hg.scrapy.org/scrapy
|
2009-08-14 01:34:35 -03:00 |
|
Daniel Grana
|
e0ccbed40d
|
imported patch improve_download_troughput.patch
|
2009-08-14 01:33:36 -03:00 |
|
Pablo Hoffman
|
02b01c3d2b
|
loaders doc: fixed outdated line
|
2009-08-13 23:24:24 -03:00 |
|
Daniel Grana
|
6a74b51371
|
remove dupes words in loaders doc, and unused import in example
--HG--
extra : rebase_source : ea7886725af7a54bd1031cb28f92efbdfe921d9e
|
2009-08-13 23:23:08 -03:00 |
|
Pablo Hoffman
|
5494237cbb
|
upgraded bundled beautifulsoup to 3.0.7a
|
2009-08-13 22:30:07 -03:00 |
|
Pablo Hoffman
|
23d0c08c65
|
added missing module in previous commit
|
2009-08-13 22:29:13 -03:00 |
|
Pablo Hoffman
|
d6f4e382ae
|
moved scrapy.utils.db module to scrapy.utils.mysql
--HG--
rename : scrapy/utils/db.py => scrapy/utils/mysql.py
|
2009-08-13 22:14:55 -03:00 |
|
Pablo Hoffman
|
8be62e2508
|
commented out line until we find a proper fix
|
2009-08-13 22:11:30 -03:00 |
|
Pablo Hoffman
|
d8722c8c34
|
cleaned up scrapy.utils.db module
|
2009-08-13 22:09:19 -03:00 |
|
Pablo Hoffman
|
bebc8b2027
|
removed obsolete scrapy.contrib.item module (RobustScrapedItem model)
|
2009-08-13 21:50:41 -03:00 |
|
Pablo Hoffman
|
1624981443
|
added tests for builtin loader processors
|
2009-08-13 15:33:20 -03:00 |
|
Ismael Carnales
|
066fd9fa84
|
renamed internal names of Item Loader
|
2009-08-13 13:32:02 -03:00 |
|
Ismael Carnales
|
ecca7f6478
|
fixes to Item Loader doc
|
2009-08-13 13:30:50 -03:00 |
|
Pablo Hoffman
|
8131d674d4
|
fixed outdated documentation (refs #97)
|
2009-08-13 09:24:41 -03:00 |
|
Pablo Hoffman
|
5d29c3427c
|
converted scrapy.newitem package to module
--HG--
rename : scrapy/newitem/__init__.py => scrapy/newitem.py
|
2009-08-12 21:53:34 -03:00 |
|
Pablo Hoffman
|
f0aea4aa4a
|
moved scrapy.newitem.exporters to scrapy.contrib.exporter
--HG--
rename : scrapy/newitem/exporters/__init__.py => scrapy/contrib/exporter/__init__.py
rename : scrapy/newitem/exporters/jsonexporter.py => scrapy/contrib/exporter/jsonexporter.py
rename : scrapy/tests/test_itemexporters.py => scrapy/tests/test_contrib_exporter.py
|
2009-08-12 21:52:15 -03:00 |
|
Pablo Hoffman
|
c99e572fdb
|
changed some variable names to avoid confusion
|
2009-08-12 21:51:29 -03:00 |
|
Pablo Hoffman
|
5aeab5b291
|
converted scrapy.item package to module
--HG--
rename : scrapy/item/models.py => scrapy/item.py
|
2009-08-12 21:31:50 -03:00 |
|
Pablo Hoffman
|
991afdf6b3
|
some minor fixes to loaders doc
|
2009-08-12 19:23:04 -03:00 |
|
Pablo Hoffman
|
6452e1934c
|
removed obsolete adaptors code
|
2009-08-12 19:09:02 -03:00 |
|
Pablo Hoffman
|
e6dd4d0955
|
renamed ApplyConcat processor to MapCompose
|
2009-08-12 18:43:08 -03:00 |
|
Pablo Hoffman
|
4e74f32401
|
renamed Pipe processor to Compose and documented it
|
2009-08-12 18:09:36 -03:00 |
|
Pablo Hoffman
|
958c3d9940
|
fixed some links to item loaders doc
|
2009-08-12 17:42:01 -03:00 |
|
Pablo Hoffman
|
d94b633565
|
renamed ItemLoader method populate_item() to load_item()
|
2009-08-12 17:40:27 -03:00 |
|
Ismael Carnales
|
77670a6b13
|
added Pipe parser
|
2009-08-12 17:23:53 -03:00 |
|
Pablo Hoffman
|
aace51f336
|
merge with ismael branch
|
2009-08-12 17:37:43 -03:00 |
|