924 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Pablo Hoffman	150f4ed815	scrapy cluster: added missing docstrings to important methods, fixed some bugs ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40324	2008-10-22 14:43:34 +00:00
Pablo Hoffman	5349030b31	creating cluster-refactor branch to fix several deficiences that the current ... cluster code has --HG-- rename : scrapy/trunk/INSTALL => scrapy/branches/cluster-refactor/INSTALL rename : scrapy/trunk/README => scrapy/branches/cluster-refactor/README rename : scrapy/trunk/docs/scrapy-architecture.dia => scrapy/branches/cluster-refactor/docs/scrapy-architecture.dia rename : scrapy/trunk/extras/sql/scraping.sql => scrapy/branches/cluster-refactor/extras/sql/scraping.sql rename : scrapy/trunk/scrapy/__init__.py => scrapy/branches/cluster-refactor/scrapy/__init__.py rename : scrapy/trunk/scrapy/bin/scrapy-admin.py => scrapy/branches/cluster-refactor/scrapy/bin/scrapy-admin.py rename : scrapy/trunk/scrapy/command/__init__.py => scrapy/branches/cluster-refactor/scrapy/command/__init__.py rename : scrapy/trunk/scrapy/command/cmdline.py => scrapy/branches/cluster-refactor/scrapy/command/cmdline.py rename : scrapy/trunk/scrapy/command/commands/__init__.py => scrapy/branches/cluster-refactor/scrapy/command/commands/__init__.py rename : scrapy/trunk/scrapy/command/commands/crawl.py => scrapy/branches/cluster-refactor/scrapy/command/commands/crawl.py rename : scrapy/trunk/scrapy/command/commands/download.py => scrapy/branches/cluster-refactor/scrapy/command/commands/download.py rename : scrapy/trunk/scrapy/command/commands/genspider.py => scrapy/branches/cluster-refactor/scrapy/command/commands/genspider.py rename : scrapy/trunk/scrapy/command/commands/getattr.py => scrapy/branches/cluster-refactor/scrapy/command/commands/getattr.py rename : scrapy/trunk/scrapy/command/commands/help.py => scrapy/branches/cluster-refactor/scrapy/command/commands/help.py rename : scrapy/trunk/scrapy/command/commands/list.py => scrapy/branches/cluster-refactor/scrapy/command/commands/list.py rename : scrapy/trunk/scrapy/command/commands/log.py => scrapy/branches/cluster-refactor/scrapy/command/commands/log.py rename : scrapy/trunk/scrapy/command/commands/parse.py => scrapy/branches/cluster-refactor/scrapy/command/commands/parse.py rename : scrapy/trunk/scrapy/command/commands/replay.py => scrapy/branches/cluster-refactor/scrapy/command/commands/replay.py rename : scrapy/trunk/scrapy/command/commands/shell.py => scrapy/branches/cluster-refactor/scrapy/command/commands/shell.py rename : scrapy/trunk/scrapy/command/commands/start.py => scrapy/branches/cluster-refactor/scrapy/command/commands/start.py rename : scrapy/trunk/scrapy/command/commands/stats.py => scrapy/branches/cluster-refactor/scrapy/command/commands/stats.py rename : scrapy/trunk/scrapy/command/models.py => scrapy/branches/cluster-refactor/scrapy/command/models.py rename : scrapy/trunk/scrapy/conf/__init__.py => scrapy/branches/cluster-refactor/scrapy/conf/__init__.py rename : scrapy/trunk/scrapy/conf/commands/__init__.py => scrapy/branches/cluster-refactor/scrapy/conf/commands/__init__.py rename : scrapy/trunk/scrapy/conf/commands/crawl.py => scrapy/branches/cluster-refactor/scrapy/conf/commands/crawl.py rename : scrapy/trunk/scrapy/conf/commands/help.py => scrapy/branches/cluster-refactor/scrapy/conf/commands/help.py rename : scrapy/trunk/scrapy/conf/commands/list.py => scrapy/branches/cluster-refactor/scrapy/conf/commands/list.py rename : scrapy/trunk/scrapy/conf/commands/log.py => scrapy/branches/cluster-refactor/scrapy/conf/commands/log.py rename : scrapy/trunk/scrapy/conf/commands/scrape.py => scrapy/branches/cluster-refactor/scrapy/conf/commands/scrape.py rename : scrapy/trunk/scrapy/conf/commands/stats.py => scrapy/branches/cluster-refactor/scrapy/conf/commands/stats.py rename : scrapy/trunk/scrapy/conf/commands/test.py => scrapy/branches/cluster-refactor/scrapy/conf/commands/test.py rename : scrapy/trunk/scrapy/conf/core_settings.py => scrapy/branches/cluster-refactor/scrapy/conf/core_settings.py rename : scrapy/trunk/scrapy/conf/project_template/__init__.py => scrapy/branches/cluster-refactor/scrapy/conf/project_template/__init__.py rename : scrapy/trunk/scrapy/conf/project_template/items.py => scrapy/branches/cluster-refactor/scrapy/conf/project_template/items.py rename : scrapy/trunk/scrapy/conf/project_template/scrapy-ctl.py => scrapy/branches/cluster-refactor/scrapy/conf/project_template/scrapy-ctl.py rename : scrapy/trunk/scrapy/conf/project_template/scrapy_settings.py => scrapy/branches/cluster-refactor/scrapy/conf/project_template/scrapy_settings.py rename : scrapy/trunk/scrapy/conf/project_template/spiders/__init__.py => scrapy/branches/cluster-refactor/scrapy/conf/project_template/spiders/__init__.py rename : scrapy/trunk/scrapy/contrib/__init__.py => scrapy/branches/cluster-refactor/scrapy/contrib/__init__.py rename : scrapy/trunk/scrapy/contrib/adaptorpipeline.py => scrapy/branches/cluster-refactor/scrapy/contrib/adaptorpipeline.py rename : scrapy/trunk/scrapy/contrib/closedomain.py => scrapy/branches/cluster-refactor/scrapy/contrib/closedomain.py rename : scrapy/trunk/scrapy/contrib/debug.py => scrapy/branches/cluster-refactor/scrapy/contrib/debug.py rename : scrapy/trunk/scrapy/contrib/downloadermiddleware/__init__.py => scrapy/branches/cluster-refactor/scrapy/contrib/downloadermiddleware/__init__.py rename : scrapy/trunk/scrapy/contrib/downloadermiddleware/cache.py => scrapy/branches/cluster-refactor/scrapy/contrib/downloadermiddleware/cache.py rename : scrapy/trunk/scrapy/contrib/downloadermiddleware/common.py => scrapy/branches/cluster-refactor/scrapy/contrib/downloadermiddleware/common.py rename : scrapy/trunk/scrapy/contrib/downloadermiddleware/compression.py => scrapy/branches/cluster-refactor/scrapy/contrib/downloadermiddleware/compression.py rename : scrapy/trunk/scrapy/contrib/downloadermiddleware/cookies.py => scrapy/branches/cluster-refactor/scrapy/contrib/downloadermiddleware/cookies.py rename : scrapy/trunk/scrapy/contrib/downloadermiddleware/debug.py => scrapy/branches/cluster-refactor/scrapy/contrib/downloadermiddleware/debug.py rename : scrapy/trunk/scrapy/contrib/downloadermiddleware/errorpages.py => scrapy/branches/cluster-refactor/scrapy/contrib/downloadermiddleware/errorpages.py rename : scrapy/trunk/scrapy/contrib/downloadermiddleware/httpauth.py => scrapy/branches/cluster-refactor/scrapy/contrib/downloadermiddleware/httpauth.py rename : scrapy/trunk/scrapy/contrib/downloadermiddleware/redirect.py => scrapy/branches/cluster-refactor/scrapy/contrib/downloadermiddleware/redirect.py rename : scrapy/trunk/scrapy/contrib/downloadermiddleware/retry.py => scrapy/branches/cluster-refactor/scrapy/contrib/downloadermiddleware/retry.py rename : scrapy/trunk/scrapy/contrib/downloadermiddleware/robots.py => scrapy/branches/cluster-refactor/scrapy/contrib/downloadermiddleware/robots.py rename : scrapy/trunk/scrapy/contrib/downloadermiddleware/useragent.py => scrapy/branches/cluster-refactor/scrapy/contrib/downloadermiddleware/useragent.py rename : scrapy/trunk/scrapy/contrib/groupsettings.py => scrapy/branches/cluster-refactor/scrapy/contrib/groupsettings.py rename : scrapy/trunk/scrapy/contrib/history/__init__.py => scrapy/branches/cluster-refactor/scrapy/contrib/history/__init__.py rename : scrapy/trunk/scrapy/contrib/history/history.py => scrapy/branches/cluster-refactor/scrapy/contrib/history/history.py rename : scrapy/trunk/scrapy/contrib/history/middleware.py => scrapy/branches/cluster-refactor/scrapy/contrib/history/middleware.py rename : scrapy/trunk/scrapy/contrib/history/scheduler.py => scrapy/branches/cluster-refactor/scrapy/contrib/history/scheduler.py rename : scrapy/trunk/scrapy/contrib/history/store.py => scrapy/branches/cluster-refactor/scrapy/contrib/history/store.py rename : scrapy/trunk/scrapy/contrib/item/__init__.py => scrapy/branches/cluster-refactor/scrapy/contrib/item/__init__.py rename : scrapy/trunk/scrapy/contrib/item/models.py => scrapy/branches/cluster-refactor/scrapy/contrib/item/models.py rename : scrapy/trunk/scrapy/contrib/memdebug.py => scrapy/branches/cluster-refactor/scrapy/contrib/memdebug.py rename : scrapy/trunk/scrapy/contrib/memusage.py => scrapy/branches/cluster-refactor/scrapy/contrib/memusage.py rename : scrapy/trunk/scrapy/contrib/pbcluster/__init__.py => scrapy/branches/cluster-refactor/scrapy/contrib/pbcluster/__init__.py rename : scrapy/trunk/scrapy/contrib/pbcluster/crawler/__init__.py => scrapy/branches/cluster-refactor/scrapy/contrib/pbcluster/crawler/__init__.py rename : scrapy/trunk/scrapy/contrib/pbcluster/crawler/manager.py => scrapy/branches/cluster-refactor/scrapy/contrib/pbcluster/crawler/manager.py rename : scrapy/trunk/scrapy/contrib/pbcluster/master/__init__.py => scrapy/branches/cluster-refactor/scrapy/contrib/pbcluster/master/__init__.py rename : scrapy/trunk/scrapy/contrib/pbcluster/master/manager.py => scrapy/branches/cluster-refactor/scrapy/contrib/pbcluster/master/manager.py rename : scrapy/trunk/scrapy/contrib/pbcluster/master/web.py => scrapy/branches/cluster-refactor/scrapy/contrib/pbcluster/master/web.py rename : scrapy/trunk/scrapy/contrib/pbcluster/master/ws_api.txt => scrapy/branches/cluster-refactor/scrapy/contrib/pbcluster/master/ws_api.txt rename : scrapy/trunk/scrapy/contrib/pbcluster/tools/scrapy-cluster-ctl.py => scrapy/branches/cluster-refactor/scrapy/contrib/pbcluster/tools/scrapy-cluster-ctl.py rename : scrapy/trunk/scrapy/contrib/pbcluster/worker/__init__.py => scrapy/branches/cluster-refactor/scrapy/contrib/pbcluster/worker/__init__.py rename : scrapy/trunk/scrapy/contrib/pbcluster/worker/manager.py => scrapy/branches/cluster-refactor/scrapy/contrib/pbcluster/worker/manager.py rename : scrapy/trunk/scrapy/contrib/pbcluster/worker/testworker.py => scrapy/branches/cluster-refactor/scrapy/contrib/pbcluster/worker/testworker.py rename : scrapy/trunk/scrapy/contrib/pipeline/__init__.py => scrapy/branches/cluster-refactor/scrapy/contrib/pipeline/__init__.py rename : scrapy/trunk/scrapy/contrib/pipeline/images.py => scrapy/branches/cluster-refactor/scrapy/contrib/pipeline/images.py rename : scrapy/trunk/scrapy/contrib/pipeline/media.py => scrapy/branches/cluster-refactor/scrapy/contrib/pipeline/media.py rename : scrapy/trunk/scrapy/contrib/pipeline/s3images.py => scrapy/branches/cluster-refactor/scrapy/contrib/pipeline/s3images.py rename : scrapy/trunk/scrapy/contrib/pipeline/shoveitem.py => scrapy/branches/cluster-refactor/scrapy/contrib/pipeline/shoveitem.py rename : scrapy/trunk/scrapy/contrib/pipeline/show.py => scrapy/branches/cluster-refactor/scrapy/contrib/pipeline/show.py rename : scrapy/trunk/scrapy/contrib/prioritizers.py => scrapy/branches/cluster-refactor/scrapy/contrib/prioritizers.py rename : scrapy/trunk/scrapy/contrib/response/__init__.py => scrapy/branches/cluster-refactor/scrapy/contrib/response/__init__.py rename : scrapy/trunk/scrapy/contrib/response/soup.py => scrapy/branches/cluster-refactor/scrapy/contrib/response/soup.py rename : scrapy/trunk/scrapy/contrib/spider/__init__.py => scrapy/branches/cluster-refactor/scrapy/contrib/spider/__init__.py rename : scrapy/trunk/scrapy/contrib/spider/profiler.py => scrapy/branches/cluster-refactor/scrapy/contrib/spider/profiler.py rename : scrapy/trunk/scrapy/contrib/spider/reloader.py => scrapy/branches/cluster-refactor/scrapy/contrib/spider/reloader.py rename : scrapy/trunk/scrapy/contrib/spidermiddleware/__init__.py => scrapy/branches/cluster-refactor/scrapy/contrib/spidermiddleware/__init__.py rename : scrapy/trunk/scrapy/contrib/spidermiddleware/depth.py => scrapy/branches/cluster-refactor/scrapy/contrib/spidermiddleware/depth.py rename : scrapy/trunk/scrapy/contrib/spidermiddleware/limit.py => scrapy/branches/cluster-refactor/scrapy/contrib/spidermiddleware/limit.py rename : scrapy/trunk/scrapy/contrib/spidermiddleware/offsite.py => scrapy/branches/cluster-refactor/scrapy/contrib/spidermiddleware/offsite.py rename : scrapy/trunk/scrapy/contrib/spidermiddleware/referer.py => scrapy/branches/cluster-refactor/scrapy/contrib/spidermiddleware/referer.py rename : scrapy/trunk/scrapy/contrib/spidermiddleware/restrict.py => scrapy/branches/cluster-refactor/scrapy/contrib/spidermiddleware/restrict.py rename : scrapy/trunk/scrapy/contrib/spidermiddleware/urlfilter.py => scrapy/branches/cluster-refactor/scrapy/contrib/spidermiddleware/urlfilter.py rename : scrapy/trunk/scrapy/contrib/spidermiddleware/urllength.py => scrapy/branches/cluster-refactor/scrapy/contrib/spidermiddleware/urllength.py rename : scrapy/trunk/scrapy/contrib/spiders.py => scrapy/branches/cluster-refactor/scrapy/contrib/spiders.py rename : scrapy/trunk/scrapy/contrib/web/__init__.py => scrapy/branches/cluster-refactor/scrapy/contrib/web/__init__.py rename : scrapy/trunk/scrapy/contrib/web/http.py => scrapy/branches/cluster-refactor/scrapy/contrib/web/http.py rename : scrapy/trunk/scrapy/contrib/web/json.py => scrapy/branches/cluster-refactor/scrapy/contrib/web/json.py rename : scrapy/trunk/scrapy/contrib/web/service.py => scrapy/branches/cluster-refactor/scrapy/contrib/web/service.py rename : scrapy/trunk/scrapy/contrib/web/site.py => scrapy/branches/cluster-refactor/scrapy/contrib/web/site.py rename : scrapy/trunk/scrapy/contrib/web/stats.py => scrapy/branches/cluster-refactor/scrapy/contrib/web/stats.py rename : scrapy/trunk/scrapy/contrib/webconsole/__init__.py => scrapy/branches/cluster-refactor/scrapy/contrib/webconsole/__init__.py rename : scrapy/trunk/scrapy/contrib/webconsole/enginestatus.py => scrapy/branches/cluster-refactor/scrapy/contrib/webconsole/enginestatus.py rename : scrapy/trunk/scrapy/contrib/webconsole/livestats.py => scrapy/branches/cluster-refactor/scrapy/contrib/webconsole/livestats.py rename : scrapy/trunk/scrapy/contrib/webconsole/schedstats.py => scrapy/branches/cluster-refactor/scrapy/contrib/webconsole/schedstats.py rename : scrapy/trunk/scrapy/contrib/webconsole/spiderctl.py => scrapy/branches/cluster-refactor/scrapy/contrib/webconsole/spiderctl.py rename : scrapy/trunk/scrapy/contrib/webconsole/spiderstats.py => scrapy/branches/cluster-refactor/scrapy/contrib/webconsole/spiderstats.py rename : scrapy/trunk/scrapy/contrib/webconsole/stats.py => scrapy/branches/cluster-refactor/scrapy/contrib/webconsole/stats.py rename : scrapy/trunk/scrapy/core/__init__.py => scrapy/branches/cluster-refactor/scrapy/core/__init__.py rename : scrapy/trunk/scrapy/core/downloader/__init__.py => scrapy/branches/cluster-refactor/scrapy/core/downloader/__init__.py rename : scrapy/trunk/scrapy/core/downloader/handlers.py => scrapy/branches/cluster-refactor/scrapy/core/downloader/handlers.py rename : scrapy/trunk/scrapy/core/downloader/manager.py => scrapy/branches/cluster-refactor/scrapy/core/downloader/manager.py rename : scrapy/trunk/scrapy/core/downloader/middleware.py => scrapy/branches/cluster-refactor/scrapy/core/downloader/middleware.py rename : scrapy/trunk/scrapy/core/engine.py => scrapy/branches/cluster-refactor/scrapy/core/engine.py rename : scrapy/trunk/scrapy/core/exceptions.py => scrapy/branches/cluster-refactor/scrapy/core/exceptions.py rename : scrapy/trunk/scrapy/core/manager.py => scrapy/branches/cluster-refactor/scrapy/core/manager.py rename : scrapy/trunk/scrapy/core/prioritizers.py => scrapy/branches/cluster-refactor/scrapy/core/prioritizers.py rename : scrapy/trunk/scrapy/core/scheduler/__init__.py => scrapy/branches/cluster-refactor/scrapy/core/scheduler/__init__.py rename : scrapy/trunk/scrapy/core/scheduler/filter.py => scrapy/branches/cluster-refactor/scrapy/core/scheduler/filter.py rename : scrapy/trunk/scrapy/core/scheduler/schedulers.py => scrapy/branches/cluster-refactor/scrapy/core/scheduler/schedulers.py rename : scrapy/trunk/scrapy/core/scheduler/store.py => scrapy/branches/cluster-refactor/scrapy/core/scheduler/store.py rename : scrapy/trunk/scrapy/core/signals.py => scrapy/branches/cluster-refactor/scrapy/core/signals.py rename : scrapy/trunk/scrapy/extension/__init__.py => scrapy/branches/cluster-refactor/scrapy/extension/__init__.py rename : scrapy/trunk/scrapy/fetcher/__init__.py => scrapy/branches/cluster-refactor/scrapy/fetcher/__init__.py rename : scrapy/trunk/scrapy/http/__init__.py => scrapy/branches/cluster-refactor/scrapy/http/__init__.py rename : scrapy/trunk/scrapy/http/headers.py => scrapy/branches/cluster-refactor/scrapy/http/headers.py rename : scrapy/trunk/scrapy/http/request.py => scrapy/branches/cluster-refactor/scrapy/http/request.py rename : scrapy/trunk/scrapy/http/response.py => scrapy/branches/cluster-refactor/scrapy/http/response.py rename : scrapy/trunk/scrapy/http/url.py => scrapy/branches/cluster-refactor/scrapy/http/url.py rename : scrapy/trunk/scrapy/item/__init__.py => scrapy/branches/cluster-refactor/scrapy/item/__init__.py rename : scrapy/trunk/scrapy/item/adaptors.py => scrapy/branches/cluster-refactor/scrapy/item/adaptors.py rename : scrapy/trunk/scrapy/item/models.py => scrapy/branches/cluster-refactor/scrapy/item/models.py rename : scrapy/trunk/scrapy/item/pipeline.py => scrapy/branches/cluster-refactor/scrapy/item/pipeline.py rename : scrapy/trunk/scrapy/link/__init__.py => scrapy/branches/cluster-refactor/scrapy/link/__init__.py rename : scrapy/trunk/scrapy/link/extractors.py => scrapy/branches/cluster-refactor/scrapy/link/extractors.py rename : scrapy/trunk/scrapy/log/__init__.py => scrapy/branches/cluster-refactor/scrapy/log/__init__.py rename : scrapy/trunk/scrapy/mail/__init__.py => scrapy/branches/cluster-refactor/scrapy/mail/__init__.py rename : scrapy/trunk/scrapy/management/__init__.py => scrapy/branches/cluster-refactor/scrapy/management/__init__.py rename : scrapy/trunk/scrapy/management/telnet.py => scrapy/branches/cluster-refactor/scrapy/management/telnet.py rename : scrapy/trunk/scrapy/management/web.py => scrapy/branches/cluster-refactor/scrapy/management/web.py rename : scrapy/trunk/scrapy/patches/__init__.py => scrapy/branches/cluster-refactor/scrapy/patches/__init__.py rename : scrapy/trunk/scrapy/patches/monkeypatches.py => scrapy/branches/cluster-refactor/scrapy/patches/monkeypatches.py rename : scrapy/trunk/scrapy/replay/__init__.py => scrapy/branches/cluster-refactor/scrapy/replay/__init__.py rename : scrapy/trunk/scrapy/spider/__init__.py => scrapy/branches/cluster-refactor/scrapy/spider/__init__.py rename : scrapy/trunk/scrapy/spider/manager.py => scrapy/branches/cluster-refactor/scrapy/spider/manager.py rename : scrapy/trunk/scrapy/spider/middleware.py => scrapy/branches/cluster-refactor/scrapy/spider/middleware.py rename : scrapy/trunk/scrapy/spider/models.py => scrapy/branches/cluster-refactor/scrapy/spider/models.py rename : scrapy/trunk/scrapy/stats/__init__.py => scrapy/branches/cluster-refactor/scrapy/stats/__init__.py rename : scrapy/trunk/scrapy/stats/corestats.py => scrapy/branches/cluster-refactor/scrapy/stats/corestats.py rename : scrapy/trunk/scrapy/stats/statscollector.py => scrapy/branches/cluster-refactor/scrapy/stats/statscollector.py rename : scrapy/trunk/scrapy/store/__init__.py => scrapy/branches/cluster-refactor/scrapy/store/__init__.py rename : scrapy/trunk/scrapy/store/db.py => scrapy/branches/cluster-refactor/scrapy/store/db.py rename : scrapy/trunk/scrapy/tests/__init__.py => scrapy/branches/cluster-refactor/scrapy/tests/__init__.py rename : scrapy/trunk/scrapy/tests/sample_data/compressed/feed-sample1.tar => scrapy/branches/cluster-refactor/scrapy/tests/sample_data/compressed/feed-sample1.tar rename : scrapy/trunk/scrapy/tests/sample_data/compressed/feed-sample1.xml => scrapy/branches/cluster-refactor/scrapy/tests/sample_data/compressed/feed-sample1.xml rename : scrapy/trunk/scrapy/tests/sample_data/compressed/feed-sample1.xml.bz2 => scrapy/branches/cluster-refactor/scrapy/tests/sample_data/compressed/feed-sample1.xml.bz2 rename : scrapy/trunk/scrapy/tests/sample_data/compressed/feed-sample1.xml.gz => scrapy/branches/cluster-refactor/scrapy/tests/sample_data/compressed/feed-sample1.xml.gz rename : scrapy/trunk/scrapy/tests/sample_data/compressed/feed-sample1.zip => scrapy/branches/cluster-refactor/scrapy/tests/sample_data/compressed/feed-sample1.zip rename : scrapy/trunk/scrapy/tests/sample_data/feeds/feed-sample1.xml => scrapy/branches/cluster-refactor/scrapy/tests/sample_data/feeds/feed-sample1.xml rename : scrapy/trunk/scrapy/tests/sample_data/feeds/feed-sample2.xml => scrapy/branches/cluster-refactor/scrapy/tests/sample_data/feeds/feed-sample2.xml rename : scrapy/trunk/scrapy/tests/sample_data/test_site/index.html => scrapy/branches/cluster-refactor/scrapy/tests/sample_data/test_site/index.html rename : scrapy/trunk/scrapy/tests/sample_data/test_site/item1.html => scrapy/branches/cluster-refactor/scrapy/tests/sample_data/test_site/item1.html rename : scrapy/trunk/scrapy/tests/sample_data/test_site/item2.html => scrapy/branches/cluster-refactor/scrapy/tests/sample_data/test_site/item2.html rename : scrapy/trunk/scrapy/tests/test_adaptors.py => scrapy/branches/cluster-refactor/scrapy/tests/test_adaptors.py rename : scrapy/trunk/scrapy/tests/test_c14nurls.py => scrapy/branches/cluster-refactor/scrapy/tests/test_c14nurls.py rename : scrapy/trunk/scrapy/tests/test_decompress.py => scrapy/branches/cluster-refactor/scrapy/tests/test_decompress.py rename : scrapy/trunk/scrapy/tests/test_defaultencoding.py => scrapy/branches/cluster-refactor/scrapy/tests/test_defaultencoding.py rename : scrapy/trunk/scrapy/tests/test_dependencies.py => scrapy/branches/cluster-refactor/scrapy/tests/test_dependencies.py rename : scrapy/trunk/scrapy/tests/test_engine.py => scrapy/branches/cluster-refactor/scrapy/tests/test_engine.py rename : scrapy/trunk/scrapy/tests/test_http_request.py => scrapy/branches/cluster-refactor/scrapy/tests/test_http_request.py rename : scrapy/trunk/scrapy/tests/test_http_response.py => scrapy/branches/cluster-refactor/scrapy/tests/test_http_response.py rename : scrapy/trunk/scrapy/tests/test_http_url.py => scrapy/branches/cluster-refactor/scrapy/tests/test_http_url.py rename : scrapy/trunk/scrapy/tests/test_libxml2.py => scrapy/branches/cluster-refactor/scrapy/tests/test_libxml2.py rename : scrapy/trunk/scrapy/tests/test_link.py => scrapy/branches/cluster-refactor/scrapy/tests/test_link.py rename : scrapy/trunk/scrapy/tests/test_pipeline_images.py => scrapy/branches/cluster-refactor/scrapy/tests/test_pipeline_images.py rename : scrapy/trunk/scrapy/tests/test_serialization.py => scrapy/branches/cluster-refactor/scrapy/tests/test_serialization.py rename : scrapy/trunk/scrapy/tests/test_spidermonkey.py => scrapy/branches/cluster-refactor/scrapy/tests/test_spidermonkey.py rename : scrapy/trunk/scrapy/tests/test_spiders/__init__.py => scrapy/branches/cluster-refactor/scrapy/tests/test_spiders/__init__.py rename : scrapy/trunk/scrapy/tests/test_spiders/testplugin.py => scrapy/branches/cluster-refactor/scrapy/tests/test_spiders/testplugin.py rename : scrapy/trunk/scrapy/tests/test_stats.py => scrapy/branches/cluster-refactor/scrapy/tests/test_stats.py rename : scrapy/trunk/scrapy/tests/test_storedb.py => scrapy/branches/cluster-refactor/scrapy/tests/test_storedb.py rename : scrapy/trunk/scrapy/tests/test_utils_datatypes.py => scrapy/branches/cluster-refactor/scrapy/tests/test_utils_datatypes.py rename : scrapy/trunk/scrapy/tests/test_utils_markup.py => scrapy/branches/cluster-refactor/scrapy/tests/test_utils_markup.py rename : scrapy/trunk/scrapy/tests/test_utils_url.py => scrapy/branches/cluster-refactor/scrapy/tests/test_utils_url.py rename : scrapy/trunk/scrapy/tests/test_utils_xml.py => scrapy/branches/cluster-refactor/scrapy/tests/test_utils_xml.py rename : scrapy/trunk/scrapy/tests/test_xpath.py => scrapy/branches/cluster-refactor/scrapy/tests/test_xpath.py rename : scrapy/trunk/scrapy/utils/__init__.py => scrapy/branches/cluster-refactor/scrapy/utils/__init__.py rename : scrapy/trunk/scrapy/utils/c14n.py => scrapy/branches/cluster-refactor/scrapy/utils/c14n.py rename : scrapy/trunk/scrapy/utils/datatypes.py => scrapy/branches/cluster-refactor/scrapy/utils/datatypes.py rename : scrapy/trunk/scrapy/utils/db.py => scrapy/branches/cluster-refactor/scrapy/utils/db.py rename : scrapy/trunk/scrapy/utils/decompressor.py => scrapy/branches/cluster-refactor/scrapy/utils/decompressor.py rename : scrapy/trunk/scrapy/utils/defer.py => scrapy/branches/cluster-refactor/scrapy/utils/defer.py rename : scrapy/trunk/scrapy/utils/display.py => scrapy/branches/cluster-refactor/scrapy/utils/display.py rename : scrapy/trunk/scrapy/utils/iterators.py => scrapy/branches/cluster-refactor/scrapy/utils/iterators.py rename : scrapy/trunk/scrapy/utils/markup.py => scrapy/branches/cluster-refactor/scrapy/utils/markup.py rename : scrapy/trunk/scrapy/utils/misc.py => scrapy/branches/cluster-refactor/scrapy/utils/misc.py rename : scrapy/trunk/scrapy/utils/python.py => scrapy/branches/cluster-refactor/scrapy/utils/python.py rename : scrapy/trunk/scrapy/utils/response.py => scrapy/branches/cluster-refactor/scrapy/utils/response.py rename : scrapy/trunk/scrapy/utils/serialization.py => scrapy/branches/cluster-refactor/scrapy/utils/serialization.py rename : scrapy/trunk/scrapy/utils/url.py => scrapy/branches/cluster-refactor/scrapy/utils/url.py rename : scrapy/trunk/scrapy/utils/xml.py => scrapy/branches/cluster-refactor/scrapy/utils/xml.py rename : scrapy/trunk/scrapy/xlib/BeautifulSoup.py => scrapy/branches/cluster-refactor/scrapy/xlib/BeautifulSoup.py rename : scrapy/trunk/scrapy/xlib/__init__.py => scrapy/branches/cluster-refactor/scrapy/xlib/__init__.py rename : scrapy/trunk/scrapy/xlib/lrucache.py => scrapy/branches/cluster-refactor/scrapy/xlib/lrucache.py rename : scrapy/trunk/scrapy/xlib/lsprofcalltree.py => scrapy/branches/cluster-refactor/scrapy/xlib/lsprofcalltree.py rename : scrapy/trunk/scrapy/xlib/pydispatch/__init__.py => scrapy/branches/cluster-refactor/scrapy/xlib/pydispatch/__init__.py rename : scrapy/trunk/scrapy/xlib/pydispatch/dispatcher.py => scrapy/branches/cluster-refactor/scrapy/xlib/pydispatch/dispatcher.py rename : scrapy/trunk/scrapy/xlib/pydispatch/errors.py => scrapy/branches/cluster-refactor/scrapy/xlib/pydispatch/errors.py rename : scrapy/trunk/scrapy/xlib/pydispatch/license.txt => scrapy/branches/cluster-refactor/scrapy/xlib/pydispatch/license.txt rename : scrapy/trunk/scrapy/xlib/pydispatch/robust.py => scrapy/branches/cluster-refactor/scrapy/xlib/pydispatch/robust.py rename : scrapy/trunk/scrapy/xlib/pydispatch/robustapply.py => scrapy/branches/cluster-refactor/scrapy/xlib/pydispatch/robustapply.py rename : scrapy/trunk/scrapy/xlib/pydispatch/saferef.py => scrapy/branches/cluster-refactor/scrapy/xlib/pydispatch/saferef.py rename : scrapy/trunk/scrapy/xlib/spidermonkey/INSTALL.scrapy => scrapy/branches/cluster-refactor/scrapy/xlib/spidermonkey/INSTALL.scrapy rename : scrapy/trunk/scrapy/xlib/spidermonkey/__init__.py => scrapy/branches/cluster-refactor/scrapy/xlib/spidermonkey/__init__.py rename : scrapy/trunk/scrapy/xlib/spidermonkey/sm_settings.py => scrapy/branches/cluster-refactor/scrapy/xlib/spidermonkey/sm_settings.py rename : scrapy/trunk/scrapy/xlib/spidermonkey/spidermonkey.py => scrapy/branches/cluster-refactor/scrapy/xlib/spidermonkey/spidermonkey.py rename : scrapy/trunk/scrapy/xpath/__init__.py => scrapy/branches/cluster-refactor/scrapy/xpath/__init__.py rename : scrapy/trunk/scrapy/xpath/constructors.py => scrapy/branches/cluster-refactor/scrapy/xpath/constructors.py rename : scrapy/trunk/scrapy/xpath/document.py => scrapy/branches/cluster-refactor/scrapy/xpath/document.py rename : scrapy/trunk/scrapy/xpath/extension.py => scrapy/branches/cluster-refactor/scrapy/xpath/extension.py rename : scrapy/trunk/scrapy/xpath/selector.py => scrapy/branches/cluster-refactor/scrapy/xpath/selector.py rename : scrapy/trunk/scrapy/xpath/types.py => scrapy/branches/cluster-refactor/scrapy/xpath/types.py rename : scrapy/trunk/scripts/rpm-install.sh => scrapy/branches/cluster-refactor/scripts/rpm-install.sh rename : scrapy/trunk/setup.cfg => scrapy/branches/cluster-refactor/setup.cfg rename : scrapy/trunk/setup.py => scrapy/branches/cluster-refactor/setup.py extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40323	2008-10-22 13:44:02 +00:00
Pablo Hoffman	00f28e3edf	using cPickle instead of pickle ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40322	2008-10-22 13:13:58 +00:00
Pablo Hoffman	52ee3781db	cleaned some code for code reusage ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40321	2008-10-21 00:42:25 +00:00
elpolilla	4ede33a46d	Fixed some bugs in the single attributes pipeline generator ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40320	2008-10-20 13:35:23 +00:00
samus_	5fddea58a7	created csv iterator to replace CSVParser iteration facilities ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40319	2008-10-17 21:53:40 +00:00
Pablo Hoffman	1146e1803e	added S3ImagesPipeline ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40318	2008-10-17 14:54:36 +00:00
elpolilla	125414c15a	Added some basic tests for the ImagePipeline (although there are a few missing yet) ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40317	2008-10-16 14:42:51 +00:00
elpolilla	3433272d71	- Added adaptors tests ... - Fixed some small bugs on a few adaptors --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40316	2008-10-16 13:29:23 +00:00
elpolilla	0ed5978abf	Improved adaptors functionality: ... - Added many basic adaptors (like extract, extract_links, regex, etc.) - Added some basic pipelines (for single data, lists, urls, etc.) - Now XPathSelectors store the response with which they were created (if any) --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40315	2008-10-13 14:30:51 +00:00
olveyra	e6f73c3dfa	removed rulengine and simpage code ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40314	2008-10-09 18:59:32 +00:00
Pablo Hoffman	e9f3913328	changed TEST_DB setting to TEST_SCRAPING_DB ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40313	2008-10-07 14:31:24 +00:00
Daniel Grana	498cd3a356	oops, missing import ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40312	2008-10-07 13:07:36 +00:00
Daniel Grana	0ae4a19e08	remove hardcoded user/password for testing storedb ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40311	2008-10-07 13:01:32 +00:00
elpolilla	fc782d5e4b	Modified ItemDeltas to work with RobustScrapedItems instead of ScrapedItems ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40310	2008-10-06 17:12:33 +00:00
Andres Moreira	05f4a26cca	Fixed bug for unicode support.The empty string ('') in some platforms is decoding as ascii, independently of the default encoding of python, changed to u''. ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40309	2008-10-06 11:58:11 +00:00
elpolilla	a309dfeb7d	Added ItemDelta objects and modified replays to make use of them ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40308	2008-10-06 10:14:52 +00:00
Pablo Hoffman	88597e3a77	removed unneeded DEFAULT_DATA_ENCODING and commented COMMANDS_MODULE in project template ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40307	2008-10-06 03:23:28 +00:00
Pablo Hoffman	d6cbaab65b	added setadaptors method to ScrapedItem, removed incorrect constructor ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40306	2008-10-06 03:22:05 +00:00
Pablo Hoffman	9c19316d21	removed unused imports and minor bug fix to media/image pipeline ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40305	2008-10-06 02:36:39 +00:00
samus_	c74364809e	removing utf-16 xpathselector_iternodes testcase since the problem comes from UnicodeDammit's conversion meaning that the results of the test are misleading ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40304	2008-10-05 18:32:39 +00:00
Pablo Hoffman	8f80603acd	added RegexLinkExtractor in new scrapy.link.extractors module ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40303	2008-10-05 07:57:51 +00:00
Pablo Hoffman	c68c478ac3	added scrapy.contrib.spiders module ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40302	2008-10-05 07:45:03 +00:00
Pablo Hoffman	7b2317d935	added scrapy.utils.response module ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40301	2008-10-05 07:39:26 +00:00
Pablo Hoffman	24df5d1602	improved scrapy-admin.py script and default project template ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40300	2008-10-05 07:37:21 +00:00
Pablo Hoffman	8dc46c16ce	removed old comment ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40299	2008-10-05 07:35:58 +00:00
Pablo Hoffman	acebe63d0a	added __nonzero__ method to XPathSelector ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40298	2008-10-05 06:58:39 +00:00
Damian Canabal	ac4688dabb	added NotSupported Exception ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40297	2008-10-03 19:50:26 +00:00
Andres Moreira	768a31a483	Added test for utils/markup.py. Added support to unicode to the new markup functions. Changed some comments. ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40296	2008-10-03 14:37:25 +00:00
Andres Moreira	453714b252	Added new functions to parse html. ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40295	2008-10-03 11:57:51 +00:00
Pablo Hoffman	71c3cea112	added another test for safe_url_string function ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40294	2008-10-03 04:14:07 +00:00
olveyra	e447044fb9	reverted an experimental code that should have been commited ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40293	2008-10-02 22:41:51 +00:00
olveyra	1f4e484a5c	added DOWNLOAD_DELAY comment in settings template ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40292	2008-10-02 20:39:36 +00:00
olveyra	58a419f45c	added support for global DOWNLOAD_DELAY setting ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40291	2008-10-02 19:59:51 +00:00
olveyra	3ec47301e3	allow to directly specify which domain corresponds to a given request ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40290	2008-10-02 19:22:34 +00:00
Pablo Hoffman	98f3314bed	simplified test without loosing functionality ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40289	2008-09-30 19:57:05 +00:00
olveyra	9ba4573b13	added test for r284 ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40288	2008-09-30 19:24:24 +00:00
olveyra	c755f5a535	better management of some redirection loops. ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40287	2008-09-30 16:56:12 +00:00
olveyra	433f35b417	small fix ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40286	2008-09-30 16:19:20 +00:00
olveyra	a38c72bb05	safe_url_string should not escape unreserved marks (see RFC 2396, sec ... 2.3) --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40285	2008-09-30 14:15:23 +00:00
olveyra	3f32bae45f	- copied original request to response.request in get_url method ... - deleted unused comment --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40284	2008-09-30 13:44:11 +00:00
elpolilla	3ae4d62f38	- Added the posibility of knowing the decompressed response's format, in the decompression tool ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40283	2008-09-29 18:14:28 +00:00
Damian Canabal	f8f2f3a542	rolled back public ent_re to private and added a function has_entities instead ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40282	2008-09-29 13:21:35 +00:00
Damian Canabal	ad00d5e632	changed private html entity regex to public ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40281	2008-09-29 12:52:54 +00:00
samus_	c07937c4df	added comment ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40280	2008-09-25 12:49:19 +00:00
samus_	c4aa1e7e8b	small fix to the regex ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40279	2008-09-25 12:45:43 +00:00
Daniel Grana	555cb0940b	images: use brief exception logging ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40278	2008-09-25 12:22:31 +00:00
olveyra	c73cbdd9b1	allow to override item class adaptor in constructor ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40277	2008-09-25 02:14:02 +00:00
samus_	79485ceb5c	added support for xml-declared encodings ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40276	2008-09-24 18:19:19 +00:00
Damian Canabal	a4b709e03f	added test for remove entities ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40275	2008-09-24 12:40:55 +00:00