365 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Damian Canabal	5e54ac52ff	fixed new_response_from_xpaths function, unicode string was passed as body instead of ResponseBody object ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40365	2008-11-04 17:45:26 +00:00
elpolilla	a2461fbeea	Wrong attribute assignation fixed again ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40364	2008-11-04 14:06:28 +00:00
elpolilla	224d3c5185	Bugfix in parse command ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40363	2008-11-04 13:27:31 +00:00
elpolilla	5bad79836f	Fixed bad checking while setting attributes ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40362	2008-11-04 11:26:23 +00:00
elpolilla	bd38a312d4	- Fixed bug in attributes assignation (empty attributes being set) ... - Added GUID setting to FeedSpider --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40361	2008-11-04 10:57:59 +00:00
samus_	9b46c20da2	moved xpathselector_iternodes from scrapy.utils.xml to scrapy.utils.iterators and renamed it to "xmliter", also renamed csv_iter to csviter and added tests ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40360	2008-11-03 16:10:43 +00:00
elpolilla	1ef65b97b5	Fixed typo ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40359	2008-11-03 14:00:33 +00:00
elpolilla	1a45754cf2	Removed an out-of-scrapy reference in the BasicSpider ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40358	2008-11-03 13:57:21 +00:00
elpolilla	bc13a5924a	Removed ugly loading of string codecs ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40357	2008-11-03 12:28:00 +00:00
elpolilla	defcb45120	Implemented CrawlSpider and XMLFeedSpider classes ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40356	2008-11-03 11:48:43 +00:00
elpolilla	f2bab50979	- Fixed bad implementation of the SetGUIDPipeline ... - Modified item's attribute method to have an optional 'add' argument - Renamed normalize_urls adaptor to canonicalize_urls --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40355	2008-11-03 11:02:58 +00:00
elpolilla	776818db71	Little change in test spider that broke the engine test ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40354	2008-11-03 10:43:40 +00:00
elpolilla	0574bbd44a	Added some improvements to LinkExtractor ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40353	2008-11-03 10:24:50 +00:00
elpolilla	6a2e288f22	Added support for x-mac-roman string codec ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40352	2008-10-31 11:52:35 +00:00
elpolilla	9882679bbb	Bugfix in ExtractImages adaptor ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40351	2008-10-29 02:25:32 +00:00
elpolilla	d61cd60756	Moved SetGUIDPipeline to contrib/item because it was unnecesary to put it on an exclusive module ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40350	2008-10-29 01:35:23 +00:00
elpolilla	c73dc5ad6c	Added normalize_urls adaptor, which was mentioned in the previous changeset, but not actually commited ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40349	2008-10-29 01:28:04 +00:00
elpolilla	377bea4976	- Added SetGUIDPipeline and the guid generation helper for the BasicSpider ... - Fixed some issues with BasicSpider - Added a normalize_url adaptor --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40348	2008-10-29 01:25:59 +00:00
elpolilla	4288cb3f17	- Removed AdaptorFunc objects ... - Changed "AdaptorPipe" to "AdaptorDict" - Moved adaptors to contrib/adaptors - Fixed some tests --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40347	2008-10-27 11:58:56 +00:00
Pablo Hoffman	ed98a84235	item sampler: added ITEMSAMPLER_MAX_RESPONSE_SIZE support, keeping only the first item scraped (in spidermiddleware) ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40346	2008-10-27 11:20:32 +00:00
Pablo Hoffman	1b4d41321e	added note for debian distros ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40345	2008-10-27 03:39:11 +00:00
Pablo Hoffman	f4dab4eb45	added ItemSamplerPipeline ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40344	2008-10-27 03:38:03 +00:00
Pablo Hoffman	968a55eaf3	improved RequestLimitMiddleware to conform to better programming standards ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40343	2008-10-27 01:24:19 +00:00
Pablo Hoffman	23524ccc86	ScrapedItem cannot have a constructor that receives the adaptor_pipe. adaptor_pipe must be assigned by calling set_adaptors() from the outside (typically in a spider) ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40342	2008-10-24 03:42:05 +00:00
elpolilla	ebe847bc6b	Activated adaptor pipeline creation at items __init__ ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40341	2008-10-24 00:37:51 +00:00
elpolilla	42e6ed74e5	Modified some code to avoid problems with the _adaptor_pipe attribute and the replays, or with the item themselves ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40340	2008-10-24 00:29:27 +00:00
elpolilla	06c2509634	Improved adaptors code and fixed some tests related with that ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40339	2008-10-24 00:20:14 +00:00
Pablo Hoffman	91ddfd6f80	removed debugging code ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40338	2008-10-23 12:44:24 +00:00
Pablo Hoffman	ce3bbd1a2b	enabled unsafeTracebacks to master for sending full tracebacks to workers, splitted master scheduled() method in 2 methods: schedule() and reschedule() ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40337	2008-10-23 12:43:31 +00:00
Pablo Hoffman	215151dd86	improved worker error logging for communication with master ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40336	2008-10-23 12:41:49 +00:00
Pablo Hoffman	005044b0d8	web console: added support for logging to a different file using WEBCONSOLE_LOGFILE setting ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40335	2008-10-23 11:39:03 +00:00
Pablo Hoffman	874ac0c256	changes to logging and DEFAULT_PRIORITY removed ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40334	2008-10-23 04:44:40 +00:00
Pablo Hoffman	52596f350c	some more fixes to cluster worker ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40333	2008-10-23 04:43:41 +00:00
Pablo Hoffman	3e1ad8d653	removed commas from log messages ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40332	2008-10-23 04:00:13 +00:00
Pablo Hoffman	928112a989	added ResponseCode class to contain all response codes, and other assorted code improvements ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40331	2008-10-23 01:26:48 +00:00
Pablo Hoffman	4b03435ca0	enabled unsafeTracebacks in worker to send full tracebacks to master ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40330	2008-10-23 01:15:37 +00:00
Pablo Hoffman	c913d6ffcf	added generic pre-run hooks to cluster workers to deocuple pysvn from worker code ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40329	2008-10-23 00:37:08 +00:00
Pablo Hoffman	f7275f6b0e	made charset=utf8 by default in mysql_connect ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40328	2008-10-22 17:23:30 +00:00
Pablo Hoffman	57848d46b2	updated test-worker script and moved to tools ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40327	2008-10-22 17:14:07 +00:00
Pablo Hoffman	1c74260752	fixed bug with process manager status() method ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40326	2008-10-22 16:11:47 +00:00
Pablo Hoffman	21c08e39f8	added cluster master connection log message to cluster worker ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40325	2008-10-22 15:20:12 +00:00
Pablo Hoffman	150f4ed815	scrapy cluster: added missing docstrings to important methods, fixed some bugs ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40324	2008-10-22 14:43:34 +00:00
Pablo Hoffman	5349030b31	creating cluster-refactor branch to fix several deficiences that the current ... cluster code has --HG-- rename : scrapy/trunk/INSTALL => scrapy/branches/cluster-refactor/INSTALL rename : scrapy/trunk/README => scrapy/branches/cluster-refactor/README rename : scrapy/trunk/docs/scrapy-architecture.dia => scrapy/branches/cluster-refactor/docs/scrapy-architecture.dia rename : scrapy/trunk/extras/sql/scraping.sql => scrapy/branches/cluster-refactor/extras/sql/scraping.sql rename : scrapy/trunk/scrapy/__init__.py => scrapy/branches/cluster-refactor/scrapy/__init__.py rename : scrapy/trunk/scrapy/bin/scrapy-admin.py => scrapy/branches/cluster-refactor/scrapy/bin/scrapy-admin.py rename : scrapy/trunk/scrapy/command/__init__.py => scrapy/branches/cluster-refactor/scrapy/command/__init__.py rename : scrapy/trunk/scrapy/command/cmdline.py => scrapy/branches/cluster-refactor/scrapy/command/cmdline.py rename : scrapy/trunk/scrapy/command/commands/__init__.py => scrapy/branches/cluster-refactor/scrapy/command/commands/__init__.py rename : scrapy/trunk/scrapy/command/commands/crawl.py => scrapy/branches/cluster-refactor/scrapy/command/commands/crawl.py rename : scrapy/trunk/scrapy/command/commands/download.py => scrapy/branches/cluster-refactor/scrapy/command/commands/download.py rename : scrapy/trunk/scrapy/command/commands/genspider.py => scrapy/branches/cluster-refactor/scrapy/command/commands/genspider.py rename : scrapy/trunk/scrapy/command/commands/getattr.py => scrapy/branches/cluster-refactor/scrapy/command/commands/getattr.py rename : scrapy/trunk/scrapy/command/commands/help.py => scrapy/branches/cluster-refactor/scrapy/command/commands/help.py rename : scrapy/trunk/scrapy/command/commands/list.py => scrapy/branches/cluster-refactor/scrapy/command/commands/list.py rename : scrapy/trunk/scrapy/command/commands/log.py => scrapy/branches/cluster-refactor/scrapy/command/commands/log.py rename : scrapy/trunk/scrapy/command/commands/parse.py => scrapy/branches/cluster-refactor/scrapy/command/commands/parse.py rename : scrapy/trunk/scrapy/command/commands/replay.py => scrapy/branches/cluster-refactor/scrapy/command/commands/replay.py rename : scrapy/trunk/scrapy/command/commands/shell.py => scrapy/branches/cluster-refactor/scrapy/command/commands/shell.py rename : scrapy/trunk/scrapy/command/commands/start.py => scrapy/branches/cluster-refactor/scrapy/command/commands/start.py rename : scrapy/trunk/scrapy/command/commands/stats.py => scrapy/branches/cluster-refactor/scrapy/command/commands/stats.py rename : scrapy/trunk/scrapy/command/models.py => scrapy/branches/cluster-refactor/scrapy/command/models.py rename : scrapy/trunk/scrapy/conf/__init__.py => scrapy/branches/cluster-refactor/scrapy/conf/__init__.py rename : scrapy/trunk/scrapy/conf/commands/__init__.py => scrapy/branches/cluster-refactor/scrapy/conf/commands/__init__.py rename : scrapy/trunk/scrapy/conf/commands/crawl.py => scrapy/branches/cluster-refactor/scrapy/conf/commands/crawl.py rename : scrapy/trunk/scrapy/conf/commands/help.py => scrapy/branches/cluster-refactor/scrapy/conf/commands/help.py rename : scrapy/trunk/scrapy/conf/commands/list.py => scrapy/branches/cluster-refactor/scrapy/conf/commands/list.py rename : scrapy/trunk/scrapy/conf/commands/log.py => scrapy/branches/cluster-refactor/scrapy/conf/commands/log.py rename : scrapy/trunk/scrapy/conf/commands/scrape.py => scrapy/branches/cluster-refactor/scrapy/conf/commands/scrape.py rename : scrapy/trunk/scrapy/conf/commands/stats.py => scrapy/branches/cluster-refactor/scrapy/conf/commands/stats.py rename : scrapy/trunk/scrapy/conf/commands/test.py => scrapy/branches/cluster-refactor/scrapy/conf/commands/test.py rename : scrapy/trunk/scrapy/conf/core_settings.py => scrapy/branches/cluster-refactor/scrapy/conf/core_settings.py rename : scrapy/trunk/scrapy/conf/project_template/__init__.py => scrapy/branches/cluster-refactor/scrapy/conf/project_template/__init__.py rename : scrapy/trunk/scrapy/conf/project_template/items.py => scrapy/branches/cluster-refactor/scrapy/conf/project_template/items.py rename : scrapy/trunk/scrapy/conf/project_template/scrapy-ctl.py => scrapy/branches/cluster-refactor/scrapy/conf/project_template/scrapy-ctl.py rename : scrapy/trunk/scrapy/conf/project_template/scrapy_settings.py => scrapy/branches/cluster-refactor/scrapy/conf/project_template/scrapy_settings.py rename : scrapy/trunk/scrapy/conf/project_template/spiders/__init__.py => scrapy/branches/cluster-refactor/scrapy/conf/project_template/spiders/__init__.py rename : scrapy/trunk/scrapy/contrib/__init__.py => scrapy/branches/cluster-refactor/scrapy/contrib/__init__.py rename : scrapy/trunk/scrapy/contrib/adaptorpipeline.py => scrapy/branches/cluster-refactor/scrapy/contrib/adaptorpipeline.py rename : scrapy/trunk/scrapy/contrib/closedomain.py => scrapy/branches/cluster-refactor/scrapy/contrib/closedomain.py rename : scrapy/trunk/scrapy/contrib/debug.py => scrapy/branches/cluster-refactor/scrapy/contrib/debug.py rename : scrapy/trunk/scrapy/contrib/downloadermiddleware/__init__.py => scrapy/branches/cluster-refactor/scrapy/contrib/downloadermiddleware/__init__.py rename : scrapy/trunk/scrapy/contrib/downloadermiddleware/cache.py => scrapy/branches/cluster-refactor/scrapy/contrib/downloadermiddleware/cache.py rename : scrapy/trunk/scrapy/contrib/downloadermiddleware/common.py => scrapy/branches/cluster-refactor/scrapy/contrib/downloadermiddleware/common.py rename : scrapy/trunk/scrapy/contrib/downloadermiddleware/compression.py => scrapy/branches/cluster-refactor/scrapy/contrib/downloadermiddleware/compression.py rename : scrapy/trunk/scrapy/contrib/downloadermiddleware/cookies.py => scrapy/branches/cluster-refactor/scrapy/contrib/downloadermiddleware/cookies.py rename : scrapy/trunk/scrapy/contrib/downloadermiddleware/debug.py => scrapy/branches/cluster-refactor/scrapy/contrib/downloadermiddleware/debug.py rename : scrapy/trunk/scrapy/contrib/downloadermiddleware/errorpages.py => scrapy/branches/cluster-refactor/scrapy/contrib/downloadermiddleware/errorpages.py rename : scrapy/trunk/scrapy/contrib/downloadermiddleware/httpauth.py => scrapy/branches/cluster-refactor/scrapy/contrib/downloadermiddleware/httpauth.py rename : scrapy/trunk/scrapy/contrib/downloadermiddleware/redirect.py => scrapy/branches/cluster-refactor/scrapy/contrib/downloadermiddleware/redirect.py rename : scrapy/trunk/scrapy/contrib/downloadermiddleware/retry.py => scrapy/branches/cluster-refactor/scrapy/contrib/downloadermiddleware/retry.py rename : scrapy/trunk/scrapy/contrib/downloadermiddleware/robots.py => scrapy/branches/cluster-refactor/scrapy/contrib/downloadermiddleware/robots.py rename : scrapy/trunk/scrapy/contrib/downloadermiddleware/useragent.py => scrapy/branches/cluster-refactor/scrapy/contrib/downloadermiddleware/useragent.py rename : scrapy/trunk/scrapy/contrib/groupsettings.py => scrapy/branches/cluster-refactor/scrapy/contrib/groupsettings.py rename : scrapy/trunk/scrapy/contrib/history/__init__.py => scrapy/branches/cluster-refactor/scrapy/contrib/history/__init__.py rename : scrapy/trunk/scrapy/contrib/history/history.py => scrapy/branches/cluster-refactor/scrapy/contrib/history/history.py rename : scrapy/trunk/scrapy/contrib/history/middleware.py => scrapy/branches/cluster-refactor/scrapy/contrib/history/middleware.py rename : scrapy/trunk/scrapy/contrib/history/scheduler.py => scrapy/branches/cluster-refactor/scrapy/contrib/history/scheduler.py rename : scrapy/trunk/scrapy/contrib/history/store.py => scrapy/branches/cluster-refactor/scrapy/contrib/history/store.py rename : scrapy/trunk/scrapy/contrib/item/__init__.py => scrapy/branches/cluster-refactor/scrapy/contrib/item/__init__.py rename : scrapy/trunk/scrapy/contrib/item/models.py => scrapy/branches/cluster-refactor/scrapy/contrib/item/models.py rename : scrapy/trunk/scrapy/contrib/memdebug.py => scrapy/branches/cluster-refactor/scrapy/contrib/memdebug.py rename : scrapy/trunk/scrapy/contrib/memusage.py => scrapy/branches/cluster-refactor/scrapy/contrib/memusage.py rename : scrapy/trunk/scrapy/contrib/pbcluster/__init__.py => scrapy/branches/cluster-refactor/scrapy/contrib/pbcluster/__init__.py rename : scrapy/trunk/scrapy/contrib/pbcluster/crawler/__init__.py => scrapy/branches/cluster-refactor/scrapy/contrib/pbcluster/crawler/__init__.py rename : scrapy/trunk/scrapy/contrib/pbcluster/crawler/manager.py => scrapy/branches/cluster-refactor/scrapy/contrib/pbcluster/crawler/manager.py rename : scrapy/trunk/scrapy/contrib/pbcluster/master/__init__.py => scrapy/branches/cluster-refactor/scrapy/contrib/pbcluster/master/__init__.py rename : scrapy/trunk/scrapy/contrib/pbcluster/master/manager.py => scrapy/branches/cluster-refactor/scrapy/contrib/pbcluster/master/manager.py rename : scrapy/trunk/scrapy/contrib/pbcluster/master/web.py => scrapy/branches/cluster-refactor/scrapy/contrib/pbcluster/master/web.py rename : scrapy/trunk/scrapy/contrib/pbcluster/master/ws_api.txt => scrapy/branches/cluster-refactor/scrapy/contrib/pbcluster/master/ws_api.txt rename : scrapy/trunk/scrapy/contrib/pbcluster/tools/scrapy-cluster-ctl.py => scrapy/branches/cluster-refactor/scrapy/contrib/pbcluster/tools/scrapy-cluster-ctl.py rename : scrapy/trunk/scrapy/contrib/pbcluster/worker/__init__.py => scrapy/branches/cluster-refactor/scrapy/contrib/pbcluster/worker/__init__.py rename : scrapy/trunk/scrapy/contrib/pbcluster/worker/manager.py => scrapy/branches/cluster-refactor/scrapy/contrib/pbcluster/worker/manager.py rename : scrapy/trunk/scrapy/contrib/pbcluster/worker/testworker.py => scrapy/branches/cluster-refactor/scrapy/contrib/pbcluster/worker/testworker.py rename : scrapy/trunk/scrapy/contrib/pipeline/__init__.py => scrapy/branches/cluster-refactor/scrapy/contrib/pipeline/__init__.py rename : scrapy/trunk/scrapy/contrib/pipeline/images.py => scrapy/branches/cluster-refactor/scrapy/contrib/pipeline/images.py rename : scrapy/trunk/scrapy/contrib/pipeline/media.py => scrapy/branches/cluster-refactor/scrapy/contrib/pipeline/media.py rename : scrapy/trunk/scrapy/contrib/pipeline/s3images.py => scrapy/branches/cluster-refactor/scrapy/contrib/pipeline/s3images.py rename : scrapy/trunk/scrapy/contrib/pipeline/shoveitem.py => scrapy/branches/cluster-refactor/scrapy/contrib/pipeline/shoveitem.py rename : scrapy/trunk/scrapy/contrib/pipeline/show.py => scrapy/branches/cluster-refactor/scrapy/contrib/pipeline/show.py rename : scrapy/trunk/scrapy/contrib/prioritizers.py => scrapy/branches/cluster-refactor/scrapy/contrib/prioritizers.py rename : scrapy/trunk/scrapy/contrib/response/__init__.py => scrapy/branches/cluster-refactor/scrapy/contrib/response/__init__.py rename : scrapy/trunk/scrapy/contrib/response/soup.py => scrapy/branches/cluster-refactor/scrapy/contrib/response/soup.py rename : scrapy/trunk/scrapy/contrib/spider/__init__.py => scrapy/branches/cluster-refactor/scrapy/contrib/spider/__init__.py rename : scrapy/trunk/scrapy/contrib/spider/profiler.py => scrapy/branches/cluster-refactor/scrapy/contrib/spider/profiler.py rename : scrapy/trunk/scrapy/contrib/spider/reloader.py => scrapy/branches/cluster-refactor/scrapy/contrib/spider/reloader.py rename : scrapy/trunk/scrapy/contrib/spidermiddleware/__init__.py => scrapy/branches/cluster-refactor/scrapy/contrib/spidermiddleware/__init__.py rename : scrapy/trunk/scrapy/contrib/spidermiddleware/depth.py => scrapy/branches/cluster-refactor/scrapy/contrib/spidermiddleware/depth.py rename : scrapy/trunk/scrapy/contrib/spidermiddleware/limit.py => scrapy/branches/cluster-refactor/scrapy/contrib/spidermiddleware/limit.py rename : scrapy/trunk/scrapy/contrib/spidermiddleware/offsite.py => scrapy/branches/cluster-refactor/scrapy/contrib/spidermiddleware/offsite.py rename : scrapy/trunk/scrapy/contrib/spidermiddleware/referer.py => scrapy/branches/cluster-refactor/scrapy/contrib/spidermiddleware/referer.py rename : scrapy/trunk/scrapy/contrib/spidermiddleware/restrict.py => scrapy/branches/cluster-refactor/scrapy/contrib/spidermiddleware/restrict.py rename : scrapy/trunk/scrapy/contrib/spidermiddleware/urlfilter.py => scrapy/branches/cluster-refactor/scrapy/contrib/spidermiddleware/urlfilter.py rename : scrapy/trunk/scrapy/contrib/spidermiddleware/urllength.py => scrapy/branches/cluster-refactor/scrapy/contrib/spidermiddleware/urllength.py rename : scrapy/trunk/scrapy/contrib/spiders.py => scrapy/branches/cluster-refactor/scrapy/contrib/spiders.py rename : scrapy/trunk/scrapy/contrib/web/__init__.py => scrapy/branches/cluster-refactor/scrapy/contrib/web/__init__.py rename : scrapy/trunk/scrapy/contrib/web/http.py => scrapy/branches/cluster-refactor/scrapy/contrib/web/http.py rename : scrapy/trunk/scrapy/contrib/web/json.py => scrapy/branches/cluster-refactor/scrapy/contrib/web/json.py rename : scrapy/trunk/scrapy/contrib/web/service.py => scrapy/branches/cluster-refactor/scrapy/contrib/web/service.py rename : scrapy/trunk/scrapy/contrib/web/site.py => scrapy/branches/cluster-refactor/scrapy/contrib/web/site.py rename : scrapy/trunk/scrapy/contrib/web/stats.py => scrapy/branches/cluster-refactor/scrapy/contrib/web/stats.py rename : scrapy/trunk/scrapy/contrib/webconsole/__init__.py => scrapy/branches/cluster-refactor/scrapy/contrib/webconsole/__init__.py rename : scrapy/trunk/scrapy/contrib/webconsole/enginestatus.py => scrapy/branches/cluster-refactor/scrapy/contrib/webconsole/enginestatus.py rename : scrapy/trunk/scrapy/contrib/webconsole/livestats.py => scrapy/branches/cluster-refactor/scrapy/contrib/webconsole/livestats.py rename : scrapy/trunk/scrapy/contrib/webconsole/schedstats.py => scrapy/branches/cluster-refactor/scrapy/contrib/webconsole/schedstats.py rename : scrapy/trunk/scrapy/contrib/webconsole/spiderctl.py => scrapy/branches/cluster-refactor/scrapy/contrib/webconsole/spiderctl.py rename : scrapy/trunk/scrapy/contrib/webconsole/spiderstats.py => scrapy/branches/cluster-refactor/scrapy/contrib/webconsole/spiderstats.py rename : scrapy/trunk/scrapy/contrib/webconsole/stats.py => scrapy/branches/cluster-refactor/scrapy/contrib/webconsole/stats.py rename : scrapy/trunk/scrapy/core/__init__.py => scrapy/branches/cluster-refactor/scrapy/core/__init__.py rename : scrapy/trunk/scrapy/core/downloader/__init__.py => scrapy/branches/cluster-refactor/scrapy/core/downloader/__init__.py rename : scrapy/trunk/scrapy/core/downloader/handlers.py => scrapy/branches/cluster-refactor/scrapy/core/downloader/handlers.py rename : scrapy/trunk/scrapy/core/downloader/manager.py => scrapy/branches/cluster-refactor/scrapy/core/downloader/manager.py rename : scrapy/trunk/scrapy/core/downloader/middleware.py => scrapy/branches/cluster-refactor/scrapy/core/downloader/middleware.py rename : scrapy/trunk/scrapy/core/engine.py => scrapy/branches/cluster-refactor/scrapy/core/engine.py rename : scrapy/trunk/scrapy/core/exceptions.py => scrapy/branches/cluster-refactor/scrapy/core/exceptions.py rename : scrapy/trunk/scrapy/core/manager.py => scrapy/branches/cluster-refactor/scrapy/core/manager.py rename : scrapy/trunk/scrapy/core/prioritizers.py => scrapy/branches/cluster-refactor/scrapy/core/prioritizers.py rename : scrapy/trunk/scrapy/core/scheduler/__init__.py => scrapy/branches/cluster-refactor/scrapy/core/scheduler/__init__.py rename : scrapy/trunk/scrapy/core/scheduler/filter.py => scrapy/branches/cluster-refactor/scrapy/core/scheduler/filter.py rename : scrapy/trunk/scrapy/core/scheduler/schedulers.py => scrapy/branches/cluster-refactor/scrapy/core/scheduler/schedulers.py rename : scrapy/trunk/scrapy/core/scheduler/store.py => scrapy/branches/cluster-refactor/scrapy/core/scheduler/store.py rename : scrapy/trunk/scrapy/core/signals.py => scrapy/branches/cluster-refactor/scrapy/core/signals.py rename : scrapy/trunk/scrapy/extension/__init__.py => scrapy/branches/cluster-refactor/scrapy/extension/__init__.py rename : scrapy/trunk/scrapy/fetcher/__init__.py => scrapy/branches/cluster-refactor/scrapy/fetcher/__init__.py rename : scrapy/trunk/scrapy/http/__init__.py => scrapy/branches/cluster-refactor/scrapy/http/__init__.py rename : scrapy/trunk/scrapy/http/headers.py => scrapy/branches/cluster-refactor/scrapy/http/headers.py rename : scrapy/trunk/scrapy/http/request.py => scrapy/branches/cluster-refactor/scrapy/http/request.py rename : scrapy/trunk/scrapy/http/response.py => scrapy/branches/cluster-refactor/scrapy/http/response.py rename : scrapy/trunk/scrapy/http/url.py => scrapy/branches/cluster-refactor/scrapy/http/url.py rename : scrapy/trunk/scrapy/item/__init__.py => scrapy/branches/cluster-refactor/scrapy/item/__init__.py rename : scrapy/trunk/scrapy/item/adaptors.py => scrapy/branches/cluster-refactor/scrapy/item/adaptors.py rename : scrapy/trunk/scrapy/item/models.py => scrapy/branches/cluster-refactor/scrapy/item/models.py rename : scrapy/trunk/scrapy/item/pipeline.py => scrapy/branches/cluster-refactor/scrapy/item/pipeline.py rename : scrapy/trunk/scrapy/link/__init__.py => scrapy/branches/cluster-refactor/scrapy/link/__init__.py rename : scrapy/trunk/scrapy/link/extractors.py => scrapy/branches/cluster-refactor/scrapy/link/extractors.py rename : scrapy/trunk/scrapy/log/__init__.py => scrapy/branches/cluster-refactor/scrapy/log/__init__.py rename : scrapy/trunk/scrapy/mail/__init__.py => scrapy/branches/cluster-refactor/scrapy/mail/__init__.py rename : scrapy/trunk/scrapy/management/__init__.py => scrapy/branches/cluster-refactor/scrapy/management/__init__.py rename : scrapy/trunk/scrapy/management/telnet.py => scrapy/branches/cluster-refactor/scrapy/management/telnet.py rename : scrapy/trunk/scrapy/management/web.py => scrapy/branches/cluster-refactor/scrapy/management/web.py rename : scrapy/trunk/scrapy/patches/__init__.py => scrapy/branches/cluster-refactor/scrapy/patches/__init__.py rename : scrapy/trunk/scrapy/patches/monkeypatches.py => scrapy/branches/cluster-refactor/scrapy/patches/monkeypatches.py rename : scrapy/trunk/scrapy/replay/__init__.py => scrapy/branches/cluster-refactor/scrapy/replay/__init__.py rename : scrapy/trunk/scrapy/spider/__init__.py => scrapy/branches/cluster-refactor/scrapy/spider/__init__.py rename : scrapy/trunk/scrapy/spider/manager.py => scrapy/branches/cluster-refactor/scrapy/spider/manager.py rename : scrapy/trunk/scrapy/spider/middleware.py => scrapy/branches/cluster-refactor/scrapy/spider/middleware.py rename : scrapy/trunk/scrapy/spider/models.py => scrapy/branches/cluster-refactor/scrapy/spider/models.py rename : scrapy/trunk/scrapy/stats/__init__.py => scrapy/branches/cluster-refactor/scrapy/stats/__init__.py rename : scrapy/trunk/scrapy/stats/corestats.py => scrapy/branches/cluster-refactor/scrapy/stats/corestats.py rename : scrapy/trunk/scrapy/stats/statscollector.py => scrapy/branches/cluster-refactor/scrapy/stats/statscollector.py rename : scrapy/trunk/scrapy/store/__init__.py => scrapy/branches/cluster-refactor/scrapy/store/__init__.py rename : scrapy/trunk/scrapy/store/db.py => scrapy/branches/cluster-refactor/scrapy/store/db.py rename : scrapy/trunk/scrapy/tests/__init__.py => scrapy/branches/cluster-refactor/scrapy/tests/__init__.py rename : scrapy/trunk/scrapy/tests/sample_data/compressed/feed-sample1.tar => scrapy/branches/cluster-refactor/scrapy/tests/sample_data/compressed/feed-sample1.tar rename : scrapy/trunk/scrapy/tests/sample_data/compressed/feed-sample1.xml => scrapy/branches/cluster-refactor/scrapy/tests/sample_data/compressed/feed-sample1.xml rename : scrapy/trunk/scrapy/tests/sample_data/compressed/feed-sample1.xml.bz2 => scrapy/branches/cluster-refactor/scrapy/tests/sample_data/compressed/feed-sample1.xml.bz2 rename : scrapy/trunk/scrapy/tests/sample_data/compressed/feed-sample1.xml.gz => scrapy/branches/cluster-refactor/scrapy/tests/sample_data/compressed/feed-sample1.xml.gz rename : scrapy/trunk/scrapy/tests/sample_data/compressed/feed-sample1.zip => scrapy/branches/cluster-refactor/scrapy/tests/sample_data/compressed/feed-sample1.zip rename : scrapy/trunk/scrapy/tests/sample_data/feeds/feed-sample1.xml => scrapy/branches/cluster-refactor/scrapy/tests/sample_data/feeds/feed-sample1.xml rename : scrapy/trunk/scrapy/tests/sample_data/feeds/feed-sample2.xml => scrapy/branches/cluster-refactor/scrapy/tests/sample_data/feeds/feed-sample2.xml rename : scrapy/trunk/scrapy/tests/sample_data/test_site/index.html => scrapy/branches/cluster-refactor/scrapy/tests/sample_data/test_site/index.html rename : scrapy/trunk/scrapy/tests/sample_data/test_site/item1.html => scrapy/branches/cluster-refactor/scrapy/tests/sample_data/test_site/item1.html rename : scrapy/trunk/scrapy/tests/sample_data/test_site/item2.html => scrapy/branches/cluster-refactor/scrapy/tests/sample_data/test_site/item2.html rename : scrapy/trunk/scrapy/tests/test_adaptors.py => scrapy/branches/cluster-refactor/scrapy/tests/test_adaptors.py rename : scrapy/trunk/scrapy/tests/test_c14nurls.py => scrapy/branches/cluster-refactor/scrapy/tests/test_c14nurls.py rename : scrapy/trunk/scrapy/tests/test_decompress.py => scrapy/branches/cluster-refactor/scrapy/tests/test_decompress.py rename : scrapy/trunk/scrapy/tests/test_defaultencoding.py => scrapy/branches/cluster-refactor/scrapy/tests/test_defaultencoding.py rename : scrapy/trunk/scrapy/tests/test_dependencies.py => scrapy/branches/cluster-refactor/scrapy/tests/test_dependencies.py rename : scrapy/trunk/scrapy/tests/test_engine.py => scrapy/branches/cluster-refactor/scrapy/tests/test_engine.py rename : scrapy/trunk/scrapy/tests/test_http_request.py => scrapy/branches/cluster-refactor/scrapy/tests/test_http_request.py rename : scrapy/trunk/scrapy/tests/test_http_response.py => scrapy/branches/cluster-refactor/scrapy/tests/test_http_response.py rename : scrapy/trunk/scrapy/tests/test_http_url.py => scrapy/branches/cluster-refactor/scrapy/tests/test_http_url.py rename : scrapy/trunk/scrapy/tests/test_libxml2.py => scrapy/branches/cluster-refactor/scrapy/tests/test_libxml2.py rename : scrapy/trunk/scrapy/tests/test_link.py => scrapy/branches/cluster-refactor/scrapy/tests/test_link.py rename : scrapy/trunk/scrapy/tests/test_pipeline_images.py => scrapy/branches/cluster-refactor/scrapy/tests/test_pipeline_images.py rename : scrapy/trunk/scrapy/tests/test_serialization.py => scrapy/branches/cluster-refactor/scrapy/tests/test_serialization.py rename : scrapy/trunk/scrapy/tests/test_spidermonkey.py => scrapy/branches/cluster-refactor/scrapy/tests/test_spidermonkey.py rename : scrapy/trunk/scrapy/tests/test_spiders/__init__.py => scrapy/branches/cluster-refactor/scrapy/tests/test_spiders/__init__.py rename : scrapy/trunk/scrapy/tests/test_spiders/testplugin.py => scrapy/branches/cluster-refactor/scrapy/tests/test_spiders/testplugin.py rename : scrapy/trunk/scrapy/tests/test_stats.py => scrapy/branches/cluster-refactor/scrapy/tests/test_stats.py rename : scrapy/trunk/scrapy/tests/test_storedb.py => scrapy/branches/cluster-refactor/scrapy/tests/test_storedb.py rename : scrapy/trunk/scrapy/tests/test_utils_datatypes.py => scrapy/branches/cluster-refactor/scrapy/tests/test_utils_datatypes.py rename : scrapy/trunk/scrapy/tests/test_utils_markup.py => scrapy/branches/cluster-refactor/scrapy/tests/test_utils_markup.py rename : scrapy/trunk/scrapy/tests/test_utils_url.py => scrapy/branches/cluster-refactor/scrapy/tests/test_utils_url.py rename : scrapy/trunk/scrapy/tests/test_utils_xml.py => scrapy/branches/cluster-refactor/scrapy/tests/test_utils_xml.py rename : scrapy/trunk/scrapy/tests/test_xpath.py => scrapy/branches/cluster-refactor/scrapy/tests/test_xpath.py rename : scrapy/trunk/scrapy/utils/__init__.py => scrapy/branches/cluster-refactor/scrapy/utils/__init__.py rename : scrapy/trunk/scrapy/utils/c14n.py => scrapy/branches/cluster-refactor/scrapy/utils/c14n.py rename : scrapy/trunk/scrapy/utils/datatypes.py => scrapy/branches/cluster-refactor/scrapy/utils/datatypes.py rename : scrapy/trunk/scrapy/utils/db.py => scrapy/branches/cluster-refactor/scrapy/utils/db.py rename : scrapy/trunk/scrapy/utils/decompressor.py => scrapy/branches/cluster-refactor/scrapy/utils/decompressor.py rename : scrapy/trunk/scrapy/utils/defer.py => scrapy/branches/cluster-refactor/scrapy/utils/defer.py rename : scrapy/trunk/scrapy/utils/display.py => scrapy/branches/cluster-refactor/scrapy/utils/display.py rename : scrapy/trunk/scrapy/utils/iterators.py => scrapy/branches/cluster-refactor/scrapy/utils/iterators.py rename : scrapy/trunk/scrapy/utils/markup.py => scrapy/branches/cluster-refactor/scrapy/utils/markup.py rename : scrapy/trunk/scrapy/utils/misc.py => scrapy/branches/cluster-refactor/scrapy/utils/misc.py rename : scrapy/trunk/scrapy/utils/python.py => scrapy/branches/cluster-refactor/scrapy/utils/python.py rename : scrapy/trunk/scrapy/utils/response.py => scrapy/branches/cluster-refactor/scrapy/utils/response.py rename : scrapy/trunk/scrapy/utils/serialization.py => scrapy/branches/cluster-refactor/scrapy/utils/serialization.py rename : scrapy/trunk/scrapy/utils/url.py => scrapy/branches/cluster-refactor/scrapy/utils/url.py rename : scrapy/trunk/scrapy/utils/xml.py => scrapy/branches/cluster-refactor/scrapy/utils/xml.py rename : scrapy/trunk/scrapy/xlib/BeautifulSoup.py => scrapy/branches/cluster-refactor/scrapy/xlib/BeautifulSoup.py rename : scrapy/trunk/scrapy/xlib/__init__.py => scrapy/branches/cluster-refactor/scrapy/xlib/__init__.py rename : scrapy/trunk/scrapy/xlib/lrucache.py => scrapy/branches/cluster-refactor/scrapy/xlib/lrucache.py rename : scrapy/trunk/scrapy/xlib/lsprofcalltree.py => scrapy/branches/cluster-refactor/scrapy/xlib/lsprofcalltree.py rename : scrapy/trunk/scrapy/xlib/pydispatch/__init__.py => scrapy/branches/cluster-refactor/scrapy/xlib/pydispatch/__init__.py rename : scrapy/trunk/scrapy/xlib/pydispatch/dispatcher.py => scrapy/branches/cluster-refactor/scrapy/xlib/pydispatch/dispatcher.py rename : scrapy/trunk/scrapy/xlib/pydispatch/errors.py => scrapy/branches/cluster-refactor/scrapy/xlib/pydispatch/errors.py rename : scrapy/trunk/scrapy/xlib/pydispatch/license.txt => scrapy/branches/cluster-refactor/scrapy/xlib/pydispatch/license.txt rename : scrapy/trunk/scrapy/xlib/pydispatch/robust.py => scrapy/branches/cluster-refactor/scrapy/xlib/pydispatch/robust.py rename : scrapy/trunk/scrapy/xlib/pydispatch/robustapply.py => scrapy/branches/cluster-refactor/scrapy/xlib/pydispatch/robustapply.py rename : scrapy/trunk/scrapy/xlib/pydispatch/saferef.py => scrapy/branches/cluster-refactor/scrapy/xlib/pydispatch/saferef.py rename : scrapy/trunk/scrapy/xlib/spidermonkey/INSTALL.scrapy => scrapy/branches/cluster-refactor/scrapy/xlib/spidermonkey/INSTALL.scrapy rename : scrapy/trunk/scrapy/xlib/spidermonkey/__init__.py => scrapy/branches/cluster-refactor/scrapy/xlib/spidermonkey/__init__.py rename : scrapy/trunk/scrapy/xlib/spidermonkey/sm_settings.py => scrapy/branches/cluster-refactor/scrapy/xlib/spidermonkey/sm_settings.py rename : scrapy/trunk/scrapy/xlib/spidermonkey/spidermonkey.py => scrapy/branches/cluster-refactor/scrapy/xlib/spidermonkey/spidermonkey.py rename : scrapy/trunk/scrapy/xpath/__init__.py => scrapy/branches/cluster-refactor/scrapy/xpath/__init__.py rename : scrapy/trunk/scrapy/xpath/constructors.py => scrapy/branches/cluster-refactor/scrapy/xpath/constructors.py rename : scrapy/trunk/scrapy/xpath/document.py => scrapy/branches/cluster-refactor/scrapy/xpath/document.py rename : scrapy/trunk/scrapy/xpath/extension.py => scrapy/branches/cluster-refactor/scrapy/xpath/extension.py rename : scrapy/trunk/scrapy/xpath/selector.py => scrapy/branches/cluster-refactor/scrapy/xpath/selector.py rename : scrapy/trunk/scrapy/xpath/types.py => scrapy/branches/cluster-refactor/scrapy/xpath/types.py rename : scrapy/trunk/scripts/rpm-install.sh => scrapy/branches/cluster-refactor/scripts/rpm-install.sh rename : scrapy/trunk/setup.cfg => scrapy/branches/cluster-refactor/setup.cfg rename : scrapy/trunk/setup.py => scrapy/branches/cluster-refactor/setup.py extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40323	2008-10-22 13:44:02 +00:00
Pablo Hoffman	00f28e3edf	using cPickle instead of pickle ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40322	2008-10-22 13:13:58 +00:00
Pablo Hoffman	52ee3781db	cleaned some code for code reusage ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40321	2008-10-21 00:42:25 +00:00
elpolilla	4ede33a46d	Fixed some bugs in the single attributes pipeline generator ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40320	2008-10-20 13:35:23 +00:00
samus_	5fddea58a7	created csv iterator to replace CSVParser iteration facilities ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40319	2008-10-17 21:53:40 +00:00
Pablo Hoffman	1146e1803e	added S3ImagesPipeline ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40318	2008-10-17 14:54:36 +00:00
elpolilla	125414c15a	Added some basic tests for the ImagePipeline (although there are a few missing yet) ... --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40317	2008-10-16 14:42:51 +00:00
elpolilla	3433272d71	- Added adaptors tests ... - Fixed some small bugs on a few adaptors --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40316	2008-10-16 13:29:23 +00:00