7 Commits

Author	SHA1	Message	Date
Pablo Hoffman	75e2c3eb33	moved spider queues to scrapyd ... --HG-- rename : scrapy/spiderqueue.py => scrapyd/spiderqueue.py rename : scrapy/tests/test_spiderqueue.py => scrapyd/tests/test_spiderqueue.py	2011-07-19 19:39:27 -03:00
Pablo Hoffman	fa644f7a5e	Some simplifications to Scrapyd architecture and internals: ... - launcher no longer knows about egg storage - removed get_spider_list_from_eggifile() file and replaced by simpler get_spider_list() which doesn't receive en egg file as argument - changed "egg runner" name to just "runner" to reflect the fact that it doesn't necesarilly run eggs (though it does in the default case) --HG-- rename : scrapyd/eggrunner.py => scrapyd/runner.py	2010-12-27 16:22:32 -02:00
Pablo Hoffman	5c4f562ec4	scrapyd: changed keys used in poller message to _project, _spider, _job, and added link to log file in web ui	2010-11-30 13:03:20 -02:00
Pablo Hoffman	df54ed0041	Some Scrapyd enhancements: ... * added minimal web ui * return unique id per job (spider scheduled) * store one log per spider run (job) and rotate them, keeping the last N logs (where N is configurable through settings)	2010-11-30 02:26:31 -02:00
Pablo Hoffman	833baa6041	Support running projects without eggs in Scrapyd. Closes #238	2010-09-14 01:44:25 -03:00
Pablo Hoffman	b76cd42690	Added tests for Scrapyd components. Closes #237	2010-09-14 01:44:10 -03:00
Pablo Hoffman	37e9c5d78e	Added new Scrapy service with support for: ... * multiple projects * uploading scrapy projects as Python eggs * scheduling spiders using a JSON API Documentation is added along with the code. Closes #218. --HG-- rename : debian/scrapy-service.default => debian/scrapyd.default rename : debian/scrapy-service.dirs => debian/scrapyd.dirs rename : debian/scrapy-service.install => debian/scrapyd.install rename : debian/scrapy-service.lintian-overrides => debian/scrapyd.lintian-overrides rename : debian/scrapy-service.postinst => debian/scrapyd.postinst rename : debian/scrapy-service.postrm => debian/scrapyd.postrm rename : debian/scrapy-service.upstart => debian/scrapyd.upstart rename : extras/scrapy.tac => extras/scrapyd.tac	2010-09-03 15:54:42 -03:00