scrapy/scrapyd/default_scrapyd.conf

[scrapyd]
eggs_dir    = eggs
logs_dir    = logs
logs_to_keep = 5
dbs_dir     = dbs
max_proc    = 0
max_proc_per_cpu = 4
finished_to_keep = 100
http_port   = 6800
debug       = off
runner      = scrapyd.runner
application = scrapyd.app.application
Some changes to Scrapyd to support multiple configuration files, to make it easier to deploy Scrapyd applications. Also documented 'egg_runner' and 'application' options --HG-- rename : debian/scrapyd.cfg => debian/000-default rename : scrapyd/default_scrapyd.cfg => scrapyd/default_scrapyd.conf 2010-09-07 09:17:25 -03:00			`[scrapyd]`
			`eggs_dir = eggs`
			`logs_dir = logs`
Some Scrapyd enhancements: * added minimal web ui * return unique id per job (spider scheduled) * store one log per spider run (job) and rotate them, keeping the last N logs (where N is configurable through settings) 2010-11-30 02:26:31 -02:00			`logs_to_keep = 5`
Some changes to Scrapyd to support multiple configuration files, to make it easier to deploy Scrapyd applications. Also documented 'egg_runner' and 'application' options --HG-- rename : debian/scrapyd.cfg => debian/000-default rename : scrapyd/default_scrapyd.cfg => scrapyd/default_scrapyd.conf 2010-09-07 09:17:25 -03:00			`dbs_dir = dbs`
			`max_proc = 0`
Some changes to Scrapyd: * Always start one process per spider * Added max_proc_per_cpu option (defaults to 4) * Return the number of spiders (instead of a list of them) in schedule.json 2010-11-29 17:19:05 -02:00			`max_proc_per_cpu = 4`
some some changes to scrapyd listjobs.json api: * the api is now a GET instead of POST (for consistency) * the api also returns pending and finished jobs, in addition to running ones * only the last 100 finished jobs are kept (can be changed through the finished_to_keep setting) 2011-12-28 15:17:52 -02:00			`finished_to_keep = 100`
Some changes to Scrapyd to support multiple configuration files, to make it easier to deploy Scrapyd applications. Also documented 'egg_runner' and 'application' options --HG-- rename : debian/scrapyd.cfg => debian/000-default rename : scrapyd/default_scrapyd.cfg => scrapyd/default_scrapyd.conf 2010-09-07 09:17:25 -03:00			`http_port = 6800`
			`debug = off`
Some simplifications to Scrapyd architecture and internals: - launcher no longer knows about egg storage - removed get_spider_list_from_eggifile() file and replaced by simpler get_spider_list() which doesn't receive en egg file as argument - changed "egg runner" name to just "runner" to reflect the fact that it doesn't necesarilly run eggs (though it does in the default case) --HG-- rename : scrapyd/eggrunner.py => scrapyd/runner.py 2010-12-27 16:22:32 -02:00			`runner = scrapyd.runner`
Some changes to Scrapyd to support multiple configuration files, to make it easier to deploy Scrapyd applications. Also documented 'egg_runner' and 'application' options --HG-- rename : debian/scrapyd.cfg => debian/000-default rename : scrapyd/default_scrapyd.cfg => scrapyd/default_scrapyd.conf 2010-09-07 09:17:25 -03:00			`application = scrapyd.app.application`