BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-27 04:44:26 +00:00
Code Releases Wiki Activity
scrapy/scrapyd/default_scrapyd.conf

26 lines
772 B
Plaintext
Raw Normal View History

Some changes to Scrapyd to support multiple configuration files, to make it easier to deploy Scrapyd applications. Also documented 'egg_runner' and 'application' options --HG-- rename : debian/scrapyd.cfg => debian/000-default rename : scrapyd/default_scrapyd.cfg => scrapyd/default_scrapyd.conf
2010-09-07 09:17:25 -03:00
[scrapyd]
eggs_dir = eggs
logs_dir = logs
scrapyd: added support for storing items by default Items are stored the same way as logs, in jsonlines format. Also renamed logs_to_keep setting to jobs_to_keep.
2012-01-03 22:56:26 -02:00
items_dir = items
jobs_to_keep = 5
Some changes to Scrapyd to support multiple configuration files, to make it easier to deploy Scrapyd applications. Also documented 'egg_runner' and 'application' options --HG-- rename : debian/scrapyd.cfg => debian/000-default rename : scrapyd/default_scrapyd.cfg => scrapyd/default_scrapyd.conf
2010-09-07 09:17:25 -03:00
dbs_dir = dbs
max_proc = 0
Some changes to Scrapyd: * Always start one process per spider * Added max_proc_per_cpu option (defaults to 4) * Return the number of spiders (instead of a list of them) in schedule.json
2010-11-29 17:19:05 -02:00
max_proc_per_cpu = 4
some some changes to scrapyd listjobs.json api: * the api is now a GET instead of POST (for consistency) * the api also returns pending and finished jobs, in addition to running ones * only the last 100 finished jobs are kept (can be changed through the finished_to_keep setting)
2011-12-28 15:17:52 -02:00
finished_to_keep = 100
Some changes to Scrapyd to support multiple configuration files, to make it easier to deploy Scrapyd applications. Also documented 'egg_runner' and 'application' options --HG-- rename : debian/scrapyd.cfg => debian/000-default rename : scrapyd/default_scrapyd.cfg => scrapyd/default_scrapyd.conf
2010-09-07 09:17:25 -03:00
http_port = 6800
debug = off
Some simplifications to Scrapyd architecture and internals: - launcher no longer knows about egg storage - removed get_spider_list_from_eggifile() file and replaced by simpler get_spider_list() which doesn't receive en egg file as argument - changed "egg runner" name to just "runner" to reflect the fact that it doesn't necesarilly run eggs (though it does in the default case) --HG-- rename : scrapyd/eggrunner.py => scrapyd/runner.py
2010-12-27 16:22:32 -02:00
runner = scrapyd.runner
Some changes to Scrapyd to support multiple configuration files, to make it easier to deploy Scrapyd applications. Also documented 'egg_runner' and 'application' options --HG-- rename : debian/scrapyd.cfg => debian/000-default rename : scrapyd/default_scrapyd.cfg => scrapyd/default_scrapyd.conf
2010-09-07 09:17:25 -03:00
application = scrapyd.app.application
Add launcher class to config
2012-06-14 12:45:42 +07:00
launcher = scrapyd.launcher.Launcher
Configure json services members
2012-06-25 12:50:48 +07:00
[services]
schedule.json = scrapyd.webservice.Schedule
cancel.json = scrapyd.webservice.Cancel
addversion.json = scrapyd.webservice.AddVersion
listprojects.json = scrapyd.webservice.ListProjects
listversions.json = scrapyd.webservice.ListVersions
listspiders.json = scrapyd.webservice.ListSpiders
delproject.json = scrapyd.webservice.DeleteProject
delversion.json = scrapyd.webservice.DeleteVersion
listjobs.json = scrapyd.webservice.ListJobs
Copy Permalink