Pablo Hoffman
b19ff21acd
scrapyd: added support for deferred spider queues
2010-12-10 15:55:40 -02:00
Pablo Hoffman
5c4f562ec4
scrapyd: changed keys used in poller message to _project, _spider, _job, and added link to log file in web ui
2010-11-30 13:03:20 -02:00
Pablo Hoffman
bbffa59497
Some changes to Scrapyd:
...
* Always start one process per spider
* Added max_proc_per_cpu option (defaults to 4)
* Return the number of spiders (instead of a list of them) in schedule.json
2010-11-29 17:19:05 -02:00
Pablo Hoffman
f3769651af
Refactored Scrapyd code to fix a couple of bugs that ocurred when running projects without eggs
2010-09-22 01:04:15 -03:00
Pablo Hoffman
37e9c5d78e
Added new Scrapy service with support for:
...
* multiple projects
* uploading scrapy projects as Python eggs
* scheduling spiders using a JSON API
Documentation is added along with the code.
Closes #218 .
--HG--
rename : debian/scrapy-service.default => debian/scrapyd.default
rename : debian/scrapy-service.dirs => debian/scrapyd.dirs
rename : debian/scrapy-service.install => debian/scrapyd.install
rename : debian/scrapy-service.lintian-overrides => debian/scrapyd.lintian-overrides
rename : debian/scrapy-service.postinst => debian/scrapyd.postinst
rename : debian/scrapy-service.postrm => debian/scrapyd.postrm
rename : debian/scrapy-service.upstart => debian/scrapyd.upstart
rename : extras/scrapy.tac => extras/scrapyd.tac
2010-09-03 15:54:42 -03:00
