Pablo Hoffman
91b9d89ffd
moved scrapy.utils.sqlite to scrapyd.sqlite
...
--HG--
rename : scrapy/utils/sqlite.py => scrapyd/sqlite.py
rename : scrapy/tests/test_utils_sqlite.py => scrapyd/tests/test_sqlite.py
2011-08-27 01:20:57 -03:00
Pablo Hoffman
75e2c3eb33
moved spider queues to scrapyd
...
--HG--
rename : scrapy/spiderqueue.py => scrapyd/spiderqueue.py
rename : scrapy/tests/test_spiderqueue.py => scrapyd/tests/test_spiderqueue.py
2011-07-19 19:39:27 -03:00
Pablo Hoffman
fa644f7a5e
Some simplifications to Scrapyd architecture and internals:
...
- launcher no longer knows about egg storage
- removed get_spider_list_from_eggifile() file and replaced by simpler
get_spider_list() which doesn't receive en egg file as argument
- changed "egg runner" name to just "runner" to reflect the fact that it
doesn't necesarilly run eggs (though it does in the default case)
--HG--
rename : scrapyd/eggrunner.py => scrapyd/runner.py
2010-12-27 16:22:32 -02:00
Pablo Hoffman
5c4f562ec4
scrapyd: changed keys used in poller message to _project, _spider, _job, and added link to log file in web ui
2010-11-30 13:03:20 -02:00
Pablo Hoffman
df54ed0041
Some Scrapyd enhancements:
...
* added minimal web ui
* return unique id per job (spider scheduled)
* store one log per spider run (job) and rotate them, keeping the last N logs (where N is configurable through settings)
2010-11-30 02:26:31 -02:00
Pablo Hoffman
833baa6041
Support running projects without eggs in Scrapyd. Closes #238
2010-09-14 01:44:25 -03:00
Pablo Hoffman
b76cd42690
Added tests for Scrapyd components. Closes #237
2010-09-14 01:44:10 -03:00
Pablo Hoffman
37e9c5d78e
Added new Scrapy service with support for:
...
* multiple projects
* uploading scrapy projects as Python eggs
* scheduling spiders using a JSON API
Documentation is added along with the code.
Closes #218 .
--HG--
rename : debian/scrapy-service.default => debian/scrapyd.default
rename : debian/scrapy-service.dirs => debian/scrapyd.dirs
rename : debian/scrapy-service.install => debian/scrapyd.install
rename : debian/scrapy-service.lintian-overrides => debian/scrapyd.lintian-overrides
rename : debian/scrapy-service.postinst => debian/scrapyd.postinst
rename : debian/scrapy-service.postrm => debian/scrapyd.postrm
rename : debian/scrapy-service.upstart => debian/scrapyd.upstart
rename : extras/scrapy.tac => extras/scrapyd.tac
2010-09-03 15:54:42 -03:00
