Pablo Hoffman
5c4f562ec4
scrapyd: changed keys used in poller message to _project, _spider, _job, and added link to log file in web ui
2010-11-30 13:03:20 -02:00
Pablo Hoffman
df54ed0041
Some Scrapyd enhancements:
...
* added minimal web ui
* return unique id per job (spider scheduled)
* store one log per spider run (job) and rotate them, keeping the last N logs (where N is configurable through settings)
2010-11-30 02:26:31 -02:00
Pablo Hoffman
46e5d694e6
Scrapyd: return project and version in addversion.json
2010-11-29 17:22:28 -02:00
Pablo Hoffman
bbffa59497
Some changes to Scrapyd:
...
* Always start one process per spider
* Added max_proc_per_cpu option (defaults to 4)
* Return the number of spiders (instead of a list of them) in schedule.json
2010-11-29 17:19:05 -02:00
Pablo Hoffman
5bdffadbe3
Simplified get_spider_list_from_eggfile() function now that it doesn't need to chdir to a custom directory (Scrapy now works when it's unable to create the SQLite database)
2010-11-05 11:48:12 -02:00
Pablo Hoffman
31bbcc9476
Raise error when egg is corrupt in activate_egg(). Use a more descriptive name for temporary dirs in get_spider_list_from_eggfile(). Make scrapyd webservice pass egg_runner to get_spider_list_from_eggfile()
2010-11-05 11:24:33 -02:00
Pablo Hoffman
de4909faca
get_spider_list_from_eggfile(): more improvements to error messages, and support passing eggruner module as argument
2010-11-04 18:56:11 -02:00
Pablo Hoffman
7ba972d8cf
get_spider_list_from_eggfile(): fail if unable to extract spider list
2010-11-04 16:27:47 -02:00
Pablo Hoffman
a8be54a8ea
scrapyd: make Environment tests independent of the current OS environment
...
--HG--
rename : scrapyd/tests/test_envion.py => scrapyd/tests/test_environ.py
2010-10-27 06:49:15 -02:00
Pablo Hoffman
a3a108dc71
fixed some compatibility issues with python 2.5 in scrapyd
2010-10-26 17:21:43 -02:00
Pablo Hoffman
a4639ffb06
Removed hacky SCRAPY_SETTINGS_DISABLED environment variable
2010-09-22 16:08:18 -03:00
Pablo Hoffman
f3769651af
Refactored Scrapyd code to fix a couple of bugs that ocurred when running projects without eggs
2010-09-22 01:04:15 -03:00
Pablo Hoffman
4c61df7abb
get_spider_list_from_eggfile(): fixed bug when SCRAPY_SETTINGS_DISABLED is set
2010-09-20 08:47:55 -03:00
Pablo Hoffman
400c4134af
Make scrapyd.eggutils compatible with Python 2.5 and added tests for get_spider_list_from_eggfile() function ( closes #242 )
2010-09-19 21:08:27 -03:00
Pablo Hoffman
4cecbcdc5b
Fixed bug in Scrapyd launcher when running projects without eggs. Refs #238
2010-09-15 21:03:43 -03:00
Pablo Hoffman
c559b06a85
Removed unused import
2010-09-14 01:53:05 -03:00
Pablo Hoffman
833baa6041
Support running projects without eggs in Scrapyd. Closes #238
2010-09-14 01:44:25 -03:00
Pablo Hoffman
b76cd42690
Added tests for Scrapyd components. Closes #237
2010-09-14 01:44:10 -03:00
Pablo Hoffman
9158e9d682
Some changes to Scrapyd to support multiple configuration files, to make it easier to deploy Scrapyd applications. Also documented 'egg_runner' and 'application' options
...
--HG--
rename : debian/scrapyd.cfg => debian/000-default
rename : scrapyd/default_scrapyd.cfg => scrapyd/default_scrapyd.conf
2010-09-07 09:17:25 -03:00
Pablo Hoffman
37e9c5d78e
Added new Scrapy service with support for:
...
* multiple projects
* uploading scrapy projects as Python eggs
* scheduling spiders using a JSON API
Documentation is added along with the code.
Closes #218 .
--HG--
rename : debian/scrapy-service.default => debian/scrapyd.default
rename : debian/scrapy-service.dirs => debian/scrapyd.dirs
rename : debian/scrapy-service.install => debian/scrapyd.install
rename : debian/scrapy-service.lintian-overrides => debian/scrapyd.lintian-overrides
rename : debian/scrapy-service.postinst => debian/scrapyd.postinst
rename : debian/scrapy-service.postrm => debian/scrapyd.postrm
rename : debian/scrapy-service.upstart => debian/scrapyd.upstart
rename : extras/scrapy.tac => extras/scrapyd.tac
2010-09-03 15:54:42 -03:00
