Pablo Hoffman
5c4f562ec4
scrapyd: changed keys used in poller message to _project, _spider, _job, and added link to log file in web ui
2010-11-30 13:03:20 -02:00
Pablo Hoffman
df54ed0041
Some Scrapyd enhancements:
...
* added minimal web ui
* return unique id per job (spider scheduled)
* store one log per spider run (job) and rotate them, keeping the last N logs (where N is configurable through settings)
2010-11-30 02:26:31 -02:00
Pablo Hoffman
bbffa59497
Some changes to Scrapyd:
...
* Always start one process per spider
* Added max_proc_per_cpu option (defaults to 4)
* Return the number of spiders (instead of a list of them) in schedule.json
2010-11-29 17:19:05 -02:00
Pablo Hoffman
a8be54a8ea
scrapyd: make Environment tests independent of the current OS environment
...
--HG--
rename : scrapyd/tests/test_envion.py => scrapyd/tests/test_environ.py
2010-10-27 06:49:15 -02:00
Pablo Hoffman
a3a108dc71
fixed some compatibility issues with python 2.5 in scrapyd
2010-10-26 17:21:43 -02:00
Pablo Hoffman
f3769651af
Refactored Scrapyd code to fix a couple of bugs that ocurred when running projects without eggs
2010-09-22 01:04:15 -03:00
Pablo Hoffman
400c4134af
Make scrapyd.eggutils compatible with Python 2.5 and added tests for get_spider_list_from_eggfile() function ( closes #242 )
2010-09-19 21:08:27 -03:00
Pablo Hoffman
833baa6041
Support running projects without eggs in Scrapyd. Closes #238
2010-09-14 01:44:25 -03:00
Pablo Hoffman
b76cd42690
Added tests for Scrapyd components. Closes #237
2010-09-14 01:44:10 -03:00
