Pablo Hoffman
58e88ed246
scrapyd: do not set SCRAPY_FEED_URI/SCRAPY_LOG_FILE if items_dir/logs_dir settings are not set
2012-05-08 17:43:00 -03:00
Pablo Hoffman
35fb01156e
removed some obsolete remaining code related to sqlite support in scrapy
2012-03-16 11:55:55 -03:00
Pablo Hoffman
dbda33efa6
scrapyd: added support for storing items by default
...
Items are stored the same way as logs, in jsonlines format.
Also renamed logs_to_keep setting to jobs_to_keep.
2012-01-03 23:08:54 -02:00
Pablo Hoffman
a1dbc62b45
removed CONCURRENT_SPIDERS setting (use scrapyd maxproc instead)
2011-09-02 18:27:39 -03:00
Pablo Hoffman
80b557849a
fixed test broken in previous commit
2011-06-12 02:55:21 -03:00
Pablo Hoffman
0d5399d0bf
fixed scrapyd tests on win32. closes #295
2011-06-12 02:46:41 -03:00
Pablo Hoffman
65fc2fbd1f
Set CONCURRENT_SPIDERS=1 in Scrapyd to force one spider per process
2011-02-04 13:30:01 -02:00
Pablo Hoffman
fa644f7a5e
Some simplifications to Scrapyd architecture and internals:
...
- launcher no longer knows about egg storage
- removed get_spider_list_from_eggifile() file and replaced by simpler
get_spider_list() which doesn't receive en egg file as argument
- changed "egg runner" name to just "runner" to reflect the fact that it
doesn't necesarilly run eggs (though it does in the default case)
--HG--
rename : scrapyd/eggrunner.py => scrapyd/runner.py
2010-12-27 16:22:32 -02:00
Pablo Hoffman
9cd649b3a0
scrapyd: populate SCRAPY_SPIDER and SCRAPY_JOB environment variables
2010-12-26 19:32:56 -02:00
Pablo Hoffman
1c8d74eb5b
scrapyd: populate SCRAPY_SLOT environment variable with the scrapyd slot number
2010-12-24 12:47:59 -02:00
Pablo Hoffman
5c4f562ec4
scrapyd: changed keys used in poller message to _project, _spider, _job, and added link to log file in web ui
2010-11-30 13:03:20 -02:00
Pablo Hoffman
df54ed0041
Some Scrapyd enhancements:
...
* added minimal web ui
* return unique id per job (spider scheduled)
* store one log per spider run (job) and rotate them, keeping the last N logs (where N is configurable through settings)
2010-11-30 02:26:31 -02:00
Pablo Hoffman
a8be54a8ea
scrapyd: make Environment tests independent of the current OS environment
...
--HG--
rename : scrapyd/tests/test_envion.py => scrapyd/tests/test_environ.py
2010-10-27 06:49:15 -02:00
