Pablo Hoffman
b56e933be9
bumped version to 0.12 in documentation
2011-01-02 17:28:33 -02:00
Pablo Hoffman
cd42bd7d0c
Bumped version to 0.13
2011-01-02 17:21:31 -02:00
Pablo Hoffman
5879389ad0
Bumped version to 0.12
2011-01-02 16:16:40 -02:00
Vikas Dhiman
fe218abbfd
scrapy.utils.memory.get_vmvalue_from_procfs()
...
causes test case failure on SunOS 5.10 i86pc. Modified it to support SunOS 5.10
2011-01-02 15:46:11 -02:00
Shane Evans
aebe5d5073
make inside_project work with SCRAPY_SETTINGS_MODULE. Closes #300
2010-12-28 15:34:39 -02:00
Pablo Hoffman
3d8b368fc6
scrapyd: use runner from config (if not specified) on get_spider_list()
2010-12-28 11:16:58 -02:00
Pablo Hoffman
fa644f7a5e
Some simplifications to Scrapyd architecture and internals:
...
- launcher no longer knows about egg storage
- removed get_spider_list_from_eggifile() file and replaced by simpler
get_spider_list() which doesn't receive en egg file as argument
- changed "egg runner" name to just "runner" to reflect the fact that it
doesn't necesarilly run eggs (though it does in the default case)
--HG--
rename : scrapyd/eggrunner.py => scrapyd/runner.py
2010-12-27 16:22:32 -02:00
Pablo Hoffman
9cd649b3a0
scrapyd: populate SCRAPY_SPIDER and SCRAPY_JOB environment variables
2010-12-26 19:32:56 -02:00
Pablo Hoffman
1c8d74eb5b
scrapyd: populate SCRAPY_SLOT environment variable with the scrapyd slot number
2010-12-24 12:47:59 -02:00
Martin Olveyra
d9a3df45c6
Remove deprecated match common prefix feature from IBL code
2010-12-23 14:40:22 -02:00
Pablo Hoffman
633ebc4c43
minor indentation improvement
2010-12-23 13:04:49 -02:00
Pablo Hoffman
db07a9a938
Added notice to documentation, pointing dev to stable versions and viceversa
2010-12-23 13:03:40 -02:00
Pablo Hoffman
544308d6d0
updated ubuntu repos doc, in preparation for the 0.11 release
2010-12-21 11:02:56 -02:00
Pablo Hoffman
fff22c269a
Added test to make sure custom mime.types shipped with Scrapy is loaded, and made Scrapy more egg-friendly by using pkutil.get_data()
2010-12-20 15:16:21 -02:00
Pablo Hoffman
87aa63da2b
moved coverage report script to extras/
2010-12-15 12:08:59 -02:00
Pablo Hoffman
432251bedd
Added rules to Makefile.buildbot for generating coverage reports
2010-12-15 11:13:45 -02:00
Pablo Hoffman
63c97f65f4
fixed csviter bug when called with a Response, not TextResponse
2010-12-14 18:21:07 -02:00
Pablo Hoffman
55ec7a0554
deploy command: cleanup temporary files before doing a new build
2010-12-14 17:20:31 -02:00
Pablo Hoffman
3df9070ffd
fixed bin\runtests.bat windows script
2010-12-14 12:20:38 -02:00
Pablo Hoffman
002abf204f
Updated item_passed signal to send passed item in 'item' argument, instead of 'output' argument, keeping backwards compatibility for the 'output' argument. Closes #273
2010-12-13 14:05:47 -02:00
Pablo Hoffman
f984d438a0
updated docs to use scrapy version on aptitude install lines
2010-12-13 14:02:42 -02:00
Pablo Hoffman
60bdb7dc0a
Removed some deprecated code and modules
2010-12-13 10:34:07 -02:00
Pablo Hoffman
97a790d659
improved mechanism for tagging version with hg revision
2010-12-13 10:21:39 -02:00
Pablo Hoffman
119fd20e91
Added verbose option to 'version' command. Closes #298
2010-12-13 00:32:44 -02:00
Pablo Hoffman
6f2cea4775
added deprecation messages to queue and runserver commands
2010-12-12 20:04:35 -02:00
Pablo Hoffman
b19ff21acd
scrapyd: added support for deferred spider queues
2010-12-10 15:55:40 -02:00
Martin Olveyra
02ccca01eb
use safe_url_string in canonicalize_url, to avoid to convert safe characters into percent representation. Lead to errors with many sites (RFC3986). closes #297
2010-12-08 16:28:38 -02:00
Pablo Hoffman
6a1b69c93f
renamed command 'scrapyd' to 'server', and deprecated 'runserver' and 'queue' commands
...
--HG--
rename : scrapy/commands/scrapyd.py => scrapy/commands/server.py
2010-11-30 20:23:27 -02:00
Pablo Hoffman
831dc818d6
scrapyd: added more information webui homepage
2010-11-30 18:43:59 -02:00
Pablo Hoffman
a3d30c35fe
scrapyd: log url where web console can be accesed
2010-11-30 17:58:34 -02:00
Pablo Hoffman
823fd9822c
scrapyd: fixed bug discovering the current project scrapy.cfg file
2010-11-30 16:19:17 -02:00
Pablo Hoffman
7b84591ea9
added command for starting a scrapyd server for the current project
2010-11-30 15:52:15 -02:00
Pablo Hoffman
5a46ce47ee
scrapyd: add extra_sources consturctor argument, and also read scrapyd configuratoin from current project's scrapy.cfg file
2010-11-30 15:47:05 -02:00
Pablo Hoffman
c02d6db6a3
scrapyd: force application to receive config as argument
2010-11-30 15:46:24 -02:00
Pablo Hoffman
85890a5092
scrapyd: log process logfile when process starts/finishes
2010-11-30 15:45:42 -02:00
Pablo Hoffman
5c4f562ec4
scrapyd: changed keys used in poller message to _project, _spider, _job, and added link to log file in web ui
2010-11-30 13:03:20 -02:00
Pablo Hoffman
df54ed0041
Some Scrapyd enhancements:
...
* added minimal web ui
* return unique id per job (spider scheduled)
* store one log per spider run (job) and rotate them, keeping the last N logs (where N is configurable through settings)
2010-11-30 02:26:31 -02:00
Pablo Hoffman
46e5d694e6
Scrapyd: return project and version in addversion.json
2010-11-29 17:22:28 -02:00
Pablo Hoffman
bbffa59497
Some changes to Scrapyd:
...
* Always start one process per spider
* Added max_proc_per_cpu option (defaults to 4)
* Return the number of spiders (instead of a list of them) in schedule.json
2010-11-29 17:19:05 -02:00
Pablo Hoffman
42e8346d06
fixed failing test on win32
2010-11-29 10:27:12 -02:00
Pablo Hoffman
3cda681755
utils.testproc: make spawned process use the original CWD, instead of the temporary one created by twisted trial
2010-11-29 09:56:14 -02:00
Pablo Hoffman
1d726063d6
* Added tests for shell/fetch/version commands ( closes #255 )
...
* Fixed bug causing Scrapy shell to fail if started without any argument (closes #294 )
2010-11-28 18:14:45 -02:00
Pablo Hoffman
6f82ea19de
Fixed bug in addversion.json with old Twisted versions. Closes #293
2010-11-25 12:12:42 -02:00
Pablo Hoffman
2557777c39
Updated doc referring to HTTP cache middleware
2010-11-24 13:27:44 -02:00
Pablo Hoffman
d59ef48231
Fixed SgmlLinkExtractor bug which failed to recognize <base> tags when using restrict_xpaths
2010-11-23 17:28:29 -02:00
Pablo Hoffman
426b6fa100
docs/intro/install.rst: added -U flag to easy_install command
2010-11-22 13:50:19 -02:00
Pablo Hoffman
91e6753035
scrapy.bat: minor fix to support spaces in python installation dir (windows)
2010-11-22 00:39:45 -02:00
Pablo Hoffman
91a7c25797
* Made Response.meta attribute map to Request.meta attribute. Closes #290
...
* Record redirected URLs in redirect middleware. Closes #291
2010-11-18 12:51:54 -02:00
Pablo Hoffman
ac007802d6
Simplified installation guide, including lxml as alternative dependency to libxml2. Closes #280
2010-11-17 21:32:23 -02:00
Pablo Hoffman
2897061b98
Make scrapy conflict with previous versionsof the debian package
2010-11-17 17:30:22 -02:00
