Pablo Hoffman
4e800e90d3
Automated merge with http://hg.scrapy.org/scrapy-0.10
2010-11-10 13:36:05 -02:00
Pablo Hoffman
7698ff2a7b
Disabled help() in telnet console. Closes #284
2010-11-10 13:35:36 -02:00
Pablo Hoffman
d988ca1ec2
Some changes to scrapy deploy command:
...
* changed deploy section names to [deploy:target]
* project is now passed through a -p|--project option
* version can now be set in the target configuration
* switched meaning of -l and -L options
* updated documentation accordingly
2010-11-08 17:01:06 -02:00
Pablo Hoffman
37c9d5feff
minor update to queue command doc
2010-11-08 02:19:54 -02:00
Pablo Hoffman
5bdffadbe3
Simplified get_spider_list_from_eggfile() function now that it doesn't need to chdir to a custom directory (Scrapy now works when it's unable to create the SQLite database)
2010-11-05 11:48:12 -02:00
Pablo Hoffman
31bbcc9476
Raise error when egg is corrupt in activate_egg(). Use a more descriptive name for temporary dirs in get_spider_list_from_eggfile(). Make scrapyd webservice pass egg_runner to get_spider_list_from_eggfile()
2010-11-05 11:24:33 -02:00
Pablo Hoffman
de4909faca
get_spider_list_from_eggfile(): more improvements to error messages, and support passing eggruner module as argument
2010-11-04 18:56:11 -02:00
Pablo Hoffman
7ba972d8cf
get_spider_list_from_eggfile(): fail if unable to extract spider list
2010-11-04 16:27:47 -02:00
Pablo Hoffman
369a149d60
fixed bug with deploy command in windows environments: WindowsError: [Error 32] The process cannot access the file because it is being used by another process
2010-11-03 17:16:08 -02:00
Pablo Hoffman
a6c6dfe9ba
Automated merge with http://hg.scrapy.org/scrapy-0.10
2010-11-01 16:10:40 -02:00
Pablo Hoffman
1a2146d11a
Fixed bug detecting response types for some urls. Closes #281
2010-11-01 16:09:33 -02:00
Pablo Hoffman
0f69e7a191
Some changes to HTTP Cache middleware:
...
* made it use the project data storage by default (closes #279 )
* added HTTPCACHE_ENABLED setting (False by default) to enable it
* made HTTPCACHE_DIR = 'httpcache' by default (inside the project data storage)
* simplified HTTPCACHE_EXPIRATION_SECS semantics: zero means don't expire,
dropped support for negative numbers
* other minor doc improvements
2010-11-01 02:38:15 -02:00
Pablo Hoffman
3c94c6cb9b
fixed sphinx doc id
2010-11-01 02:31:20 -02:00
Pablo Hoffman
0b7e815888
Added Didier to AUTHORS
2010-11-01 01:25:30 -02:00
dfdeshom
130276605b
Bind the web server and telnet server to a configurable interface (WEBSERVICE_HOST). The default is to bind to all interfaces. Also add documentation for WEBSERVICE_HOST and TELNETCONSOLE_HOST.
2010-11-01 00:59:04 -02:00
Pablo Hoffman
b76c5c597f
* Added support for project data storage ( closes #276 )
...
* Documented project file structure
* Moved default location of SQLite database to project data storage dir (closes #277 )
2010-10-31 03:25:37 -02:00
Pablo Hoffman
dfa6745e91
Automated merge with http://hg.scrapy.org/scrapy-0.10
2010-10-30 16:05:53 -02:00
Pablo Hoffman
a0d9b43031
fixed typo in scrapyd doc
2010-10-30 16:05:32 -02:00
Pablo Hoffman
3d96016da1
runtests.sh: switched to 'text' repoter in trial
2010-10-30 16:03:00 -02:00
Pablo Hoffman
f73449fed6
removed LxmlItemLoader, as it has been obsoleted by the new lxml selector backend
2010-10-30 16:02:14 -02:00
Pablo Hoffman
d67152ab0f
Automated merge with http://hg.scrapy.org/scrapy-0.10
2010-10-30 01:56:12 -02:00
Pablo Hoffman
75451cbe84
scrapyd doc: fixed delversion.json example
2010-10-30 01:56:00 -02:00
Pablo Hoffman
20efdc0273
added --egg argument to scrapy deploy command, and log message when building the egg
2010-10-29 16:21:36 -02:00
Pablo Hoffman
85dec82688
Automated merge with http://hg.scrapy.org/scrapy-0.10
2010-10-29 03:42:54 -02:00
Pablo Hoffman
1cc5cba69b
Fixed bug logging Passed items. Closes #274
2010-10-29 03:42:21 -02:00
Pablo Hoffman
836e40896a
minor fixes for python 2.5 compatibility
2010-10-29 02:23:10 -02:00
Pablo Hoffman
22283854d4
avoid stripping trailing spaces on lxml-based selectors. closes #270
2010-10-27 21:39:28 -02:00
Pablo Hoffman
7f646541c3
added trackref stats to memory debugger report. closes #272
2010-10-27 21:18:58 -02:00
Pablo Hoffman
1d5c56089c
Automated merge with http://hg.scrapy.org/scrapy-0.10
2010-10-27 14:42:47 -02:00
Pablo Hoffman
c3e5b4bb03
changed pid file name to scrapyd
2010-10-27 14:42:22 -02:00
Pablo Hoffman
2bba87f69f
Automated merge with http://hg.scrapy.org/scrapy-0.10
2010-10-27 14:20:21 -02:00
Pablo Hoffman
f47b9f608c
simplified lockfile used by scrapyd (/var/run/scrapyd.pid instead of /var/run/scrapyd/scrapyd.pid). closes #271
2010-10-27 14:18:40 -02:00
Daniel Grana
bc2d78406c
MediaPipeline fails to assign crawler.engine.download as download function because crawler is configured after pipelines are loaded
2010-10-27 12:36:24 -02:00
Pablo Hoffman
158f75450b
Automated merge with http://hg.scrapy.org/scrapy-0.10
2010-10-27 09:18:48 -02:00
Pablo Hoffman
6f4be21d4c
changed robots.txt forbidden log level to DEBUG. closes #268
2010-10-27 09:17:58 -02:00
Pablo Hoffman
e625a8d56e
moved all similar selector tests to common selector tests, to reuse them among all backends
2010-10-27 08:54:32 -02:00
Pablo Hoffman
d1f63237ad
refactored selectors tests, by splitting tests in: common tests, lxml-specific tests and libxml2-specific tests. refs #147
2010-10-27 08:37:02 -02:00
Pablo Hoffman
665578bfe8
fixed imports in scrapy.xlib.simplejson
2010-10-27 08:05:08 -02:00
Pablo Hoffman
9b9ab37804
fixed bug with boolean results in lxml-based selectors
2010-10-27 08:03:37 -02:00
Pablo Hoffman
a8be54a8ea
scrapyd: make Environment tests independent of the current OS environment
...
--HG--
rename : scrapyd/tests/test_envion.py => scrapyd/tests/test_environ.py
2010-10-27 06:49:15 -02:00
Pablo Hoffman
17a6adde1f
some refactoring to selectors code, to reuse more code between lxml and libxml2 backends (refs #147 ). also added tests for dummy backend
2010-10-27 06:19:49 -02:00
Pablo Hoffman
9c9a655cb4
selectors: no need to pass encoding on re() method
2010-10-27 05:53:46 -02:00
Pablo Hoffman
bd7def8fd4
lxml selectors: cache and reuse XPathEvaluator object, for performance. refs #147
2010-10-27 05:51:30 -02:00
Pablo Hoffman
1ead888db8
make trial run doctests
2010-10-26 20:50:41 -02:00
Pablo Hoffman
a3a108dc71
fixed some compatibility issues with python 2.5 in scrapyd
2010-10-26 17:21:43 -02:00
Pablo Hoffman
b7c9503d9c
disable lxml selectors tests which was failing on certain versions of lxml-libxml2
2010-10-26 16:50:34 -02:00
Pablo Hoffman
a09dd18bbb
use absolute imports for compatibility with python 2.5
2010-10-26 16:37:16 -02:00
Martin Olveyra
41a85f9d14
Added support for variants when applied to tag attributes. fixed handling of variants with single attribute. Removed unneeded object attribute surrounds_variant. Added new tests cases for fixes.
2010-10-26 16:11:04 -02:00
Daniel Grana
a5a6c2ae31
Automated merge with ssh://hg.scrapy.org/scrapy-0.10
2010-10-25 15:57:56 -02:00
Daniel Grana
99096cdacf
Image pipeline should upload images with image/jpeg content type. closes #257
2010-10-25 15:56:07 -02:00
