BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-28 20:38:10 +00:00
Code Releases Wiki Activity
3,050 Commits 39 Branches 119 Tags
Commit Graph

3050 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Pablo Hoffman
0f69e7a191 Some changes to HTTP Cache middleware: 
* made it use the project data storage by default (closes #279)
* added HTTPCACHE_ENABLED setting (False by default) to enable it
* made HTTPCACHE_DIR = 'httpcache' by default (inside the project data storage)
* simplified HTTPCACHE_EXPIRATION_SECS semantics: zero means don't expire,
  dropped support for negative numbers
* other minor doc improvements
 2010-11-01 02:38:15 -02:00
Pablo Hoffman
3c94c6cb9b fixed sphinx doc id 2010-11-01 02:31:20 -02:00
Pablo Hoffman
0b7e815888 Added Didier to AUTHORS 2010-11-01 01:25:30 -02:00
dfdeshom
130276605b Bind the web server and telnet server to a configurable interface (WEBSERVICE_HOST). The default is to bind to all interfaces. Also add documentation for WEBSERVICE_HOST and TELNETCONSOLE_HOST. 2010-11-01 00:59:04 -02:00
Pablo Hoffman
b76c5c597f * Added support for project data storage (closes #276) 
* Documented project file structure
* Moved default location of SQLite database to project data storage dir (closes #277)
 2010-10-31 03:25:37 -02:00
Pablo Hoffman
dfa6745e91 Automated merge with http://hg.scrapy.org/scrapy-0.10 2010-10-30 16:05:53 -02:00
Pablo Hoffman
a0d9b43031 fixed typo in scrapyd doc 2010-10-30 16:05:32 -02:00
Pablo Hoffman
3d96016da1 runtests.sh: switched to 'text' repoter in trial 2010-10-30 16:03:00 -02:00
Pablo Hoffman
f73449fed6 removed LxmlItemLoader, as it has been obsoleted by the new lxml selector backend 2010-10-30 16:02:14 -02:00
Pablo Hoffman
d67152ab0f Automated merge with http://hg.scrapy.org/scrapy-0.10 2010-10-30 01:56:12 -02:00
Pablo Hoffman
75451cbe84 scrapyd doc: fixed delversion.json example 2010-10-30 01:56:00 -02:00
Pablo Hoffman
20efdc0273 added --egg argument to scrapy deploy command, and log message when building the egg 2010-10-29 16:21:36 -02:00
Pablo Hoffman
85dec82688 Automated merge with http://hg.scrapy.org/scrapy-0.10 2010-10-29 03:42:54 -02:00
Pablo Hoffman
1cc5cba69b Fixed bug logging Passed items. Closes #274 2010-10-29 03:42:21 -02:00
Pablo Hoffman
836e40896a minor fixes for python 2.5 compatibility 2010-10-29 02:23:10 -02:00
Pablo Hoffman
22283854d4 avoid stripping trailing spaces on lxml-based selectors. closes #270 2010-10-27 21:39:28 -02:00
Pablo Hoffman
7f646541c3 added trackref stats to memory debugger report. closes #272 2010-10-27 21:18:58 -02:00
Pablo Hoffman
1d5c56089c Automated merge with http://hg.scrapy.org/scrapy-0.10 2010-10-27 14:42:47 -02:00
Pablo Hoffman
c3e5b4bb03 changed pid file name to scrapyd 2010-10-27 14:42:22 -02:00
Pablo Hoffman
2bba87f69f Automated merge with http://hg.scrapy.org/scrapy-0.10 2010-10-27 14:20:21 -02:00
Pablo Hoffman
f47b9f608c simplified lockfile used by scrapyd (/var/run/scrapyd.pid instead of /var/run/scrapyd/scrapyd.pid). closes #271 2010-10-27 14:18:40 -02:00
Daniel Grana
bc2d78406c MediaPipeline fails to assign crawler.engine.download as download function because crawler is configured after pipelines are loaded 2010-10-27 12:36:24 -02:00
Pablo Hoffman
158f75450b Automated merge with http://hg.scrapy.org/scrapy-0.10 2010-10-27 09:18:48 -02:00
Pablo Hoffman
6f4be21d4c changed robots.txt forbidden log level to DEBUG. closes #268 2010-10-27 09:17:58 -02:00
Pablo Hoffman
e625a8d56e moved all similar selector tests to common selector tests, to reuse them among all backends 2010-10-27 08:54:32 -02:00
Pablo Hoffman
d1f63237ad refactored selectors tests, by splitting tests in: common tests, lxml-specific tests and libxml2-specific tests. refs #147 2010-10-27 08:37:02 -02:00
Pablo Hoffman
665578bfe8 fixed imports in scrapy.xlib.simplejson 2010-10-27 08:05:08 -02:00
Pablo Hoffman
9b9ab37804 fixed bug with boolean results in lxml-based selectors 2010-10-27 08:03:37 -02:00
Pablo Hoffman
a8be54a8ea scrapyd: make Environment tests independent of the current OS environment 
--HG--
rename : scrapyd/tests/test_envion.py => scrapyd/tests/test_environ.py
 2010-10-27 06:49:15 -02:00
Pablo Hoffman
17a6adde1f some refactoring to selectors code, to reuse more code between lxml and libxml2 backends (refs #147). also added tests for dummy backend 2010-10-27 06:19:49 -02:00
Pablo Hoffman
9c9a655cb4 selectors: no need to pass encoding on re() method 2010-10-27 05:53:46 -02:00
Pablo Hoffman
bd7def8fd4 lxml selectors: cache and reuse XPathEvaluator object, for performance. refs #147 2010-10-27 05:51:30 -02:00
Pablo Hoffman
1ead888db8 make trial run doctests 2010-10-26 20:50:41 -02:00
Pablo Hoffman
a3a108dc71 fixed some compatibility issues with python 2.5 in scrapyd 2010-10-26 17:21:43 -02:00
Pablo Hoffman
b7c9503d9c disable lxml selectors tests which was failing on certain versions of lxml-libxml2 2010-10-26 16:50:34 -02:00
Pablo Hoffman
a09dd18bbb use absolute imports for compatibility with python 2.5 2010-10-26 16:37:16 -02:00
Martin Olveyra
41a85f9d14 Added support for variants when applied to tag attributes. fixed handling of variants with single attribute. Removed unneeded object attribute surrounds_variant. Added new tests cases for fixes. 2010-10-26 16:11:04 -02:00
Daniel Grana
a5a6c2ae31 Automated merge with ssh://hg.scrapy.org/scrapy-0.10 2010-10-25 15:57:56 -02:00
Daniel Grana
99096cdacf Image pipeline should upload images with image/jpeg content type. closes #257 2010-10-25 15:56:07 -02:00
Pablo Hoffman
df63577243 added lxml to setup.py install_requires, if using setuptools. refs #147 2010-10-25 14:57:50 -02:00
Pablo Hoffman
f7283ad18e skip lxml selector tests if lxml is not available. refs #147 2010-10-25 14:56:33 -02:00
Pablo Hoffman
a59bfb539d * Added lxml backend for XPath selectors. Closes #147 
* Added new setting (SELECTORS_BACKEND) to choose which backend to use
* Deprecated the extract_unquoted() function from selectors
* Made libxml2 optional by adding a dummy selector backend. Closes #260

--HG--
rename : scrapy/tests/test_selector.py => scrapy/tests/test_selector_libxml2.py
 2010-10-25 14:47:10 -02:00
Daniel Grana
7640e99979 test media_to_download mediapipeline hook. ref #269 2010-10-25 13:16:11 -02:00
Daniel Grana
fe1dd3a93d disconnect signals before uninstalling crawler in image tests 2010-10-23 05:10:52 -02:00
Daniel Grana
ad43917322 Add tests to MediaPipeline. closes #269 
--HG--
extra : rebase_source : ccf726e147b5c97f7cba60d20ce2fca58c687a3e
 2010-10-23 04:44:55 -02:00
Martin Olveyra
d17edd4a59 Fix wrong slice of tokens in recursive extraction of follow region 2010-10-22 14:21:52 -02:00
Daniel Grana
72d08383bb Automated merge with ssh://hg.scrapy.org/scrapy-0.10 2010-10-22 00:28:35 -02:00
Daniel Grana
2873c4d9fe SimpleDB stats doesn't use AWS auth from settings.py (thanks geoffwatts). closes #264 2010-10-22 00:21:54 -02:00
Pablo Hoffman
992683ac5c Deploy command requires project 2010-10-21 13:24:02 -02:00
Pablo Hoffman
f8b4d1dc5d Fixed compatibility with Python 2.5 2010-10-21 12:53:40 -02:00