BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-23 16:44:17 +00:00
Code Releases Wiki Activity
2,358 Commits 39 Branches 119 Tags
Commit Graph

2358 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Pablo Hoffman
f47b9f608c simplified lockfile used by scrapyd (/var/run/scrapyd.pid instead of /var/run/scrapyd/scrapyd.pid). closes #271 2010-10-27 14:18:40 -02:00
Pablo Hoffman
6f4be21d4c changed robots.txt forbidden log level to DEBUG. closes #268 2010-10-27 09:17:58 -02:00
Daniel Grana
99096cdacf Image pipeline should upload images with image/jpeg content type. closes #257 2010-10-25 15:56:07 -02:00
Daniel Grana
2873c4d9fe SimpleDB stats doesn't use AWS auth from settings.py (thanks geoffwatts). closes #264 2010-10-22 00:21:54 -02:00
Pablo Hoffman
deb9c7ef04 Reversed scrapy.cfg lookup order so that the one in the current project has more precedence. Also added alternative system-wide location for windows. 2010-10-18 03:18:54 -02:00
Pablo Hoffman
98662e53ea Formatting fix in Scrapyd doc 2010-10-17 03:20:23 -02:00
Pablo Hoffman
5f65c26080 Some minor improvements to feature list in Scrapy at a Glance documentation page 2010-10-16 19:02:08 -02:00
Pablo Hoffman
f5b188b179 Make RetryMiddleware obey Request.meta 'dont_retry' key when processing exceptions. Closes #259 2010-10-11 21:28:42 -02:00
Pablo Hoffman
b4fbc6c5fa Updated Scrapy Tutorial to reference feed exports, instead a custom written pipeline, and extended item pipeline documentation to include a JSON writer. 2010-10-10 20:31:05 -02:00
Pablo Hoffman
0b91c04007 Fixed issue with non-standard line ending in HTTP headers. Closes #258 2010-10-09 20:43:05 -02:00
Pablo Hoffman
f4accb6c7f Updated dmoz xpaths of Scrapy tutorial 2010-10-07 18:22:01 -02:00
Pablo Hoffman
4bbcbd7b77 Don't fail if twisted is not available on scrapy/__init__.py, to avoid making setup.py depend on Twisted. Closes #256 2010-10-05 12:43:34 -02:00
Pablo Hoffman
61ab9b86b7 Bumped version to 0.10.4 2010-09-29 13:36:36 -03:00
Pablo Hoffman
ad0f180dd9 Added tag 0.10.3 for changeset 803efdb19e0b 2010-09-29 13:34:56 -03:00
Pablo Hoffman
49ffe528a3 Fixed listen_tcp function when receiving None or 0 in portrange argument. Closes #252 0.10.3 2010-09-27 12:27:32 -03:00
Pablo Hoffman
9206806770 setup.py: added support for generating version from hg revision 2010-09-27 08:19:32 -03:00
Pablo Hoffman
52d198afc9 Removed forked cookielib tests, because Python cookielib has been suffering several changes and maintaining a fork of the tests has become a pain. Instead, we've added specific tests for the urllib2 request/response wrappers 2010-09-27 07:55:27 -03:00
Pablo Hoffman
3db8d367c4 Another fix to tests, broken by recent Python changes 2010-09-26 21:09:14 -03:00
Pablo Hoffman
5d4e0655d0 Fixed test_utils_url, broken on Windows after recent Python urllib change. Closes #251 2010-09-26 16:57:42 -03:00
Pablo Hoffman
1ef2cd400c Fixed issue with unicode keyword arguments bug in earlier Python versions (see http://bugs.python.org/issue2646). Closes #250 2010-09-25 21:17:36 -03:00
Pablo Hoffman
b11e11037f Make custom project commands discovery work with eggified projects. Fixes #247 2010-09-22 15:42:31 -03:00
Pablo Hoffman
2ebfa7e68d Removed unneeded code (since autodoc is not used in Sphinx doc) 2010-09-22 10:52:02 -03:00
Pablo Hoffman
f3769651af Refactored Scrapyd code to fix a couple of bugs that ocurred when running projects without eggs 2010-09-22 01:04:15 -03:00
Pablo Hoffman
0ffc8d29b6 Don't silence warnings after log is started. Closes #246 2010-09-22 01:00:34 -03:00
Pablo Hoffman
4c61df7abb get_spider_list_from_eggfile(): fixed bug when SCRAPY_SETTINGS_DISABLED is set 2010-09-20 08:47:55 -03:00
Pablo Hoffman
8329491296 Added Shuaib Khan to AUTHORS 2010-09-20 08:34:22 -03:00
Shuaib
9288f622f9 Added formname parameter for FormRequest.from_response 2010-09-20 08:33:24 -03:00
Pablo Hoffman
400c4134af Make scrapyd.eggutils compatible with Python 2.5 and added tests for get_spider_list_from_eggfile() function (closes #242) 2010-09-19 21:08:27 -03:00
Ping Yin
aa86f180b9 BaseSgmlLinkExtractor: Fix extract '>>' as '>' 
The anchor text is extracted as '>' in
<a href="/">&gt;&gt;</a>

Signed-off-by: Ping Yin <pkufranky@gmail.com>
 2010-09-19 19:28:35 -03:00
Ping Yin
f41fd3214e test_engine.py: fix typo brwoser => browser 
Signed-off-by: Ping Yin <pkufranky@gmail.com>
 2010-09-19 19:28:13 -03:00
Pablo Hoffman
08952562e9 Make Ubuntu package conflict with standard Scrapy Debian/Ubuntu package 2010-09-17 16:25:02 -03:00
Daniel Grana
cad077574b Fix black shapes appearing round transparent images 2010-09-16 14:19:32 -03:00
Pablo Hoffman
4cecbcdc5b Fixed bug in Scrapyd launcher when running projects without eggs. Refs #238 2010-09-15 21:03:43 -03:00
Pablo Hoffman
a5908f052f Bumped version to 0.10.3 2010-09-15 21:03:21 -03:00
Pablo Hoffman
b33c8e49e2 Added tag 0.10.2 for changeset 68d08e757571 2010-09-15 14:48:47 -03:00
Pablo Hoffman
c01b8651ff Bumped version to 0.10.2 0.10.2 2010-09-15 14:04:33 -03:00
Pablo Hoffman
5fe9a49e51 Force recalculating the encoding on HTTPCompression middleware until we are sure the responsetypes guessing is reliable. Refs #239, #240 2010-09-15 13:57:17 -03:00
Pablo Hoffman
28f6bdf7bb Added tag 0.10.1 for changeset bb80107f959d 2010-09-15 02:44:41 -03:00
Pablo Hoffman
beee7e4fbc Merge with trunk 0.10.1 2010-09-15 01:50:42 -03:00
Pablo Hoffman
39499a2437 Fixed bug in HTTP Compression middleware which was failing to properly discover the encoding when the encoding was declared inside the response body. Closes #239. Also changed responsetypes to return Response class (instead of HtmlResponse) when the response has a Content-Encoding header 2010-09-14 20:22:25 -03:00
Pablo Hoffman
9acc99e723 Added 'inthread' decorator 2010-09-14 14:36:35 -03:00
Pablo Hoffman
c559b06a85 Removed unused import 2010-09-14 01:53:05 -03:00
Pablo Hoffman
833baa6041 Support running projects without eggs in Scrapyd. Closes #238 2010-09-14 01:44:25 -03:00
Pablo Hoffman
b76cd42690 Added tests for Scrapyd components. Closes #237 2010-09-14 01:44:10 -03:00
Pablo Hoffman
d0c7a946bd Bumped version to 0.10.1 2010-09-13 19:02:21 -03:00
Pablo Hoffman
1fa96dbcd3 Bumped version to 0.11 2010-09-13 19:02:15 -03:00
Pablo Hoffman
e2a3ab5141 Added tag 0.10 for changeset 3b32720c96a3 2010-09-13 18:49:33 -03:00
Pablo Hoffman
a3fc60cfcd Added missing mime.types files to MANIFEST.in 0.10 2010-09-11 04:30:08 -03:00
Pablo Hoffman
bf467fc37a Check 'dont_merge_cookies' membership in request.meta, instead of getting its value 2010-09-10 15:29:15 -03:00
Pablo Hoffman
e7a958a035 Fixed exception thrown when in FreeBSD when /proc exists but it's not mounted. Closes #235 2010-09-10 15:17:36 -03:00