BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-25 14:03:41 +00:00
Code Releases Wiki Activity
2,532 Commits 39 Branches 119 Tags
Commit Graph

2532 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Martin Olveyra
fafaee51d5 htmlpage tests reorganization and fixes: improved how differences between expected and result are shown, and check also correct parsing of tag_type 2010-10-07 14:55:00 -02:00
Pablo Hoffman
571aeb559b Automated merge with http://hg.scrapy.org/scrapy-0.10 2010-10-05 12:44:41 -02:00
Pablo Hoffman
4bbcbd7b77 Don't fail if twisted is not available on scrapy/__init__.py, to avoid making setup.py depend on Twisted. Closes #256 2010-10-05 12:43:34 -02:00
Pablo Hoffman
2d40705ea0 CloseSpider extension: Added support for closing spider after N errors have been raised. Closes #254 2010-09-30 20:17:44 -03:00
Pablo Hoffman
b6e7a38a3a Automated merge with http://hg.scrapy.org/scrapy-0.10 2010-09-29 13:37:33 -03:00
Pablo Hoffman
61ab9b86b7 Bumped version to 0.10.4 2010-09-29 13:36:36 -03:00
Pablo Hoffman
ad0f180dd9 Added tag 0.10.3 for changeset 803efdb19e0b 2010-09-29 13:34:56 -03:00
Pablo Hoffman
d15a97ff61 Updated Scrapy version in debian/changelog 2010-09-28 16:45:05 -03:00
Pablo Hoffman
7826869cb2 Added missing colon 2010-09-28 16:44:53 -03:00
Martin Santos
0bf9e4627c added support to CloseSpider extension, for close the spider after N pages have been crawled. Using the CLOSESPIDER_PAGECOUNT setting. closes #253 2010-09-28 16:29:37 -03:00
Pablo Hoffman
0976e0788e Automated merge with http://hg.scrapy.org/scrapy-0.10 2010-09-27 12:27:58 -03:00
Pablo Hoffman
49ffe528a3 Fixed listen_tcp function when receiving None or 0 in portrange argument. Closes #252 0.10.3 2010-09-27 12:27:32 -03:00
Pablo Hoffman
50e57b08b0 Automated merge with http://hg.scrapy.org/scrapy-0.10 2010-09-27 08:20:05 -03:00
Pablo Hoffman
9206806770 setup.py: added support for generating version from hg revision 2010-09-27 08:19:32 -03:00
Pablo Hoffman
51325fc93e Automated merge with http://hg.scrapy.org/scrapy-0.10 2010-09-27 07:57:23 -03:00
Pablo Hoffman
52d198afc9 Removed forked cookielib tests, because Python cookielib has been suffering several changes and maintaining a fork of the tests has become a pain. Instead, we've added specific tests for the urllib2 request/response wrappers 2010-09-27 07:55:27 -03:00
Pablo Hoffman
3db8d367c4 Another fix to tests, broken by recent Python changes 2010-09-26 21:09:14 -03:00
Pablo Hoffman
e55ae718e3 Automated merge with http://hg.scrapy.org/scrapy-0.10 2010-09-26 17:00:10 -03:00
Pablo Hoffman
5d4e0655d0 Fixed test_utils_url, broken on Windows after recent Python urllib change. Closes #251 2010-09-26 16:57:42 -03:00
Pablo Hoffman
279dcc245f Fixed role name in Sphinx doc 2010-09-26 01:01:06 -03:00
Pablo Hoffman
fd84d8d5c1 Automated merge with http://hg.scrapy.org/scrapy-0.10 2010-09-25 21:20:15 -03:00
Pablo Hoffman
1ef2cd400c Fixed issue with unicode keyword arguments bug in earlier Python versions (see http://bugs.python.org/issue2646). Closes #250 2010-09-25 21:17:36 -03:00
Pablo Hoffman
622834bc08 Removed unused imports, and use crawler.settings instead of scrapy.conf.settings in Scrapy Shell 2010-09-23 14:01:22 -03:00
Pablo Hoffman
b78284b680 Fixed spider variable not properly populated in the Scrapy shell 2010-09-23 13:59:41 -03:00
Pablo Hoffman
318f7f4c58 Added support for passing code to evaluate in Scrapy shell command (closes #249) and simplified handling of shell errors 2010-09-23 13:49:29 -03:00
Pablo Hoffman
a5ee05e814 Added support for setting exit code in Scrapy commands. Closes #248 2010-09-23 13:43:21 -03:00
Pablo Hoffman
79c0e34968 Simplified CrawlerSettings.__str__() 2010-09-23 12:50:46 -03:00
Pablo Hoffman
754d0f53f9 Fixed unbounded spider error in shell, and enclosed fetch() method in a try/except block for logging errors more reliably 2010-09-23 12:33:24 -03:00
Pablo Hoffman
37c25fe9a8 Fixed CrawlerSettings.__str__() method when settings_module is None 2010-09-23 12:32:49 -03:00
Pablo Hoffman
f29b346f79 Fixed access to settings module name, broken after recent changes to Settings classes 2010-09-22 22:21:29 -03:00
Pablo Hoffman
9599bde3e9 Removed RequestLimitMiddleware 2010-09-22 16:09:13 -03:00
Pablo Hoffman
97d77c79c2 Added tests for CrawlerSettings and SpiderSettings classes 2010-09-22 16:09:13 -03:00
Pablo Hoffman
ed4aec187f Ported code to use new unified access to spider settings, keeping backwards compatibility for old spider attributes. Refs #245 2010-09-22 16:09:13 -03:00
Pablo Hoffman
2459d20cc0 Added support for unifying access to per-spider settings. Refs #245 2010-09-22 16:09:13 -03:00
Pablo Hoffman
a4639ffb06 Removed hacky SCRAPY_SETTINGS_DISABLED environment variable 2010-09-22 16:08:18 -03:00
Pablo Hoffman
b6c2b55e5b Splitted settings classes from settings singleton. Closes #244 
--HG--
rename : scrapy/conf/__init__.py => scrapy/conf.py
rename : scrapy/conf/default_settings.py => scrapy/settings/default_settings.py
rename : scrapy/tests/test_conf.py => scrapy/tests/test_settings.py
 2010-09-22 15:47:33 -03:00
Pablo Hoffman
1c20a5e5c7 Automated merge with http://hg.scrapy.org/scrapy-0.10 2010-09-22 15:44:11 -03:00
Pablo Hoffman
b11e11037f Make custom project commands discovery work with eggified projects. Fixes #247 2010-09-22 15:42:31 -03:00
Pablo Hoffman
2ebfa7e68d Removed unneeded code (since autodoc is not used in Sphinx doc) 2010-09-22 10:52:02 -03:00
Pablo Hoffman
f3769651af Refactored Scrapyd code to fix a couple of bugs that ocurred when running projects without eggs 2010-09-22 01:04:15 -03:00
Pablo Hoffman
0ffc8d29b6 Don't silence warnings after log is started. Closes #246 2010-09-22 01:00:34 -03:00
Pablo Hoffman
be3293df52 Automated merge with http://hg.scrapy.org/scrapy-0.10 2010-09-20 08:48:42 -03:00
Pablo Hoffman
4c61df7abb get_spider_list_from_eggfile(): fixed bug when SCRAPY_SETTINGS_DISABLED is set 2010-09-20 08:47:55 -03:00
Pablo Hoffman
8329491296 Added Shuaib Khan to AUTHORS 2010-09-20 08:34:22 -03:00
Shuaib
9288f622f9 Added formname parameter for FormRequest.from_response 2010-09-20 08:33:24 -03:00
Pablo Hoffman
400c4134af Make scrapyd.eggutils compatible with Python 2.5 and added tests for get_spider_list_from_eggfile() function (closes #242) 2010-09-19 21:08:27 -03:00
Pablo Hoffman
b6a95f943e Automated merge with http://hg.scrapy.org/scrapy-0.10 2010-09-19 19:54:17 -03:00
Ping Yin
aa86f180b9 BaseSgmlLinkExtractor: Fix extract '>>' as '>' 
The anchor text is extracted as '>' in
<a href="/">&gt;&gt;</a>

Signed-off-by: Ping Yin <pkufranky@gmail.com>
 2010-09-19 19:28:35 -03:00
Ping Yin
f41fd3214e test_engine.py: fix typo brwoser => browser 
Signed-off-by: Ping Yin <pkufranky@gmail.com>
 2010-09-19 19:28:13 -03:00
Pablo Hoffman
a8325a3017 Automated merge with http://hg.scrapy.org/scrapy-0.10 2010-09-17 17:21:32 -03:00