BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-24 02:04:22 +00:00
Code Releases Wiki Activity
2,358 Commits 39 Branches 119 Tags
Commit Graph

2358 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Pablo Hoffman
b16eff227b Bumped version to 0.10 2010-09-10 01:48:45 -03:00
Pablo Hoffman
9b9b3a2b64 Removed unused import 2010-09-09 23:35:51 -03:00
Pablo Hoffman
7d14a52234 Reference dont_merge_cookies in list of special Request.meta keys 2010-09-09 21:54:26 -03:00
Pablo Hoffman
7f21a6384f Documented handle_httpstatus_list request.meta key 2010-09-09 21:50:40 -03:00
Pablo Hoffman
f1c943543a Added dont_retry request.meta key to make RetryMiddleware ignore requests. Closes #234 2010-09-09 21:43:44 -03:00
Pablo Hoffman
9f01e3e79e Added dont_redirect request.meta key to make RedirectMiddleware ignore requests. Closes #233 2010-09-09 21:37:35 -03:00
Pablo Hoffman
ad2b979e0f Changed TextResponse.replace() behaviour by keeping previous encoding when not specified 2010-09-09 18:01:00 -03:00
Pablo Hoffman
b9ea9dcf1a Restored wrongly removed import 2010-09-09 17:42:54 -03:00
Pablo Hoffman
c041328b45 Fixed bug with thread unsafety calls from Scrapy shell 2010-09-09 15:26:46 -03:00
Pablo Hoffman
24c0088bde Added tag 0.10-rc1 for changeset 131f8e906f75 2010-09-09 00:41:18 -03:00
Pablo Hoffman
38bbec8dca Bumped version to 0.10-rc1 0.10-rc1 2010-09-09 00:40:58 -03:00
Pablo Hoffman
fe77794f76 Added SpiderContext extension. Closes #203 2010-09-08 15:52:35 -03:00
Pablo Hoffman
1d5a7df125 Fixed encoding error with some item dropped messages. Closes #232 2010-09-08 15:34:13 -03:00
Pablo Hoffman
cb7cc7c8bb Fixed bugs with SQLiteDict and added tests 2010-09-08 14:32:28 -03:00
Pablo Hoffman
7da79b90fe Make url/body attributes of Request/Response objects read-only - use replace() to change them. Deprecation warning left for backwards compatibilty. 2010-09-08 00:15:11 -03:00
Pablo Hoffman
c1aab2f58e Copy callback/errback attributes when copying Requests 2010-09-08 00:15:09 -03:00
Pablo Hoffman
e9ebebb230 Removed UrlFilterMiddleware from scrapy.contrib - see this snippet for an alternative: http://snippets.scrapy.org/snippets/12/ 2010-09-07 17:51:02 -03:00
Pablo Hoffman
8ba6fa2096 Fixed bugs introduced by previous scheduler middleware change 2010-09-07 17:47:47 -03:00
Pablo Hoffman
dbc0ba7236 Added class constructors scrapy.contrib.ibl FieldDescriptor and ItemDescriptor 2010-09-07 14:26:50 -03:00
Pablo Hoffman
23b13f0cc8 Simplified SchedulerMiddlewareManager (to be removed in Scrapy 0.11) by making it inherit from MiddlewareManager 2010-09-07 13:10:49 -03:00
Pablo Hoffman
5918d0163e Automated merge with ssh://hg.scrapy.org:2222/scrapy 2010-09-07 13:03:16 -03:00
Pablo Hoffman
c40e5a7491 Moved module: scrapy.core.downloader.manager to scrapy.core.downloader 
--HG--
rename : scrapy/core/downloader/manager.py => scrapy/core/downloader/__init__.py
 2010-09-07 13:02:15 -03:00
Daniel Grana
12b04b068f make download_timeout configurable by request. closes #229 
--HG--
extra : rebase_source : e57dfd4aeb98d48b04fc4d0c6469e9a85e4b33a8
 2010-09-07 13:01:40 -03:00
Pablo Hoffman
a23b6981e8 Simplified DownloaderMiddlewareManager by making it inherit from MiddlewareManager 2010-09-07 13:00:36 -03:00
Pablo Hoffman
37f4ce24ea Fixed spider middleware order for methods: process_spider_output, process_spider_exception 2010-09-07 12:59:02 -03:00
Pablo Hoffman
d0081290f2 Improve error logging when failing to create a spider in the execution queue 2010-09-07 10:07:47 -03:00
Pablo Hoffman
9158e9d682 Some changes to Scrapyd to support multiple configuration files, to make it easier to deploy Scrapyd applications. Also documented 'egg_runner' and 'application' options 
--HG--
rename : debian/scrapyd.cfg => debian/000-default
rename : scrapyd/default_scrapyd.cfg => scrapyd/default_scrapyd.conf
 2010-09-07 09:17:25 -03:00
Daniel Grana
3414bf13ee remove request_uploaded signal and move response_received and response_downloaded to downloader manager. closes #228 
--HG--
extra : rebase_source : 4af0d2a01b34de8a21048bb7f4a66bfc484b3b8f
 2010-09-06 23:23:14 -03:00
Pablo Hoffman
3c5ab10688 Added FAQ entry about __VIEWSTATE parameter 2010-09-06 13:17:08 -03:00
Pablo Hoffman
3a72e5c051 Removed settings.disabled hack used in some tests. Closes #143 2010-09-06 11:04:27 -03:00
Pablo Hoffman
5f58af2005 Simplified SpiderMiddlewareManager by making it inherit from MiddlewareManager 2010-09-06 10:40:33 -03:00
Pablo Hoffman
cc72f03e10 Added IFeedStorage interface and test all Feed Storages conform to it. Also added test for StdoutFeedStorage 2010-09-06 10:22:28 -03:00
Pablo Hoffman
e3d67d74f7 docs/intro/overview.rst: add example of scraped data and introduce loaders 2010-09-06 10:04:00 -03:00
Pablo Hoffman
ff9de424c8 Added SpiderQueue tests. SQS spider queue not tested because operations take too long to complete and it's not easy to know when they have. Closes #227 2010-09-06 09:47:45 -03:00
Daniel Grana
8d1d3493e7 Added a weak key factory based cache 
--HG--
extra : rebase_source : 2bc7cb5fdb0fd3adb63cf7fe3aedd2f1d15e49f0
 2010-09-06 00:50:56 -03:00
Pablo Hoffman
00d55fbbd1 Updated 'Scrapy at a glance' document replacing item pipeline example by a simpler usage of feed exports 2010-09-05 23:38:37 -03:00
Pablo Hoffman
5ffc7650bd Removed code no longer needed 2010-09-05 20:08:59 -03:00
Pablo Hoffman
766f2d910d Renamed Request Handlers to Download Handlers 2010-09-05 19:35:53 -03:00
Pablo Hoffman
067ec65d97 Removed download_any singleton 2010-09-05 19:09:42 -03:00
Pablo Hoffman
a5cf71cb06 Updated Ubuntu package signing key location 2010-09-05 19:04:15 -03:00
Pablo Hoffman
6bf52fb50e Make telnet console and web service try a range of ports for binding, instead of just one. Closes #226 2010-09-05 06:48:08 -03:00
Pablo Hoffman
ce884192c9 Fixed test broken by previous commit 2010-09-05 06:05:34 -03:00
Pablo Hoffman
630db4fecf Simplified file:// download handler, adding support for reading binary files 2010-09-05 05:59:40 -03:00
Pablo Hoffman
14e985b076 Updated Command line tool documentation 2010-09-05 05:29:58 -03:00
Pablo Hoffman
1190f97944 Updated settings documentation 2010-09-05 04:58:14 -03:00
Pablo Hoffman
ebdb733e95 Updated some old messages in Scrapy shell doc 2010-09-05 04:45:43 -03:00
Pablo Hoffman
2f12618890 Post reference to Scrapyd in FAQ 2010-09-05 04:35:27 -03:00
Pablo Hoffman
a66bef7925 Make execution queue poll interval configurable through a new QUEUE_POLL_INTERVAL setting 2010-09-05 02:23:08 -03:00
Pablo Hoffman
b800fdcb4d SqliteSpiderQueue: failback to in-memory SQLite if database cannot be opened (typically due to missing write permissions) 2010-09-04 03:49:46 -03:00
Pablo Hoffman
bf34094e5a Added versionadded:: notice to new documentation topics 2010-09-04 03:30:45 -03:00