BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-23 04:23:47 +00:00
Code Releases Wiki Activity
280 Commits 39 Branches 119 Tags
Commit Graph

280 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
samus_
c07937c4df added comment 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40280
 2008-09-25 12:49:19 +00:00
samus_
c4aa1e7e8b small fix to the regex 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40279
 2008-09-25 12:45:43 +00:00
Daniel Grana
555cb0940b images: use brief exception logging 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40278
 2008-09-25 12:22:31 +00:00
olveyra
c73cbdd9b1 allow to override item class adaptor in constructor 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40277
 2008-09-25 02:14:02 +00:00
samus_
79485ceb5c added support for xml-declared encodings 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40276
 2008-09-24 18:19:19 +00:00
Damian Canabal
a4b709e03f added test for remove entities 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40275
 2008-09-24 12:40:55 +00:00
Damian Canabal
4e7e539b03 html entities regexp improvement 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40274
 2008-09-24 12:25:10 +00:00
Pablo Hoffman
79be2ca97d moved scrapy.core.log module to scrapy.log 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40273
 2008-09-23 22:18:20 +00:00
Pablo Hoffman
fbec2b3a43 moved scrapy.core.mail module to scrapy.mail 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40272
 2008-09-23 21:52:05 +00:00
Pablo Hoffman
cb8e0d9bdd added scrapy.utils.defer, moved deferred functions from scrapy.utils.misc to that module 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40271
 2008-09-23 21:48:48 +00:00
Pablo Hoffman
6637d8cf68 removed location_str function incorrectly added to this module 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40270
 2008-09-23 21:34:05 +00:00
samus_
dbfc6341c2 removed duplicated function convert_entity from scrapy.utils.misc 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40269
 2008-09-23 20:36:32 +00:00
german
4842fe0679 removed unquote_html 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40268
 2008-09-22 18:04:30 +00:00
Matias Aguirre
2b6ed80a66 Change site-media with static in css file 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40267
 2008-09-19 20:00:08 +00:00
Matias Aguirre
e985b6c368 Comment blog templatetags calls 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40266
 2008-09-19 19:57:45 +00:00
Matias Aguirre
d3f66b7d39 Remove blog url an installed app setting 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40265
 2008-09-19 19:53:57 +00:00
Matias Aguirre
bd28045b99 Remove blog application which is not django 1.0 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40264
 2008-09-19 19:19:28 +00:00
Matias Aguirre
4d388c946e Django 1.0 support over download application 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40263
 2008-09-19 19:17:41 +00:00
Daniel Grana
62975817af remove decobot reference and fix test_plugin 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40262
 2008-09-19 16:57:57 +00:00
Daniel Grana
baabdb004d remove decobot reference 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40261
 2008-09-19 16:57:21 +00:00
Daniel Grana
d87f979d2a grrr.. stupid bugfix 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40260
 2008-09-19 15:21:47 +00:00
eduardo
c05dcdab1d removed references to python2.5 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40259
 2008-09-19 14:11:14 +00:00
olveyra
ddd7393a0a reverted changeset 254 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40258
 2008-09-19 12:50:19 +00:00
olveyra
332460851b ouch! reverted a failed commit and now yes, removed an unneeded None 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40257
 2008-09-18 20:19:26 +00:00
olveyra
133e9242d3 removed unneeded None 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40256
 2008-09-18 20:13:31 +00:00
olveyra
6ce7ae4df8 allow to set download delay as a scrapy setting 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40255
 2008-09-18 20:11:30 +00:00
eduardo
cfb56e39bb bump version 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40254
 2008-09-18 19:21:40 +00:00
eduardo
c6d4a37a98 include templates 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40253
 2008-09-18 17:48:35 +00:00
Daniel Grana
818d9baf69 new setup.py 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40252
 2008-09-18 16:08:33 +00:00
eduardo
a85e53053d in the way of setuptooling 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40251
 2008-09-18 14:58:43 +00:00
Daniel Grana
93c313853d use request.deferred for cached valued too 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40250
 2008-09-17 12:34:48 +00:00
Daniel Grana
6f2d36199b media: use deferred of request to allow adding callback from get_media_requests 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40249
 2008-09-16 23:40:48 +00:00
Daniel Grana
7932a8a5b6 add image pipeline that stores and can thumbs images in different sizes 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40248
 2008-09-16 23:40:06 +00:00
Daniel Grana
4807f2eb82 move media_to_download hook to allow caching and prevent downloading 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40247
 2008-09-16 23:39:26 +00:00
Daniel Grana
5048491f8d mediapipeline: change cache variable by domaininfo 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40246
 2008-09-16 19:31:25 +00:00
Daniel Grana
41a4f22c27 messy typo adding errback 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40245
 2008-09-16 14:03:39 +00:00
anibal
e0d90f9d87 Scrapy architecture diagram 1st version, dia source 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40244
 2008-09-16 11:56:15 +00:00
Pablo Hoffman
0a7a5e5b3d added docs dir, guess for what? :) 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40243
 2008-09-16 11:30:22 +00:00
Pablo Hoffman
b57148dfad renamed scrapy.lib to scrapy.xlib 
--HG--
rename : scrapy/trunk/scrapy/lib/BeautifulSoup.py => scrapy/trunk/scrapy/xlib/BeautifulSoup.py
rename : scrapy/trunk/scrapy/lib/__init__.py => scrapy/trunk/scrapy/xlib/__init__.py
rename : scrapy/trunk/scrapy/lib/lrucache.py => scrapy/trunk/scrapy/xlib/lrucache.py
rename : scrapy/trunk/scrapy/lib/lsprofcalltree.py => scrapy/trunk/scrapy/xlib/lsprofcalltree.py
rename : scrapy/trunk/scrapy/lib/pydispatch/__init__.py => scrapy/trunk/scrapy/xlib/pydispatch/__init__.py
rename : scrapy/trunk/scrapy/lib/pydispatch/dispatcher.py => scrapy/trunk/scrapy/xlib/pydispatch/dispatcher.py
rename : scrapy/trunk/scrapy/lib/pydispatch/errors.py => scrapy/trunk/scrapy/xlib/pydispatch/errors.py
rename : scrapy/trunk/scrapy/lib/pydispatch/license.txt => scrapy/trunk/scrapy/xlib/pydispatch/license.txt
rename : scrapy/trunk/scrapy/lib/pydispatch/robust.py => scrapy/trunk/scrapy/xlib/pydispatch/robust.py
rename : scrapy/trunk/scrapy/lib/pydispatch/robustapply.py => scrapy/trunk/scrapy/xlib/pydispatch/robustapply.py
rename : scrapy/trunk/scrapy/lib/pydispatch/saferef.py => scrapy/trunk/scrapy/xlib/pydispatch/saferef.py
rename : scrapy/trunk/scrapy/lib/spidermonkey/INSTALL.scrapy => scrapy/trunk/scrapy/xlib/spidermonkey/INSTALL.scrapy
rename : scrapy/trunk/scrapy/lib/spidermonkey/__init__.py => scrapy/trunk/scrapy/xlib/spidermonkey/__init__.py
rename : scrapy/trunk/scrapy/lib/spidermonkey/sm_settings.py => scrapy/trunk/scrapy/xlib/spidermonkey/sm_settings.py
rename : scrapy/trunk/scrapy/lib/spidermonkey/spidermonkey.py => scrapy/trunk/scrapy/xlib/spidermonkey/spidermonkey.py
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40242
 2008-09-15 15:37:44 +00:00
Pablo Hoffman
2cb300b926 removed simplejson from scrapy code 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40241
 2008-09-15 15:29:12 +00:00
Daniel Grana
08feb4df1c ername methods and enforce extraction of Request objects as item media to download 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40240
 2008-09-15 13:55:23 +00:00
Pablo Hoffman
0090fcb0f5 improved fix to support IE7 (thanks bill\!) 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40239
 2008-09-15 13:08:44 +00:00
Pablo Hoffman
d7a88e8d77 fixed hack for IE6 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40238
 2008-09-15 12:58:19 +00:00
Pablo Hoffman
31d3b2cf5a added hack for IE6 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40237
 2008-09-15 12:56:31 +00:00
Daniel Grana
4944762341 move bugtraps to avoid silence of bugs in media_downloaded or media_failed 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40236
 2008-09-15 12:46:34 +00:00
Daniel Grana
ca695c8c79 small bugfix and add bugtraps 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40235
 2008-09-15 08:57:51 +00:00
Daniel Grana
6407a744e3 remove item.image_urls from get_urls_from_item 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40234
 2008-09-15 08:27:15 +00:00
Daniel Grana
af0808be95 add info object to public methods 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40233
 2008-09-15 07:08:27 +00:00
Daniel Grana
4263c3a2e1 Adds docstrings to public methods, remove unused imports, normalize urls to requests. 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40232
 2008-09-15 06:01:40 +00:00
Daniel Grana
31e497e5eb add media_to_download hook support, and return cached result if available 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40231
 2008-09-15 04:33:40 +00:00