BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-27 17:03:55 +00:00
Code Releases Wiki Activity
425 Commits 39 Branches 119 Tags
Commit Graph

425 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Ismael Carnales
89bd6ede43 first commit of doc apps 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40425
 2008-11-25 18:16:04 +00:00
Ismael Carnales
4241171947 style h1s 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40424
 2008-11-25 17:42:10 +00:00
Daniel Grana
c61d0648e6 use hashlib instead of deprecated md5 module 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40423
 2008-11-25 17:17:25 +00:00
Daniel Grana
81416c4dd1 s3: add image md5 content checksum to image path 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40422
 2008-11-25 17:09:26 +00:00
Pablo Hoffman
33b20be7b5 cleanup handling of adaptor pipe debugging code 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40421
 2008-11-25 17:06:36 +00:00
Ismael Carnales
e8df55d101 added header and footer templates 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40420
 2008-11-25 16:23:37 +00:00
elpolilla
011adf4dd7 Turned "strip_list" adaptor into "strip" 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40419
 2008-11-25 15:27:04 +00:00
Daniel Grana
1a60f6c1cb comment default thumb sizes 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40418
 2008-11-25 13:44:06 +00:00
Daniel Grana
f61be3f078 fix syntax typo 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40417
 2008-11-25 11:58:31 +00:00
Daniel Grana
072fed8d4a aws: request signing function and tests 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40416
 2008-11-25 11:06:34 +00:00
Pablo Hoffman
0fbb7579f6 added new CrawlSpider that's gonna replace the old CrawlSpider soon 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40415
 2008-11-25 02:38:26 +00:00
elpolilla
f558a49d2c Fixed ExtractImages adaptor, which wasnt looking for a base tag 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40414
 2008-11-25 01:41:22 +00:00
Pablo Hoffman
5028146e8c added __slots__ to decrease memory footprint of Link objects 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40413
 2008-11-25 00:39:11 +00:00
elpolilla
a8903d5877 Changed deletion of _adaptor_dict attribute in parse command that could trigger exceptions 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40412
 2008-11-24 18:31:05 +00:00
elpolilla
68e497e532 Fixed some little bugs in adaptors 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40411
 2008-11-24 13:09:08 +00:00
Ismael Carnales
3cbfb554d2 home must be the fist link in a left-to-right lecture world 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40410
 2008-11-24 12:51:22 +00:00
Ismael Carnales
859733d8ab added link to insophia page 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40409
 2008-11-24 12:48:45 +00:00
Ismael Carnales
3dab0c12a0 use relative links for images 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40408
 2008-11-24 12:40:59 +00:00
elpolilla
e9e8fa00d9 Added check to the previous commit, because it could have failed 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40407
 2008-11-24 12:38:41 +00:00
elpolilla
8f01a70ee7 Removed ugly printing of adaptors dict in parse command 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40406
 2008-11-24 12:35:26 +00:00
elpolilla
82cf911192 Added method for inserting adaptors into a pipeline 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40405
 2008-11-24 12:05:23 +00:00
elpolilla
0f9eba8a97 Removed AdaptorDict and added AdaptorPipe, plus the adaptize helper 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40404
 2008-11-24 11:28:18 +00:00
Daniel Grana
cbffced594 add amamzon ws helper module 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40403
 2008-11-24 10:59:59 +00:00
Daniel Grana
2142d4f7e0 s3 image pipeline: use scrapyengine for downloads 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40402
 2008-11-24 10:19:26 +00:00
Daniel Grana
c4434590b0 images pipeline factorization 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40401
 2008-11-24 10:18:55 +00:00
Daniel Grana
98bc8ec3e1 spidermiddleware: remove not used "filter" hook 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40400
 2008-11-24 10:18:27 +00:00
Daniel Grana
b56a5d933e clean spidermiddleware callback/errback handling 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40399
 2008-11-24 10:17:56 +00:00
Daniel Grana
c7b3fae97a add basic spider template 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40398
 2008-11-24 10:17:32 +00:00
Daniel Grana
0938984de0 media pipeline: no need for complex deferred execution 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40397
 2008-11-24 10:17:02 +00:00
Daniel Grana
a6ff165332 media pipeline: dont hide list of results for item_complete 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40396
 2008-11-24 10:16:33 +00:00
Pablo Hoffman
5ccd4df0cd moved log method to BaseSpider, moved adapt_feed doc to docstring 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40395
 2008-11-23 01:32:21 +00:00
samus_
d4040d73d9 added docstring 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40394
 2008-11-22 19:20:53 +00:00
Daniel Grana
1458686cbf media pipeline: support deferred execution of media_to_download hook 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40393
 2008-11-21 10:37:32 +00:00
elpolilla
c15a24c25e Added --force option switch to genspider command 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40392
 2008-11-21 10:08:22 +00:00
samus_
836c532ef1 some cleanup to the identify mechanism 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40391
 2008-11-20 15:36:48 +00:00
elpolilla
17d45f5216 Added CSVFeedSpider 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40390
 2008-11-20 14:34:59 +00:00
elpolilla
77a917df6b Modified FOLLOW_LINKS setting name to a more appropiate one 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40389
 2008-11-20 13:10:06 +00:00
elpolilla
99c4c06380 Added --nofollow option switch to crawl command, and its functionality into CrawlSpiders 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40388
 2008-11-20 12:10:36 +00:00
elpolilla
90fe6ef1b8 Fixed bug that didnt show extracted links in new spiders parse command 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40387
 2008-11-19 12:29:04 +00:00
Daniel Grana
56e3d5ae66 Revert "Modified image pipeline to make it able to manage 304 responses" 
This reverts commit 3ff5303de7a3d2cffce0244b5bd9aa2b0f5c9c83.

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40386
 2008-11-18 17:37:25 +00:00
elpolilla
d5469eabb2 Modified image pipeline to make it able to manage 304 responses 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40385
 2008-11-18 15:55:27 +00:00
elpolilla
9a7b5a7cbf Improved genspider command 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40384
 2008-11-18 11:34:03 +00:00
Daniel Grana
862def2d73 move teplate renderer function to utils package, it can be reused by genspider command 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40383
 2008-11-17 10:33:49 +00:00
Daniel Grana
d6d32fa672 substitute scrapy_settings template variables on startproject 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40382
 2008-11-17 10:33:15 +00:00
elpolilla
bddfbee0d4 Fixed some unicode issues in adaptors, improved docstrings and a test 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40381
 2008-11-14 13:36:18 +00:00
elpolilla
c209f214a8 -Fixed bug in ExtractImages adaptor that made it fail if it received a string 
-Removed BasicSpider and it's guid generation method because it wasnt generic enough to be in the framework

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40380
 2008-11-14 12:25:56 +00:00
elpolilla
f5eb71fb69 Fixed bug in url_query_cleaner that returned wrong parameters for urls with fragments and added test 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40379
 2008-11-14 01:32:49 +00:00
Pablo Hoffman
79c9ac388d updated scrapy.utils.misc.hash_values to drop usage of deprecated sha module in favor or hashlib, added tests to hash_values function 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40378
 2008-11-13 15:50:27 +00:00
elpolilla
01b157d10b Reverted delist adaptor 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40377
 2008-11-13 12:27:36 +00:00
elpolilla
5f4053d9fc Reverted to revision 370 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40376
 2008-11-13 12:20:32 +00:00