BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-22 15:03:46 +00:00
Code Releases Wiki Activity
7,850 Commits 39 Branches 119 Tags
Commit Graph

7850 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Daniel Grana
98bc8ec3e1 spidermiddleware: remove not used "filter" hook 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40400
 2008-11-24 10:18:27 +00:00
Daniel Grana
b56a5d933e clean spidermiddleware callback/errback handling 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40399
 2008-11-24 10:17:56 +00:00
Daniel Grana
c7b3fae97a add basic spider template 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40398
 2008-11-24 10:17:32 +00:00
Daniel Grana
0938984de0 media pipeline: no need for complex deferred execution 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40397
 2008-11-24 10:17:02 +00:00
Daniel Grana
a6ff165332 media pipeline: dont hide list of results for item_complete 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40396
 2008-11-24 10:16:33 +00:00
Pablo Hoffman
5ccd4df0cd moved log method to BaseSpider, moved adapt_feed doc to docstring 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40395
 2008-11-23 01:32:21 +00:00
samus_
d4040d73d9 added docstring 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40394
 2008-11-22 19:20:53 +00:00
Daniel Grana
1458686cbf media pipeline: support deferred execution of media_to_download hook 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40393
 2008-11-21 10:37:32 +00:00
elpolilla
c15a24c25e Added --force option switch to genspider command 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40392
 2008-11-21 10:08:22 +00:00
samus_
836c532ef1 some cleanup to the identify mechanism 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40391
 2008-11-20 15:36:48 +00:00
elpolilla
17d45f5216 Added CSVFeedSpider 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40390
 2008-11-20 14:34:59 +00:00
elpolilla
77a917df6b Modified FOLLOW_LINKS setting name to a more appropiate one 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40389
 2008-11-20 13:10:06 +00:00
elpolilla
99c4c06380 Added --nofollow option switch to crawl command, and its functionality into CrawlSpiders 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40388
 2008-11-20 12:10:36 +00:00
elpolilla
90fe6ef1b8 Fixed bug that didnt show extracted links in new spiders parse command 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40387
 2008-11-19 12:29:04 +00:00
Daniel Grana
56e3d5ae66 Revert "Modified image pipeline to make it able to manage 304 responses" 
This reverts commit 3ff5303de7a3d2cffce0244b5bd9aa2b0f5c9c83.

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40386
 2008-11-18 17:37:25 +00:00
elpolilla
d5469eabb2 Modified image pipeline to make it able to manage 304 responses 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40385
 2008-11-18 15:55:27 +00:00
elpolilla
9a7b5a7cbf Improved genspider command 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40384
 2008-11-18 11:34:03 +00:00
Daniel Grana
862def2d73 move teplate renderer function to utils package, it can be reused by genspider command 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40383
 2008-11-17 10:33:49 +00:00
Daniel Grana
d6d32fa672 substitute scrapy_settings template variables on startproject 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40382
 2008-11-17 10:33:15 +00:00
elpolilla
bddfbee0d4 Fixed some unicode issues in adaptors, improved docstrings and a test 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40381
 2008-11-14 13:36:18 +00:00
elpolilla
c209f214a8 -Fixed bug in ExtractImages adaptor that made it fail if it received a string 
-Removed BasicSpider and it's guid generation method because it wasnt generic enough to be in the framework

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40380
 2008-11-14 12:25:56 +00:00
elpolilla
f5eb71fb69 Fixed bug in url_query_cleaner that returned wrong parameters for urls with fragments and added test 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40379
 2008-11-14 01:32:49 +00:00
Pablo Hoffman
79c9ac388d updated scrapy.utils.misc.hash_values to drop usage of deprecated sha module in favor or hashlib, added tests to hash_values function 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40378
 2008-11-13 15:50:27 +00:00
elpolilla
01b157d10b Reverted delist adaptor 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40377
 2008-11-13 12:27:36 +00:00
elpolilla
5f4053d9fc Reverted to revision 370 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40376
 2008-11-13 12:20:32 +00:00
elpolilla
fab0e6938c Removed unnecesary check in RobustScrapedItem's getattr 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40375
 2008-11-13 11:01:44 +00:00
elpolilla
4afa29b26a Fixed adaptors test 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40374
 2008-11-13 10:46:43 +00:00
elpolilla
4ef7c46303 Added ItemAttribute objects 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40373
 2008-11-13 10:35:24 +00:00
elpolilla
90d28badf6 Turned unquote adaptor into a function, and added adaptor factory 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40372
 2008-11-13 10:13:11 +00:00
elpolilla
36080ada86 Added some missing docstrings to adaptors 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40371
 2008-11-13 10:04:04 +00:00
elpolilla
323241b6d6 Reverted changes in r368 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40370
 2008-11-11 12:51:39 +00:00
elpolilla
c5d134053b Added possibility of checking whether a response should or shouldnt be parsed with the corresponding parse_suffix method (by defining check_suffix) in CrawlSpider 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40369
 2008-11-10 15:30:45 +00:00
elpolilla
6db6b8c52e GUID wasnt being set when calling a spider's parse_url method (from the parse command). Fixed that. 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40368
 2008-11-10 12:26:11 +00:00
samus_
22d713bba9 seems scrapy uses a test spider (scrapy.tests.test_spiders.testplugin.TestSpider) moving the test to decobot 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40367
 2008-11-08 13:34:02 +00:00
samus_
3e88890bc3 added check for valid code on spiders 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40366
 2008-11-08 12:26:54 +00:00
Damian Canabal
5e54ac52ff fixed new_response_from_xpaths function, unicode string was passed as body instead of ResponseBody object 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40365
 2008-11-04 17:45:26 +00:00
elpolilla
a2461fbeea Wrong attribute assignation fixed again 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40364
 2008-11-04 14:06:28 +00:00
elpolilla
224d3c5185 Bugfix in parse command 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40363
 2008-11-04 13:27:31 +00:00
elpolilla
5bad79836f Fixed bad checking while setting attributes 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40362
 2008-11-04 11:26:23 +00:00
elpolilla
bd38a312d4 - Fixed bug in attributes assignation (empty attributes being set) 
- Added GUID setting to FeedSpider

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40361
 2008-11-04 10:57:59 +00:00
samus_
9b46c20da2 moved xpathselector_iternodes from scrapy.utils.xml to scrapy.utils.iterators and renamed it to "xmliter", also renamed csv_iter to csviter and added tests 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40360
 2008-11-03 16:10:43 +00:00
elpolilla
1ef65b97b5 Fixed typo 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40359
 2008-11-03 14:00:33 +00:00
elpolilla
1a45754cf2 Removed an out-of-scrapy reference in the BasicSpider 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40358
 2008-11-03 13:57:21 +00:00
elpolilla
bc13a5924a Removed ugly loading of string codecs 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40357
 2008-11-03 12:28:00 +00:00
elpolilla
defcb45120 Implemented CrawlSpider and XMLFeedSpider classes 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40356
 2008-11-03 11:48:43 +00:00
elpolilla
f2bab50979 - Fixed bad implementation of the SetGUIDPipeline 
- Modified item's attribute method to have an optional 'add' argument
- Renamed normalize_urls adaptor to canonicalize_urls

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40355
 2008-11-03 11:02:58 +00:00
elpolilla
776818db71 Little change in test spider that broke the engine test 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40354
 2008-11-03 10:43:40 +00:00
elpolilla
0574bbd44a Added some improvements to LinkExtractor 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40353
 2008-11-03 10:24:50 +00:00
elpolilla
6a2e288f22 Added support for x-mac-roman string codec 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40352
 2008-10-31 11:52:35 +00:00
elpolilla
9882679bbb Bugfix in ExtractImages adaptor 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40351
 2008-10-29 02:25:32 +00:00