BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-26 15:43:44 +00:00
Code Releases Wiki Activity

fix to filter start_urls too

Browse Source 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40869
This commit is contained in:
samus_ 2009-02-19 04:19:05 +00:00
parent 9739f9320e
commit de13bbe7f5
1 changed files with 1 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
scrapy/trunk/scrapy/contrib/spidermiddleware/duplicatesfilter.py
View File

@ -40,6 +40,7 @@ class DuplicatesFilterMiddleware(object):
def process_spider_output(self, response, result, spider): def process_spider_output(self, response, result, spider):
domain = spider.domain_name domain = spider.domain_name
self.filter.add(domain, response.request)
for req in result: for req in result:
if isinstance(req, Request): if isinstance(req, Request):