BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-27 10:04:58 +00:00
Code Releases Wiki Activity

fix name errors on robots.txt middleware during spider_close

Browse Source
This commit is contained in:
ajones1@gmail.com 2009-11-25 12:59:28 -08:00
parent 1cfd959828
commit 0dcf47a706
1 changed files with 2 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
scrapy/contrib/downloadermiddleware/robotstxt.py
View File

@ -58,7 +58,7 @@ class RobotsTxtMiddleware(object):
or settings['USER_AGENT'] or settings['USER_AGENT']
def spider_closed(self, spider): def spider_closed(self, spider):
for netloc in self._spider_netlocs[domain]: for netloc in self._spider_netlocs[spider]:
del self._parsers[netloc] del self._parsers[netloc]
del self._spider_netlocs[domain] del self._spider_netlocs[spider]
del self._useragents[spider] del self._useragents[spider]