mirror of
https://github.com/scrapy/scrapy.git
synced 2025-02-26 17:04:07 +00:00
RetryMiddleware: added ConnectionLost to retried exceptions
twisted >8.0 has a ConnectionClosed exception parent of ConnectionLost and ConnectionDone, but twisted 2.5 hasn't. I add ConnectionLost until we can move forward to twisted >8.0 this is the docstring of ConnectionLost, hopes it is self explanatory: """Connection to the other side was lost in a non-clean fashion""" --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40757
This commit is contained in:
parent
bef1fa967a
commit
8ff4dc4d02
@ -2,7 +2,7 @@
|
||||
An extension to retry failed requests that are potentially caused by temporary
|
||||
problems such as a connection timeout or HTTP 500 error.
|
||||
|
||||
You can change the behaviour of this moddileware by modifing the scraping settings:
|
||||
You can change the behaviour of this moddileware by modifing the scraping settings:
|
||||
RETRY_TIMES - how many times to retry a failed page
|
||||
RETRY_HTTP_CODES - which HTTP response codes to retry
|
||||
|
||||
@ -21,12 +21,13 @@ About HTTP errors to consider:
|
||||
indicate server overload, which would be something we want to retry
|
||||
- 200 is included by default (and shoudln't be removed) to check for partial
|
||||
downloads errors, which means the TCP connection has broken in the middle of
|
||||
a HTTP download
|
||||
"""
|
||||
a HTTP download
|
||||
"""
|
||||
|
||||
from twisted.internet.error import TimeoutError as ServerTimeoutError, DNSLookupError, \
|
||||
ConnectionRefusedError, ConnectionDone, ConnectError
|
||||
from twisted.internet.defer import TimeoutError as UserTimeoutError
|
||||
ConnectionRefusedError, ConnectionDone, ConnectError, \
|
||||
ConnectionLost
|
||||
from twisted.internet.defer import TimeoutError as UserTimeoutError
|
||||
|
||||
from scrapy import log
|
||||
from scrapy.core.exceptions import HttpException
|
||||
@ -35,8 +36,9 @@ from scrapy.conf import settings
|
||||
|
||||
class RetryMiddleware(object):
|
||||
|
||||
EXCEPTIONS_TO_RETRY = (ServerTimeoutError, UserTimeoutError, DNSLookupError,
|
||||
ConnectionRefusedError, ConnectionDone, ConnectError)
|
||||
EXCEPTIONS_TO_RETRY = (ServerTimeoutError, UserTimeoutError, DNSLookupError,
|
||||
ConnectionRefusedError, ConnectionDone, ConnectError,
|
||||
ConnectionLost)
|
||||
|
||||
def __init__(self):
|
||||
self.failed_count = {}
|
||||
|
Loading…
x
Reference in New Issue
Block a user