mirror of
https://github.com/scrapy/scrapy.git
synced 2025-02-26 11:44:11 +00:00
409aaade0b
downloading (and enqueued for download) requests to finish and their responses to be processed in the scraper/spiders, before closing the spider. This will be required in the future to avoid loosing requests when we add scheduler persistence and it's also a more correct behaviour overall. The closing process has also been refactored to remove unneeded closing state from downloader and leave it only in the engine. Finally, some unused methods has been removed too, like spider_is_open() for engine and scheduler.
This is Scrapy, an opensource screen scraping framework written in Python. For more info visit the project home page at http://scrapy.org