Execution Manager:
* added control_reactor argument to delegate external twisted
reactor control (for example by twistd)
* now it loads spiders (if not already loaded)
* now it stars the log (if not already started)
* removed *args from configure() method
* removed **opts from runonce and start methods
Execution engine:
* added control_reactor argument to to delegate external twisted
reactor control (for example by twistd)
* changed some functions and method names for clarity
* improve handling of exceptions in st() method
* regrouped close_domain, closed_domain, and _close_domain method
for legibilty
Scheduler:
* replaced pending_domains_count (dict) by pending_domains (set)
* simplified some doc
Execution Manager:
* added control_reactor argument to delegate external twisted
reactor control (for example by twistd)
* now it loads spiders (if not already loaded)
* now it stars the log (if not already started)
* removed *args from configure() method
* removed **opts from runonce and start methods
Execution engine:
* added control_reactor argument to to delegate external twisted
reactor control (for example by twistd)
* changed some functions and method names for clarity
* improve handling of exceptions in st() method
* regrouped close_domain, closed_domain, and _close_domain method
for legibilty
Scheduler:
* replaced pending_domains_count (dict) by pending_domains (set)
* simplified some doc
constructor (domain prioritizers will be refactored later anyway) and
simplified Scrapy Manager code thanks to this.
Added make_request_from_url method to BaseSpider, splitting funtionality to
create requests from URLs which was previously done all in start_requests.
in this changeset:
* remove HttpException from engine and core exceptions
* replace dwmw ErrorPages with spidermw HttpError
* bugfix image pipeline media_to_download method when stat_key returns None
* documented
* moved from scrapy.contrib.downloadermiddleware.cache.CacheMiddleware to
scrapy.contrib.downloadermiddleware.httpcache.HttpCacheMiddleware
* settings prefix changed from CACHE2_ to HTTPCACHE_
--HG--
rename : scrapy/contrib/downloadermiddleware/cache.py => scrapy/contrib/downloadermiddleware/httpcache.py
scrapy.contrib.linkextractors.
The most relevant being:
scrapy.link.extractors.RegexLinkExtractor
which was moved to:
scrapy.contrib.linkextractors.sgml.SgmlLinkExtractor
The old location still works but throws a deprecation warning. It will be
removed before the 0.7 release.
Documentation and tests were also updated.
Also, in this changeset, a new regex-based link extractor was added to
scrapy.contrib.linkextractors.regex.
--HG--
rename : scrapy/tests/sample_data/link_extractor/regex_linkextractor.html => scrapy/tests/sample_data/link_extractor/sgml_linkextractor.html
rename : scrapy/tests/test_link.py => scrapy/tests/test_contrib_linkextractors.py
