* remove debug messages
* move deactivating of downloads to last callback
* simplify calling of download_any function
* raises RuntimeError while openinng/closing twice
* removed starters from engine
* moved schedulermiddleware to scheduler
* raise RuntimeError when trying to open/close a scheduler domain twice
* removed dupesfilter singleton and spidermw dupefilter middleware
--HG--
extra : rebase_source : e4c3ad4b970cbc8f532bc751ba9d8a944ca16be5
spider level. A new spider (InitSpider, not yet documented) was added to
provide initialization facilities.
Also renamed make_request_from_url to make_requests_from_url and allowed it to
return iterables.
Execution Manager:
* added control_reactor argument to delegate external twisted
reactor control (for example by twistd)
* now it loads spiders (if not already loaded)
* now it stars the log (if not already started)
* removed *args from configure() method
* removed **opts from runonce and start methods
Execution engine:
* added control_reactor argument to to delegate external twisted
reactor control (for example by twistd)
* changed some functions and method names for clarity
* improve handling of exceptions in st() method
* regrouped close_domain, closed_domain, and _close_domain method
for legibilty
Scheduler:
* replaced pending_domains_count (dict) by pending_domains (set)
* simplified some doc
Execution Manager:
* added control_reactor argument to delegate external twisted
reactor control (for example by twistd)
* now it loads spiders (if not already loaded)
* now it stars the log (if not already started)
* removed *args from configure() method
* removed **opts from runonce and start methods
Execution engine:
* added control_reactor argument to to delegate external twisted
reactor control (for example by twistd)
* changed some functions and method names for clarity
* improve handling of exceptions in st() method
* regrouped close_domain, closed_domain, and _close_domain method
for legibilty
Scheduler:
* replaced pending_domains_count (dict) by pending_domains (set)
* simplified some doc
constructor (domain prioritizers will be refactored later anyway) and
simplified Scrapy Manager code thanks to this.
Added make_request_from_url method to BaseSpider, splitting funtionality to
create requests from URLs which was previously done all in start_requests.
