* delay is enforced per website, not per spider;
* document download_delay attribute (it was previously documented only in FAQ about 999 error codes);
* document how CONCURRENT_REQUESTS_PER_IP affects download delays.
This is the salient point of this code compared to the last example. We have a selector now and this is how we use it. Especially since the user has just come from the shell where the pre-instantiated selector is taken for granted.
Also:
* Change terminology to eliminate uses of terms such as "request middleware" to refer to the process_request methods of installed middleware.
* Remove description of "immediate redirection", as it is misleading.
Further changes.
Ported .get() from _Element and .text_content() from HTMLMixin
Add CSS selectors to scrapy shell
Documenting CSS Selectors: Constructing selectors
Documenting CSS Selectors: Using Selectors
Make CSS Selectors a default feature.
Adds XPath powers to CSS Selectors and some syntactic sugar.
Removes methods copied over from lxml.html.HtmlMixin.
Updating docs to use new CSS Selector super powers.
Documenting CSS Selectors: Regular Expressions
Moving section after Nesting section, since it mentions it.
Documenting CSS Selectors: Nesting Selectors
Fix XPath specificity in lxml.selector.CSSSelectorMixin.text
Cleaning up unused stuff from cssel.py
Changing the behavior of lxml.selector.CSSSelectorMixin.text.
Concatenating all of the descendant text nodes is more useful
than returning it in pieces (there's xpath() if you need that).
Documenting CSS Selectors: CSS Selector objects
Documenting CSS Selectors: CSSSelectorList objects
Documenting CSS Selectors: HtmlCSSSelector objects
Documenting CSS Selectors: XmlCSSSelector objects
Fixing some documentations typos and errors
Enforcing the 80-char width lines
Tidying up CSS selectors and CSSSelectorMixin objects
Adding some missing references in documentation.
Fixing lxml.selector.CSSSelectorList.text
With this modification scrapy runs the spider with project settings. The previous example ran only with default settings resulting in ignoring all user settings as pipelines for example.
