Elias Dorneles
57a5ee0097
added example value to set for proxy meta key
2015-03-12 23:20:44 -03:00
Elias Dorneles
f7031c08ff
updating list of Request.meta special keys
2015-03-10 22:29:07 -03:00
Pablo Hoffman
675fd5ba04
Merge pull request #898 from scrapy/download-timeout
...
[MRG] DOC document download_timeout
2014-10-24 16:52:42 -02:00
Pablo Hoffman
0dce283459
Merge pull request #893 from kmike/less-ads
...
[MRG] DOC simplify extension docs
2014-10-21 17:13:59 -02:00
Mikhail Korobov
7d68b084a4
DOC document download_timeout Request.meta key and download_timeout spider attribute.
2014-10-07 04:23:11 +06:00
Mikhail Korobov
36eec8f413
dont_obey_robotstxt meta key; don't process requests to /robots.txt
2014-09-23 00:10:43 +06:00
Mikhail Korobov
e435b3e3a3
DOC simplify extension docs
2014-09-21 00:19:24 +06:00
Rocio Aramberri
51b0bd281d
fix dont settings on meta behaviour, add docs and tests
2014-08-15 13:47:42 -07:00
Daniel Graña
a90fd81c97
add leveldb cache backend docs
2014-06-23 14:37:29 -03:00
Mikhail Korobov
2d3803672b
DOC use top-level shortcuts in docs
2014-04-15 01:09:35 +06:00
Nikolaos-Digenis Karagiannis
43a797e2f7
downloaderMW doc typo (spiderMW doc copy remnant)
2014-02-11 22:30:00 +02:00
Rolando Espinoza
a6279fe95b
DOC Fixed HTTPCACHE_STORAGE typo in the default value which is now Filesystem instead Dbm.
2014-01-30 11:53:42 -04:00
Mikhail Korobov
7b7a1d8dfd
Make Filesystem storage backend default again. See GH-500.
2014-01-17 04:32:08 +06:00
Mikhail Korobov
b03fe04999
Rename AjaxCrawlableMiddleware to AjaxCrawlMiddleware
2014-01-16 23:09:37 +06:00
Pablo Hoffman
ed6fd4933f
Merge pull request #524 from hobsonlane/master
...
documentation code example corrections per pablohoffman
2014-01-16 06:44:51 -08:00
Daniel Graña
b9bb9bed6b
Merge pull request #343 from kmike/ajax-crawlable
...
[MRG] AjaxCrawlableMiddleware
2014-01-16 05:07:39 -08:00
Hobson Lane
6ba0857a5c
documentation code example correction corrections per pablohoffman
2014-01-10 10:37:27 -08:00
Mikhail Korobov
a27d91f0a6
Rename BaseSpider to Spider. See GH-495.
2013-12-30 19:46:41 +06:00
Mikhail Korobov
943a0bd264
AjaxCrawlableMiddleware in Broad Crawl docs
2013-12-19 01:01:26 +06:00
Mikhail Korobov
a87b3bd1c8
AjaxCrawlableMiddleware
2013-12-19 00:06:47 +06:00
Travis Briggs
3043a5ba37
DownloaderMiddleware docs: Update process_request, proper explanation of IgnoreRequest.
...
Also:
* Change terminology to eliminate uses of terms such as "request middleware" to refer to the process_request methods of installed middleware.
* Remove description of "immediate redirection", as it is misleading.
Further changes.
2013-10-17 00:23:21 +00:00
Rocio Aramberri
d227d530f6
Added COMPRESSION_ENABLED setting to enable or disable the HttpCompressionMiddleware
...
Added COMPRESSION_ENABLE setting to docs
Added COMPRESSION_ENABLED setting to default settings
2013-08-01 11:31:28 -03:00
Mikhail Korobov
ac2fadf3ab
DownloaderMiddleware.process_response docs fix
...
"returns an exception" -> "raises an exception"
2013-07-08 19:41:58 +06:00
Mikhail Korobov
39e5da5f66
improve docs for DownloaderMiddleware.process_response
2013-07-08 19:17:29 +06:00
nramirezuy
83b2774354
remove wrong default httpcache
2013-06-25 17:01:29 -03:00
Pablo Hoffman
96c2332e0e
fix inaccurate downloader middleware documentation. refs #280
2013-04-02 11:35:32 -03:00
Pablo Hoffman
7400ceb1ed
added 502 to RETRY_HTTP_CODES
2013-02-22 19:12:59 -02:00
Chris Tilden
aae6aed4fb
fixes spelling errors in documentation
2013-01-22 14:52:18 -08:00
Pablo Hoffman
c31441a273
revert default HTTP cache policy to dummy (instead of RFC2616)
2013-01-17 13:08:29 -02:00
Daniel Graña
75563b3f00
Add list of supported and missing RFC2616 caching features
2013-01-08 18:16:44 -02:00
Daniel Graña
d8a760bf57
Merge branch 'http-cache-middleware'
...
Conflicts:
scrapy/contrib/downloadermiddleware/httpcache.py
scrapy/contrib/httpcache.py
scrapy/tests/test_downloadermiddleware_httpcache.py
2013-01-08 17:34:48 -02:00
Daniel Graña
864a7aef87
More httpcache updates
...
* Change default cache policy to RFC2616
* Update HttpCacheMiddleware documentation
* Move policies to scrapy.contrib.httpcache
* remove a lint error for .has_key() usage in DBM storage backend
2013-01-08 17:26:32 -02:00
Daniel Graña
defc4f89b5
update metarefresh settings
2013-01-08 11:41:19 -02:00
Daniel Graña
6a2b23883a
Add MetaRefreshMiddleware docs
2013-01-08 11:25:38 -02:00
Pedro Faustino
5d3a4d755f
Update downloader middleware documentation
2013-01-06 18:53:14 +00:00
Hasnain Lakhani
93a1102189
Implemented policies for HTTP Cache
2012-12-26 16:29:48 -08:00
Pedro Faustino
fdaa35f6e8
Updated the downloader middleware documentation to reflect changes introduced by the support for real HTTP caching.
2012-12-24 19:37:53 +01:00
stav
99f164fc87
correct docs for default storage backend
2012-11-22 14:05:47 -06:00
Pablo Hoffman
eed6eb49da
make DBM the new default storage backend for HTTP cache middleware, simplified DBM storage backend code to avoid dealing with many spiders at once (not needed), and update httpcache stats names (hit -> hits, miss -> misses)
2012-09-17 10:11:07 -03:00
Pablo Hoffman
2b16ebdc11
added minor clarification on cookiejar request meta key usage
2012-02-29 07:19:01 -02:00
Pablo Hoffman
81abb45000
fixed bug in new cookiejar documentation
2012-02-28 11:08:25 -02:00
Pablo Hoffman
26c8004125
added documentation for the new cookiejar Request.meta key
2012-02-27 19:58:58 -02:00
Pablo Hoffman
19e6da59d8
added new downloader middleware: ChunkedTransferMiddleware
2011-08-09 03:03:25 -03:00
Pablo Hoffman
39a2ea97c8
redirect mw: added REDIRECT_ENABLED setting and documented the other settings
2011-07-13 14:18:15 -03:00
Pablo Hoffman
541ed3913b
retry middleware: added RETRY_ENABLED setting and documented the other settings more properly, also improved messages when no longer retrying requests
2011-07-13 11:55:05 -03:00
Pablo Hoffman
2fa0f75f2d
added COOKIES_ENABLED setting to support disabling the cookies middleware
2011-05-27 00:35:34 -03:00
Pablo Hoffman
3ee2c94e93
Improved cookies middleware by making COOKIES_DEBUG nicer and documenting it
2011-04-06 14:54:48 -03:00
Pablo Hoffman
3954e600ca
added DBM storage backend for HTTP cache
2011-03-23 21:32:02 -03:00
Pablo Hoffman
8f7e163b04
Fixed wrong method name in downloader middleware documentation
2011-02-24 15:26:32 -02:00
Pablo Hoffman
91a7c25797
* Made Response.meta attribute map to Request.meta attribute. Closes #290
...
* Record redirected URLs in redirect middleware. Closes #291
2010-11-18 12:51:54 -02:00
