Julia Medina
d7c444fefb
Move scrapy/contrib/downloadermiddleware to scrapy/downloadermiddlewares
2015-04-29 21:24:30 -03:00
Julia Medina
7a92dae4c8
Change Scrapy log output through docs
2015-04-22 17:27:24 -03:00
Pablo Hoffman
c81eefaf81
fix doc links
2015-03-19 17:42:48 -03:00
Julia Medina
4fb818a250
Run linkfix over current docs
2015-03-18 20:04:14 -03:00
Shadab Zafar
5a58d64131
Fix some redirection links in documentation
...
Fixes #606
2015-03-18 19:41:26 -03:00
Pablo Hoffman
934584a355
Merge pull request #1020 from jojje/gzip_http_cache
...
[MRG+1] add gzip compression to filesystem http cache backend
2015-03-17 14:32:06 -03:00
nramirezuy
c13e23641b
httpcache dont_cache meta #19 #689
2015-03-16 11:50:04 -03:00
Elias Dorneles
57a5ee0097
added example value to set for proxy meta key
2015-03-12 23:20:44 -03:00
Elias Dorneles
f7031c08ff
updating list of Request.meta special keys
2015-03-10 22:29:07 -03:00
Jonas Tingeborn
bd5d99a2d2
add gzip compression to filesystem http cache backend
2015-01-21 20:18:11 +01:00
Pablo Hoffman
675fd5ba04
Merge pull request #898 from scrapy/download-timeout
...
[MRG] DOC document download_timeout
2014-10-24 16:52:42 -02:00
Pablo Hoffman
0dce283459
Merge pull request #893 from kmike/less-ads
...
[MRG] DOC simplify extension docs
2014-10-21 17:13:59 -02:00
Mikhail Korobov
7d68b084a4
DOC document download_timeout Request.meta key and download_timeout spider attribute.
2014-10-07 04:23:11 +06:00
Mikhail Korobov
36eec8f413
dont_obey_robotstxt meta key; don't process requests to /robots.txt
2014-09-23 00:10:43 +06:00
Mikhail Korobov
e435b3e3a3
DOC simplify extension docs
2014-09-21 00:19:24 +06:00
Rocio Aramberri
51b0bd281d
fix dont settings on meta behaviour, add docs and tests
2014-08-15 13:47:42 -07:00
Daniel Graña
a90fd81c97
add leveldb cache backend docs
2014-06-23 14:37:29 -03:00
Mikhail Korobov
2d3803672b
DOC use top-level shortcuts in docs
2014-04-15 01:09:35 +06:00
Nikolaos-Digenis Karagiannis
43a797e2f7
downloaderMW doc typo (spiderMW doc copy remnant)
2014-02-11 22:30:00 +02:00
Rolando Espinoza
a6279fe95b
DOC Fixed HTTPCACHE_STORAGE typo in the default value which is now Filesystem instead Dbm.
2014-01-30 11:53:42 -04:00
Mikhail Korobov
7b7a1d8dfd
Make Filesystem storage backend default again. See GH-500.
2014-01-17 04:32:08 +06:00
Mikhail Korobov
b03fe04999
Rename AjaxCrawlableMiddleware to AjaxCrawlMiddleware
2014-01-16 23:09:37 +06:00
Pablo Hoffman
ed6fd4933f
Merge pull request #524 from hobsonlane/master
...
documentation code example corrections per pablohoffman
2014-01-16 06:44:51 -08:00
Daniel Graña
b9bb9bed6b
Merge pull request #343 from kmike/ajax-crawlable
...
[MRG] AjaxCrawlableMiddleware
2014-01-16 05:07:39 -08:00
Hobson Lane
6ba0857a5c
documentation code example correction corrections per pablohoffman
2014-01-10 10:37:27 -08:00
Mikhail Korobov
a27d91f0a6
Rename BaseSpider to Spider. See GH-495.
2013-12-30 19:46:41 +06:00
Mikhail Korobov
943a0bd264
AjaxCrawlableMiddleware in Broad Crawl docs
2013-12-19 01:01:26 +06:00
Mikhail Korobov
a87b3bd1c8
AjaxCrawlableMiddleware
2013-12-19 00:06:47 +06:00
Travis Briggs
3043a5ba37
DownloaderMiddleware docs: Update process_request, proper explanation of IgnoreRequest.
...
Also:
* Change terminology to eliminate uses of terms such as "request middleware" to refer to the process_request methods of installed middleware.
* Remove description of "immediate redirection", as it is misleading.
Further changes.
2013-10-17 00:23:21 +00:00
Rocio Aramberri
d227d530f6
Added COMPRESSION_ENABLED setting to enable or disable the HttpCompressionMiddleware
...
Added COMPRESSION_ENABLE setting to docs
Added COMPRESSION_ENABLED setting to default settings
2013-08-01 11:31:28 -03:00
Mikhail Korobov
ac2fadf3ab
DownloaderMiddleware.process_response docs fix
...
"returns an exception" -> "raises an exception"
2013-07-08 19:41:58 +06:00
Mikhail Korobov
39e5da5f66
improve docs for DownloaderMiddleware.process_response
2013-07-08 19:17:29 +06:00
nramirezuy
83b2774354
remove wrong default httpcache
2013-06-25 17:01:29 -03:00
Pablo Hoffman
96c2332e0e
fix inaccurate downloader middleware documentation. refs #280
2013-04-02 11:35:32 -03:00
Pablo Hoffman
7400ceb1ed
added 502 to RETRY_HTTP_CODES
2013-02-22 19:12:59 -02:00
Chris Tilden
aae6aed4fb
fixes spelling errors in documentation
2013-01-22 14:52:18 -08:00
Pablo Hoffman
c31441a273
revert default HTTP cache policy to dummy (instead of RFC2616)
2013-01-17 13:08:29 -02:00
Daniel Graña
75563b3f00
Add list of supported and missing RFC2616 caching features
2013-01-08 18:16:44 -02:00
Daniel Graña
d8a760bf57
Merge branch 'http-cache-middleware'
...
Conflicts:
scrapy/contrib/downloadermiddleware/httpcache.py
scrapy/contrib/httpcache.py
scrapy/tests/test_downloadermiddleware_httpcache.py
2013-01-08 17:34:48 -02:00
Daniel Graña
864a7aef87
More httpcache updates
...
* Change default cache policy to RFC2616
* Update HttpCacheMiddleware documentation
* Move policies to scrapy.contrib.httpcache
* remove a lint error for .has_key() usage in DBM storage backend
2013-01-08 17:26:32 -02:00
Daniel Graña
defc4f89b5
update metarefresh settings
2013-01-08 11:41:19 -02:00
Daniel Graña
6a2b23883a
Add MetaRefreshMiddleware docs
2013-01-08 11:25:38 -02:00
Pedro Faustino
5d3a4d755f
Update downloader middleware documentation
2013-01-06 18:53:14 +00:00
Hasnain Lakhani
93a1102189
Implemented policies for HTTP Cache
2012-12-26 16:29:48 -08:00
Pedro Faustino
fdaa35f6e8
Updated the downloader middleware documentation to reflect changes introduced by the support for real HTTP caching.
2012-12-24 19:37:53 +01:00
stav
99f164fc87
correct docs for default storage backend
2012-11-22 14:05:47 -06:00
Pablo Hoffman
eed6eb49da
make DBM the new default storage backend for HTTP cache middleware, simplified DBM storage backend code to avoid dealing with many spiders at once (not needed), and update httpcache stats names (hit -> hits, miss -> misses)
2012-09-17 10:11:07 -03:00
Pablo Hoffman
2b16ebdc11
added minor clarification on cookiejar request meta key usage
2012-02-29 07:19:01 -02:00
Pablo Hoffman
81abb45000
fixed bug in new cookiejar documentation
2012-02-28 11:08:25 -02:00
Pablo Hoffman
26c8004125
added documentation for the new cookiejar Request.meta key
2012-02-27 19:58:58 -02:00
