BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-26 22:44:05 +00:00
Code Releases Wiki Activity

Merge pull request #541 from kmike/fs-as-default-cache

Browse Source 
[MRG] Make Filesystem storage backend default again.
This commit is contained in:
Daniel Graña 2014-01-17 09:10:26 -08:00
commit 3c1e22618b
2 changed files with 21 additions and 22 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

41
docs/topics/downloader-middleware.rst
View File

@ -307,8 +307,8 @@ HttpCacheMiddleware
Scrapy ships with two HTTP cache storage backends: Scrapy ships with two HTTP cache storage backends:
* :ref:`httpcache-storage-dbm`
* :ref:`httpcache-storage-fs` * :ref:`httpcache-storage-fs`
* :ref:`httpcache-storage-dbm`
You can change the HTTP cache storage backend with the :setting:`HTTPCACHE_STORAGE` You can change the HTTP cache storage backend with the :setting:`HTTPCACHE_STORAGE`
setting. Or you can also implement your own storage backend. setting. Or you can also implement your own storage backend.
@ -376,29 +376,12 @@ In order to use this policy, set:
* :setting:`HTTPCACHE_POLICY` to ``scrapy.contrib.httpcache.RFC2616Policy`` * :setting:`HTTPCACHE_POLICY` to ``scrapy.contrib.httpcache.RFC2616Policy``
.. _httpcache-storage-dbm:
DBM storage backend (default)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. versionadded:: 0.13
A DBM_ storage backend is available for the HTTP cache middleware.
By default, it uses the anydbm_ module, but you can change it with the
:setting:`HTTPCACHE_DBM_MODULE` setting.
In order to use this storage backend, set:
* :setting:`HTTPCACHE_STORAGE` to ``scrapy.contrib.httpcache.DbmCacheStorage``
.. _httpcache-storage-fs: .. _httpcache-storage-fs:
Filesystem storage backend Filesystem storage backend (default)
~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
A file system storage backend is also available for the HTTP cache middleware. File system storage backend is available for the HTTP cache middleware.
In order to use this storage backend, set: In order to use this storage backend, set:
@ -423,6 +406,22 @@ inefficient in many file systems). An example directory could be::
/path/to/cache/dir/example.com/72/72811f648e718090f041317756c03adb0ada46c7 /path/to/cache/dir/example.com/72/72811f648e718090f041317756c03adb0ada46c7
.. _httpcache-storage-dbm:
DBM storage backend
~~~~~~~~~~~~~~~~~~~
.. versionadded:: 0.13
A DBM_ storage backend is also available for the HTTP cache middleware.
By default, it uses the anydbm_ module, but you can change it with the
:setting:`HTTPCACHE_DBM_MODULE` setting.
In order to use this storage backend, set:
* :setting:`HTTPCACHE_STORAGE` to ``scrapy.contrib.httpcache.DbmCacheStorage``
HTTPCache middleware settings HTTPCache middleware settings
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

2
scrapy/settings/default_settings.py
View File

@ -145,7 +145,7 @@ FEED_EXPORTERS_BASE = {
HTTPCACHE_ENABLED = False HTTPCACHE_ENABLED = False
HTTPCACHE_DIR = 'httpcache' HTTPCACHE_DIR = 'httpcache'
HTTPCACHE_IGNORE_MISSING = False HTTPCACHE_IGNORE_MISSING = False
HTTPCACHE_STORAGE = 'scrapy.contrib.httpcache.DbmCacheStorage' HTTPCACHE_STORAGE = 'scrapy.contrib.httpcache.FilesystemCacheStorage'
HTTPCACHE_EXPIRATION_SECS = 0 HTTPCACHE_EXPIRATION_SECS = 0
HTTPCACHE_IGNORE_HTTP_CODES = [] HTTPCACHE_IGNORE_HTTP_CODES = []
HTTPCACHE_IGNORE_SCHEMES = ['file'] HTTPCACHE_IGNORE_SCHEMES = ['file']