BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-06 09:07:32 +00:00
Code Releases Wiki Activity

Merge pull request #6656 from Laerte/master

Browse Source 
docs: Remove AjaxCrawlMiddleware mention from built-in downloader middleware
This commit is contained in:
Andrey Rakhmatullin 2025-02-05 21:05:54 +04:00 committed by GitHub
commit 02ad6bd1f6
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 20 additions and 56 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

76
docs/topics/downloader-middleware.rst
View File

@ -763,6 +763,26 @@ HttpProxyMiddleware
Keep in mind this value will take precedence over ``http_proxy``/``https_proxy``
environment variables, and it will also ignore ``no_proxy`` environment variable.
HttpProxyMiddleware settings
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. setting:: HTTPPROXY_ENABLED
.. setting:: HTTPPROXY_AUTH_ENCODING
HTTPPROXY_ENABLED
^^^^^^^^^^^^^^^^^
Default: ``True``
Whether or not to enable the :class:`HttpProxyMiddleware`.
HTTPPROXY_AUTH_ENCODING
^^^^^^^^^^^^^^^^^^^^^^^
Default: ``"latin-1"``
The default encoding for proxy authentication on :class:`HttpProxyMiddleware`.
OffsiteMiddleware
-----------------
@ -1220,60 +1240,4 @@ UserAgentMiddleware
In order for a spider to override the default user agent, its ``user_agent``
attribute must be set.
.. _ajaxcrawl-middleware:
AjaxCrawlMiddleware
-------------------
.. module:: scrapy.downloadermiddlewares.ajaxcrawl
.. class:: AjaxCrawlMiddleware
Middleware that finds 'AJAX crawlable' page variants based
on meta-fragment html tag.
.. note::
Scrapy finds 'AJAX crawlable' pages for URLs like
``'http://example.com/!#foo=bar'`` even without this middleware.
AjaxCrawlMiddleware is necessary when URL doesn't contain ``'!#'``.
This is often a case for 'index' or 'main' website pages.
AjaxCrawlMiddleware Settings
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. setting:: AJAXCRAWL_ENABLED
AJAXCRAWL_ENABLED
^^^^^^^^^^^^^^^^^
Default: ``False``
Whether the AjaxCrawlMiddleware will be enabled.
.. note::
This middleware is deprecated and will be removed in a future Scrapy release.
HttpProxyMiddleware settings
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. setting:: HTTPPROXY_ENABLED
.. setting:: HTTPPROXY_AUTH_ENCODING
HTTPPROXY_ENABLED
^^^^^^^^^^^^^^^^^
Default: ``True``
Whether or not to enable the :class:`HttpProxyMiddleware`.
HTTPPROXY_AUTH_ENCODING
^^^^^^^^^^^^^^^^^^^^^^^
Default: ``"latin-1"``
The default encoding for proxy authentication on :class:`HttpProxyMiddleware`.
.. _DBM: https://en.wikipedia.org/wiki/Dbm