BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-06 08:49:32 +00:00
Code Releases Wiki Activity

Remove AjaxCrawlMiddleware mention from built-in downloader middleware

Browse Source
This commit is contained in:
Laerte Pereira 2025-02-05 13:16:51 -03:00
parent 9fd08a92a8
commit 2eb3c75c69
1 changed files with 20 additions and 56 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

76
docs/topics/downloader-middleware.rst
View File

@ -763,6 +763,26 @@ HttpProxyMiddleware
Keep in mind this value will take precedence over ``http_proxy``/``https_proxy``
environment variables, and it will also ignore ``no_proxy`` environment variable.
HttpProxyMiddleware settings
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. setting:: HTTPPROXY_ENABLED
.. setting:: HTTPPROXY_AUTH_ENCODING
HTTPPROXY_ENABLED
^^^^^^^^^^^^^^^^^
Default: ``True``
Whether or not to enable the :class:`HttpProxyMiddleware`.
HTTPPROXY_AUTH_ENCODING
^^^^^^^^^^^^^^^^^^^^^^^
Default: ``"latin-1"``
The default encoding for proxy authentication on :class:`HttpProxyMiddleware`.
OffsiteMiddleware
-----------------
@ -1220,60 +1240,4 @@ UserAgentMiddleware
In order for a spider to override the default user agent, its ``user_agent``
attribute must be set.
.. _ajaxcrawl-middleware:
AjaxCrawlMiddleware
-------------------
.. module:: scrapy.downloadermiddlewares.ajaxcrawl
.. class:: AjaxCrawlMiddleware
Middleware that finds 'AJAX crawlable' page variants based
on meta-fragment html tag.
.. note::
Scrapy finds 'AJAX crawlable' pages for URLs like
``'http://example.com/!#foo=bar'`` even without this middleware.
AjaxCrawlMiddleware is necessary when URL doesn't contain ``'!#'``.
This is often a case for 'index' or 'main' website pages.
AjaxCrawlMiddleware Settings
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. setting:: AJAXCRAWL_ENABLED
AJAXCRAWL_ENABLED
^^^^^^^^^^^^^^^^^
Default: ``False``
Whether the AjaxCrawlMiddleware will be enabled.
.. note::
This middleware is deprecated and will be removed in a future Scrapy release.
HttpProxyMiddleware settings
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. setting:: HTTPPROXY_ENABLED
.. setting:: HTTPPROXY_AUTH_ENCODING
HTTPPROXY_ENABLED
^^^^^^^^^^^^^^^^^
Default: ``True``
Whether or not to enable the :class:`HttpProxyMiddleware`.
HTTPPROXY_AUTH_ENCODING
^^^^^^^^^^^^^^^^^^^^^^^
Default: ``"latin-1"``
The default encoding for proxy authentication on :class:`HttpProxyMiddleware`.
.. _DBM: https://en.wikipedia.org/wiki/Dbm