scrapy/docs/ref/scheduler-middleware.rst

.. _ref-scheduler-middleware:

========================================
Built-in scheduler middleware reference
========================================

This page describes all scheduler middleware components that come with
Scrapy. 

For a list of the components enabled by default (and their orders) see the
:setting:`SCHEDULER_MIDDLEWARES_BASE` setting.

Available scheduler middlewares
===============================

DuplicatesFilterMiddleware
--------------------------

.. module:: scrapy.contrib.schedulermiddleware.duplicatesfilter

.. class:: DuplicatesFilterMiddleware

   Filter out already visited urls.

   The :class:`DuplicatesFilterMiddleware` can be configured through the following
   settings (see the settings documentation for more info):

      * :setting:`DUPEFILTER_CLASS` - The class used to detect and filter
        duplicate requests.