BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-24 18:24:00 +00:00
Code Releases Wiki Activity

Update docs.

Browse Source 
modified:   docs/topics/media-pipeline.rst
This commit is contained in:
djunzu 2016-03-31 19:20:48 -03:00
parent 537083524e
commit 6988e9cd4b
1 changed files with 56 additions and 27 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

83
docs/topics/media-pipeline.rst
View File

@ -77,30 +77,6 @@ PIL.
.. _Python Imaging Library: http://www.pythonware.com/products/pil/
Usage example
=============
In order to use a media pipeline first, :ref:`enable it
<topics-media-pipeline-enabling>`.
Then, if a spider returns a dict with the URLs key ('file_urls' or
'image_urls', for the Files or Images Pipeline respectively), the pipeline will
put the results under respective key ('files' or images').
If you prefer to use :class:`~.Item`, then define a custom item with the
necessary fields, like in this example for Images Pipeline::
import scrapy
class MyItem(scrapy.Item):
# ... other item fields ...
image_urls = scrapy.Field()
images = scrapy.Field()
If you need something more complex and want to override the custom pipeline
behaviour, see :ref:`topics-media-pipeline-override`.
.. _topics-media-pipeline-enabling:
Enabling your Media Pipeline
@ -171,6 +147,51 @@ Where:
used). For more info see :ref:`topics-images-thumbnails`.
Usage example
=============
.. setting:: FILES_URLS_FIELD
.. setting:: FILES_RESULT_FIELD
.. setting:: IMAGES_URLS_FIELD
.. setting:: IMAGES_RESULT_FIELD
In order to use a media pipeline first, :ref:`enable it
<topics-media-pipeline-enabling>`.
Then, if a spider returns a dict with the URLs key (``file_urls`` or
``image_urls``, for the Files or Images Pipeline respectively), the pipeline will
put the results under respective key (``files`` or ``images``).
If you prefer to use :class:`~.Item`, then define a custom item with the
necessary fields, like in this example for Images Pipeline::
import scrapy
class MyItem(scrapy.Item):
# ... other item fields ...
image_urls = scrapy.Field()
images = scrapy.Field()
If you want to use another field name for the URLs key or for the results key,
it is also possible to override it.
For the Files Pipeline, set :setting:`FILES_URLS_FIELD` and/or
:setting:`FILES_RESULT_FIELD` settings::
FILES_URLS_FIELD = 'field_name_for_your_files_urls'
FILES_RESULT_FIELD = 'field_name_for_your_processed_files'
For the Images Pipeline, set :setting:`IMAGES_URLS_FIELD` and/or
:setting:`IMAGES_RESULT_FIELD` settings::
IMAGES_URLS_FIELD = 'field_name_for_your_images_urls'
IMAGES_RESULT_FIELD = 'field_name_for_your_processed_images'
If you need something more complex and want to override the custom pipeline
behaviour, see :ref:`topics-media-pipeline-override`.
Additional features
===================
@ -185,12 +206,14 @@ adjust this retention delay use the :setting:`FILES_EXPIRES` setting (or
:setting:`IMAGES_EXPIRES`, in case of Images Pipeline), which
specifies the delay in number of days::
# 90 days of delay for files expiration
FILES_EXPIRES = 90
# 120 days of delay for files expiration
FILES_EXPIRES = 120
# 30 days of delay for images expiration
IMAGES_EXPIRES = 30
The default value for both settings is 90 days.
.. _topics-images-thumbnails:
Thumbnail generation for images
@ -249,7 +272,13 @@ For example::
IMAGES_MIN_HEIGHT = 110
IMAGES_MIN_WIDTH = 110
Note: these size constraints don't affect thumbnail generation at all.
.. note::
The size constraints don't affect thumbnail generation at all.
It is possible to set just one size constraint or both. When setting both of
them, only images that satisfy both minimum sizes will be saved. For the
above example, images of sizes (105 x 105) or (105 x 200) or (200 x 105) will
all be dropped because at least one dimension is shorter than the constraint.
By default, there are no size constraints, so all images are processed.