BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-23 21:44:19 +00:00
Code Releases Wiki Activity

renamed to sitemap_alternate_links and added default value, see #360

Browse Source
This commit is contained in:
Stefan 2013-09-08 10:38:28 +02:00
parent 8ed2d0cda1
commit 6994959181
2 changed files with 6 additions and 5 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
docs/topics/spiders.rst
View File

@ -560,7 +560,7 @@ SitemapSpider
By default, all sitemaps are followed.
.. attribute:: use_alternate_links
.. attribute:: sitemap_alternate_links
Specifies if alternate links for one ``url`` should be followed. These
are links for the same website in another language passed within
@ -573,11 +573,11 @@ SitemapSpider
<xhtml:link rel="alternate" hreflang="de" href="http://example.com/de"/>
</url>
With ``use_alternate_links`` set, this would retrieve both URLs. With
``use_alternate_links`` disabled, only ``http://example.com/`` would be
With ``sitemap_alternate_links`` set, this would retrieve both URLs. With
``sitemap_alternate_links`` disabled, only ``http://example.com/`` would be
retrieved.
Default is ``use_alternate_links`` disabled.
Default is ``sitemap_alternate_links`` disabled.
SitemapSpider examples

3
scrapy/contrib/spiders/sitemap.py
View File

@ -11,6 +11,7 @@ class SitemapSpider(BaseSpider):
sitemap_urls = ()
sitemap_rules = [('', 'parse')]
sitemap_follow = ['']
sitemap_alternate_links = False
def __init__(self, *a, **kw):
super(SitemapSpider, self).__init__(*a, **kw)
@ -37,7 +38,7 @@ class SitemapSpider(BaseSpider):
s = Sitemap(body)
if s.type == 'sitemapindex':
for loc in iterloc(s, self.use_alternate_links):
for loc in iterloc(s, self.sitemap_alternate_links):
if any(x.search(loc) for x in self._follow):
yield Request(loc, callback=self._parse_sitemap)
elif s.type == 'urlset':