BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-25 00:04:03 +00:00
Code Releases Wiki Activity

removed redundant botname from log lines

Browse Source
This commit is contained in:
Pablo Hoffman 2009-06-25 16:48:04 -03:00
parent 18301b7e66
commit 80cd534f92
2 changed files with 11 additions and 11 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

16
docs/intro/tutorial.rst
View File

@ -138,16 +138,16 @@ you'll get an output like this::
[dmoz] INFO: Enabled item pipelines: ...
[-] scrapy.management.web.WebConsole starting on 60738
[-] scrapy.management.telnet.TelnetConsole starting on 51506
[dmoz/dmoz.org] INFO: Domain opened
[dmoz/dmoz.org] DEBUG: Crawled <http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/> from <None>
[dmoz/dmoz.org] DEBUG: Crawled <http://www.dmoz.org/Computers/Programming/Languages/Python/Books/> from <None>
[dmoz/dmoz.org] INFO: Domain closed (finished)
[dmoz.org] INFO: Domain opened
[dmoz.org] DEBUG: Crawled <http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/> from <None>
[dmoz.org] DEBUG: Crawled <http://www.dmoz.org/Computers/Programming/Languages/Python/Books/> from <None>
[dmoz.org] INFO: Domain closed (finished)
[scrapy.management.web.WebConsole] (Port 60738 Closed)
[scrapy.management.telnet.TelnetConsole] (Port 51506 Closed)
[-] Main loop terminated.
Pay attention to the lines labeled ``[dmoz/dmoz.org]``, which corresponds to
our spider identified by the domain "dmoz.org". You can see a log line for each
Pay attention to the lines containing ``[dmoz.org]``, which corresponds to
our spider (identified by the domain "dmoz.org"). You can see a log line for each
URL defined in ``start_urls``. Because these URLs are the starting ones, they
have no referrers, which is shown at the end of the log line, where it says
``from <None>``.
@ -386,8 +386,8 @@ should be like this::
Now doing a crawl on the dmoz.org domain yields ``DmozItem``'s::
[dmoz/dmoz.org] INFO: Scraped DmozItem({'title': [u'Text Processing in Python'], 'link': [u'http://gnosis.cx/TPiP/'], 'desc': [u' - By David Mertz; Addison Wesley. Book in progress, full text, ASCII format. Asks for feedback. [author website, Gnosis Software, Inc.]\n']}) in <http://www.dmoz.org/Computers/Programming/Languages/Python/Books/>
[dmoz/dmoz.org] INFO: Scraped DmozItem({'title': [u'XML Processing with Python'], 'link': [u'http://www.informit.com/store/product.aspx?isbn=0130211192'], 'desc': [u' - By Sean McGrath; Prentice Hall PTR, 2000, ISBN 0130211192, has CD-ROM. Methods to build XML applications fast, Python tutorial, DOM and SAX, new Pyxie open source XML processing library. [Prentice Hall PTR]\n']}) in <http://www.dmoz.org/Computers/Programming/Languages/Python/Books/>
[dmoz.org] INFO: Scraped DmozItem({'title': [u'Text Processing in Python'], 'link': [u'http://gnosis.cx/TPiP/'], 'desc': [u' - By David Mertz; Addison Wesley. Book in progress, full text, ASCII format. Asks for feedback. [author website, Gnosis Software, Inc.]\n']}) in <http://www.dmoz.org/Computers/Programming/Languages/Python/Books/>
[dmoz.org] INFO: Scraped DmozItem({'title': [u'XML Processing with Python'], 'link': [u'http://www.informit.com/store/product.aspx?isbn=0130211192'], 'desc': [u' - By Sean McGrath; Prentice Hall PTR, 2000, ISBN 0130211192, has CD-ROM. Methods to build XML applications fast, Python tutorial, DOM and SAX, new Pyxie open source XML processing library. [Prentice Hall PTR]\n']}) in <http://www.dmoz.org/Computers/Programming/Languages/Python/Books/>
Storing the data (using an Item Pipeline)

6
scrapy/log/__init__.py
View File

@ -51,10 +51,10 @@ def start(logfile=None, loglevel=None, log_stdout=None):
def msg(message, level=INFO, component=BOT_NAME, domain=None):
"""Log message according to the level"""
component = "%s/%s" % (component, domain) if domain else component
system = domain if domain else component
if level <= log_level:
msg_txt = unicode_to_str("%s: %s" % (level_names[level], message))
log.msg(msg_txt, system=component)
log.msg(msg_txt, system=system)
def exc(message, level=ERROR, component=BOT_NAME, domain=None):
message = message + '\n' + format_exc()
@ -63,5 +63,5 @@ def exc(message, level=ERROR, component=BOT_NAME, domain=None):
def err(*args, **kwargs):
domain = kwargs.pop('domain', None)
component = kwargs.pop('component', BOT_NAME)
kwargs['system'] = "%s/%s" % (component, domain) if domain else component
kwargs['system'] = domain if domain else component
log.err(*args, **kwargs)