BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-25 04:23:45 +00:00
Code Releases Wiki Activity

removed redundant botname from log lines

Browse Source
This commit is contained in:
Pablo Hoffman 2009-06-25 16:48:04 -03:00
parent 18301b7e66
commit 80cd534f92
2 changed files with 11 additions and 11 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

16
docs/intro/tutorial.rst
View File

@ -138,16 +138,16 @@ you'll get an output like this::
[dmoz] INFO: Enabled item pipelines: ... [dmoz] INFO: Enabled item pipelines: ...
[-] scrapy.management.web.WebConsole starting on 60738 [-] scrapy.management.web.WebConsole starting on 60738
[-] scrapy.management.telnet.TelnetConsole starting on 51506 [-] scrapy.management.telnet.TelnetConsole starting on 51506
[dmoz/dmoz.org] INFO: Domain opened [dmoz.org] INFO: Domain opened
[dmoz/dmoz.org] DEBUG: Crawled <http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/> from <None> [dmoz.org] DEBUG: Crawled <http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/> from <None>
[dmoz/dmoz.org] DEBUG: Crawled <http://www.dmoz.org/Computers/Programming/Languages/Python/Books/> from <None> [dmoz.org] DEBUG: Crawled <http://www.dmoz.org/Computers/Programming/Languages/Python/Books/> from <None>
[dmoz/dmoz.org] INFO: Domain closed (finished) [dmoz.org] INFO: Domain closed (finished)
[scrapy.management.web.WebConsole] (Port 60738 Closed) [scrapy.management.web.WebConsole] (Port 60738 Closed)
[scrapy.management.telnet.TelnetConsole] (Port 51506 Closed) [scrapy.management.telnet.TelnetConsole] (Port 51506 Closed)
[-] Main loop terminated. [-] Main loop terminated.
Pay attention to the lines labeled ``[dmoz/dmoz.org]``, which corresponds to Pay attention to the lines containing ``[dmoz.org]``, which corresponds to
our spider identified by the domain "dmoz.org". You can see a log line for each our spider (identified by the domain "dmoz.org"). You can see a log line for each
URL defined in ``start_urls``. Because these URLs are the starting ones, they URL defined in ``start_urls``. Because these URLs are the starting ones, they
have no referrers, which is shown at the end of the log line, where it says have no referrers, which is shown at the end of the log line, where it says
``from <None>``. ``from <None>``.
@ -386,8 +386,8 @@ should be like this::
Now doing a crawl on the dmoz.org domain yields ``DmozItem``'s:: Now doing a crawl on the dmoz.org domain yields ``DmozItem``'s::
[dmoz/dmoz.org] INFO: Scraped DmozItem({'title': [u'Text Processing in Python'], 'link': [u'http://gnosis.cx/TPiP/'], 'desc': [u' - By David Mertz; Addison Wesley. Book in progress, full text, ASCII format. Asks for feedback. [author website, Gnosis Software, Inc.]\n']}) in <http://www.dmoz.org/Computers/Programming/Languages/Python/Books/> [dmoz.org] INFO: Scraped DmozItem({'title': [u'Text Processing in Python'], 'link': [u'http://gnosis.cx/TPiP/'], 'desc': [u' - By David Mertz; Addison Wesley. Book in progress, full text, ASCII format. Asks for feedback. [author website, Gnosis Software, Inc.]\n']}) in <http://www.dmoz.org/Computers/Programming/Languages/Python/Books/>
[dmoz/dmoz.org] INFO: Scraped DmozItem({'title': [u'XML Processing with Python'], 'link': [u'http://www.informit.com/store/product.aspx?isbn=0130211192'], 'desc': [u' - By Sean McGrath; Prentice Hall PTR, 2000, ISBN 0130211192, has CD-ROM. Methods to build XML applications fast, Python tutorial, DOM and SAX, new Pyxie open source XML processing library. [Prentice Hall PTR]\n']}) in <http://www.dmoz.org/Computers/Programming/Languages/Python/Books/> [dmoz.org] INFO: Scraped DmozItem({'title': [u'XML Processing with Python'], 'link': [u'http://www.informit.com/store/product.aspx?isbn=0130211192'], 'desc': [u' - By Sean McGrath; Prentice Hall PTR, 2000, ISBN 0130211192, has CD-ROM. Methods to build XML applications fast, Python tutorial, DOM and SAX, new Pyxie open source XML processing library. [Prentice Hall PTR]\n']}) in <http://www.dmoz.org/Computers/Programming/Languages/Python/Books/>
Storing the data (using an Item Pipeline) Storing the data (using an Item Pipeline)

6
scrapy/log/__init__.py
View File

@ -51,10 +51,10 @@ def start(logfile=None, loglevel=None, log_stdout=None):
def msg(message, level=INFO, component=BOT_NAME, domain=None): def msg(message, level=INFO, component=BOT_NAME, domain=None):
"""Log message according to the level""" """Log message according to the level"""
component = "%s/%s" % (component, domain) if domain else component system = domain if domain else component
if level <= log_level: if level <= log_level:
msg_txt = unicode_to_str("%s: %s" % (level_names[level], message)) msg_txt = unicode_to_str("%s: %s" % (level_names[level], message))
log.msg(msg_txt, system=component) log.msg(msg_txt, system=system)
def exc(message, level=ERROR, component=BOT_NAME, domain=None): def exc(message, level=ERROR, component=BOT_NAME, domain=None):
message = message + '\n' + format_exc() message = message + '\n' + format_exc()
@ -63,5 +63,5 @@ def exc(message, level=ERROR, component=BOT_NAME, domain=None):
def err(*args, **kwargs): def err(*args, **kwargs):
domain = kwargs.pop('domain', None) domain = kwargs.pop('domain', None)
component = kwargs.pop('component', BOT_NAME) component = kwargs.pop('component', BOT_NAME)
kwargs['system'] = "%s/%s" % (component, domain) if domain else component kwargs['system'] = domain if domain else component
log.err(*args, **kwargs) log.err(*args, **kwargs)