mirror of
https://github.com/scrapy/scrapy.git
synced 2025-02-25 21:23:44 +00:00
129 lines
5.2 KiB
ReStructuredText
129 lines
5.2 KiB
ReStructuredText
.. _topics-telnetconsole:
|
|
|
|
==============
|
|
Telnet Console
|
|
==============
|
|
|
|
.. module:: scrapy.management.telnet
|
|
:synopsis: The Telnet Console
|
|
|
|
Scrapy comes with a built-in telnet console for inspecting and controlling a
|
|
Scrapy running process. The telnet console is just a regular python shell
|
|
running inside the Scrapy process, so you can do literally anything from it.
|
|
|
|
The telnet console is a :ref:`built-in Scrapy extension
|
|
<topics-extensions-ref>` which comes enabled by default, but you can also
|
|
disable it if you want. For more information about the extension itself see
|
|
:ref:`topics-extensions-ref-telnetconsole`.
|
|
|
|
.. highlight:: none
|
|
|
|
How to access the telnet console
|
|
================================
|
|
|
|
The telnet console listens in the TCP port defined in the
|
|
:setting:`TELNETCONSOLE_PORT` setting, which defaults to ``6023``. To access
|
|
the console you need to type::
|
|
|
|
telnet localhost 6023
|
|
>>>
|
|
|
|
You need the telnet program which comes installed by default in Windows, and
|
|
most Linux distros.
|
|
|
|
Available aliases in the telnet console
|
|
=======================================
|
|
|
|
The telnet console is like a regular Python shell running inside the Scrapy
|
|
process, so you can do anything from it including imports, etc.
|
|
|
|
However, the telnet console comes with some default shortcuts defined for
|
|
convenience:
|
|
|
|
+----------------+-------------------------------------------------------------------+
|
|
| Shortcut | Description |
|
|
+================+===================================================================+
|
|
| ``engine`` | the Scrapy engine object (``scrapy.core.engine.scrapyengine``) |
|
|
+----------------+-------------------------------------------------------------------+
|
|
| ``manager`` | the Scrapy manager object (``scrapy.core.manager.scrapymanager``) |
|
|
+----------------+-------------------------------------------------------------------+
|
|
| ``extensions`` | the extensions object (``scrapy.extension.extensions``) |
|
|
+----------------+-------------------------------------------------------------------+
|
|
| ``stats`` | the Scrapy stats object (``scrapy.stats.stats``) |
|
|
+----------------+-------------------------------------------------------------------+
|
|
| ``settings`` | the Scrapy settings object (``scrapy.conf.settings``) |
|
|
+----------------+-------------------------------------------------------------------+
|
|
| ``est`` | print a report of the current engine status |
|
|
+----------------+-------------------------------------------------------------------+
|
|
| ``prefs`` | for memory debugging (see :ref:`topics-leaks`) |
|
|
+----------------+-------------------------------------------------------------------+
|
|
| ``p`` | a shortcut to the `pprint.pprint`_ function |
|
|
+----------------+-------------------------------------------------------------------+
|
|
| ``hpy`` | for memory debugging (see :ref:`topics-leaks`) |
|
|
+----------------+-------------------------------------------------------------------+
|
|
|
|
.. _pprint.pprint: http://docs.python.org/library/pprint.html#pprint.pprint
|
|
|
|
Some example of using the telnet console
|
|
========================================
|
|
|
|
Here are some example tasks you can do with the telnet console:
|
|
|
|
View engine status
|
|
------------------
|
|
|
|
You can use the ``st()`` method of the Scrapy engine to quickly show its state
|
|
using the telnet console::
|
|
|
|
telnet localhost 6023
|
|
>>> est()
|
|
Execution engine status
|
|
|
|
datetime.now()-self.start_time : 0:00:09.051588
|
|
self.is_idle() : False
|
|
self.scheduler.is_idle() : False
|
|
len(self.scheduler.pending_requests) : 1
|
|
self.downloader.is_idle() : False
|
|
len(self.downloader.sites) : 1
|
|
self.downloader.has_capacity() : True
|
|
self.pipeline.is_idle() : False
|
|
len(self.pipeline.domaininfo) : 1
|
|
len(self._scraping) : 1
|
|
|
|
example.com
|
|
self.domain_is_idle(domain) : False
|
|
self.closing.get(domain) : None
|
|
self.scheduler.domain_has_pending_requests(domain) : True
|
|
len(self.scheduler.pending_requests[domain]) : 97
|
|
len(self.downloader.sites[domain].queue) : 17
|
|
len(self.downloader.sites[domain].active) : 25
|
|
len(self.downloader.sites[domain].transferring) : 8
|
|
self.downloader.sites[domain].closing : False
|
|
self.downloader.sites[domain].lastseen : 2009-06-23 15:20:16.563675
|
|
self.pipeline.domain_is_idle(domain) : True
|
|
len(self.pipeline.domaininfo[domain]) : 0
|
|
len(self._scraping[domain]) : 0
|
|
|
|
|
|
Pause, resume and stop Scrapy engine
|
|
------------------------------------
|
|
|
|
To pause::
|
|
|
|
telnet localhost 6023
|
|
>>> engine.pause()
|
|
>>>
|
|
|
|
To resume::
|
|
|
|
telnet localhost 6023
|
|
>>> engine.unpause()
|
|
>>>
|
|
|
|
To stop::
|
|
|
|
telnet localhost 6023
|
|
>>> engine.stop()
|
|
Connection closed by foreign host.
|
|
|