Pablo Hoffman
984be35461
Some telnet console changes:
...
* renamed manager alias to crawler
* added aliases: spider, slot
* fixed est() function
2011-08-08 15:01:08 -03:00
Pablo Hoffman
f03af7874d
fixed bug in scheduler 'has_pending_requests' method which prevented spiders to close properly in some cases
2011-08-08 14:52:54 -03:00
Daniel Grana
c35a7519c0
Correctly handle query parameters on s3:// urls
2011-08-08 13:23:45 -03:00
Pablo Hoffman
5c63b2307f
Another step towards singleton removal: deprecated crawler singleton import (from scrapy.project import crawler) by a new class method that extensions can implement to receive the crawler
2011-08-08 11:42:44 -03:00
Pablo Hoffman
0eaa1d95f6
replaced DeprecationWarning by a new ScrapyDeprecationWarning category, since the default DeprecationWarning is silenced on Python 2.7+
2011-08-08 10:39:53 -03:00
Pablo Hoffman
f7c0aeccc6
added note about engine_started signal
2011-08-07 03:57:09 -03:00
Pablo Hoffman
a2b0737a1d
scrapy.utils.sitemap: added one more case of parsing invalid sitemaps
2011-08-07 03:24:32 -03:00
Pablo Hoffman
cea0dae1b2
scrapy.utils.sitemap: added support for parsing sitemaps with wrong namespaces, found in some bogus websites
2011-08-07 03:13:55 -03:00
Pablo Hoffman
259dccaf58
moved module scrapy.core.downloader.responsetypes to scrapy.responsetypes
...
--HG--
rename : scrapy/core/downloader/responsetypes/mime.types => scrapy/mime.types
rename : scrapy/core/downloader/responsetypes/__init__.py => scrapy/responsetypes.py
2011-08-07 02:49:57 -03:00
Pablo Hoffman
9f60c27612
added setting to support disabling DNS cache: DNSCACHE_ENABLED
2011-08-05 20:41:59 -03:00
Pablo Hoffman
bb67cfd955
added MarshalDiskQueue unittests
2011-08-05 20:32:22 -03:00
Pablo Hoffman
5c938cc029
removed no longer working tests from get_engine_status()
2011-08-05 20:26:24 -03:00
Pablo Hoffman
38e193d480
MarshalDiskQueue bug fix
2011-08-05 17:06:31 -03:00
Pablo Hoffman
1ce84046d8
scheduler: bug fix to use in-memory queues when request can't be serialized by the disk-queues
2011-08-05 12:39:29 -03:00
Pablo Hoffman
76cbb6a2e6
removed wrong blocking api usage (socket.gethostbyname()) from downloader when using CONCURRENT_REQUESTS_PER_IP
2011-08-03 23:55:59 -03:00
Pablo Hoffman
ebb892e554
updated get_engine_status() after scheduler changes
2011-08-03 23:19:22 -03:00
Pablo Hoffman
cd8470b309
fixed crawlspider bug introduced after scheduler refactoring
2011-08-03 20:25:14 -03:00
Pablo Hoffman
cb95d7a5af
added marshal to formats supported by feed exports
2011-08-03 16:16:48 -03:00
Pablo Hoffman
884dc93ab7
Automated merge with ssh://hg.scrapy.org:2222/scrapy-0.12
2011-08-02 22:46:02 -03:00
Pablo Hoffman
3191a08560
fix start_python_console() to work with IPython >= 0.11. closes #335
2011-08-02 22:45:51 -03:00
Pablo Hoffman
888dac867f
fixed bug with scheduler.__len__() when disk queues are disabled
2011-08-02 15:56:32 -03:00
Pablo Hoffman
f8c9b40345
added __len__() method to scheduler
2011-08-02 15:46:35 -03:00
Pablo Hoffman
fbf0e9ef43
cleaned up lxml-based link extractors, and left one of them
...
--HG--
rename : scrapy/contrib/linkextractors/lxmlparser.py => scrapy/contrib/linkextractors/lxmlhtml.py
2011-08-02 15:10:25 -03:00
Pablo Hoffman
c6f29f02c4
tie request to downloader failures, so that they can be accessed from request errbacks
2011-08-02 12:02:08 -03:00
Pablo Hoffman
549725215e
Initial support for a persistent scheduler, to support pausing and resuming
...
crawls.
* requests are serialized (using marshal by default) and stored on disk, using
one queue per priority
* request priorities must be integers now
* breadh-first and depth-first crawling orders can now be configured
through a new DEPTH_PRIORITY setting (see doc). backwards compatilibty with
SCHEDULER_ORDER was kept.
* requests that can't be serialized (for example, non serializable callbacks)
are always kept in memory queues
* adapted crawl spider to work with persitent scheduler
2011-08-02 11:57:55 -03:00
Pablo Hoffman
6d989e3fb0
imported patch scheduler_single_spider.patch
2011-07-31 03:32:25 -03:00
Pablo Hoffman
4d1e01a4d4
removed obsolete profiling/ dir
2011-07-31 02:47:14 -03:00
Pablo Hoffman
f354a49d0f
added FAQ about preventing bots getting banned
2011-07-28 00:40:30 -03:00
Pablo Hoffman
ce38022665
restored support for download delays after downlaoder refactoring, also restored support for spider attributes: max_concurrent_requests and download_delay
2011-07-27 15:14:27 -03:00
Pablo Hoffman
ce7a787970
Big downloader refactoring to support real concurrency limits per domain/ip,
...
instead of global limits per spider which were a bit useless.
This removes the setting CONCURRENT_REQUESTS_PER_SPIDER and adds thre new
settings:
* CONCURRENT_REQUESTS
* CONCURRENT_REQUESTS_PER_DOMAIN
* CONCURRENT_REQUESTS_PER_IP (overrides per domain)
The AutoThrottle extension had to be disabled, but will be ported and
re-enabled soon.
2011-07-27 13:38:09 -03:00
Pablo Hoffman
a45dca32f5
removed deprecation warning (scheduled to be removed on Scrapy 0.11)
2011-07-27 13:21:58 -03:00
Pablo Hoffman
b47f5330fd
added fragment attribute ot Link object
2011-07-27 12:28:36 -03:00
Pablo Hoffman
e9ca309b6d
minor comments adjustments
2011-07-27 03:48:38 -03:00
Pablo Hoffman
c59340150f
Added cached DNS resolver based on old caching resolver extension from scrapy.contrib.resolver. This new one is *not* an extension, it comes builtin and always enabled.
2011-07-27 03:45:15 -03:00
Pablo Hoffman
90b716f7e4
downloader: removed unneeded code, and some minor refactoring
2011-07-26 19:06:52 -03:00
Pablo Hoffman
549298e38d
spidermanager: more detailed error message now that scrapy crawl command will raise the exception directly
2011-07-26 19:05:29 -03:00
Pablo Hoffman
cb9c937f50
minor code rearrangement for consistency
2011-07-26 18:49:01 -03:00
Pablo Hoffman
dd020e184f
removed rather useless (and some deprecated) docstrings
2011-07-26 18:45:51 -03:00
Pablo Hoffman
70493c754d
retry middleware: added TCPTimedOutError to exceptions to retry
2011-07-25 14:52:24 -03:00
Pablo Hoffman
6f0e492390
fixed bug with scraper KeyError's on some ConnectionLost errors. closes #334
2011-07-25 12:24:26 -03:00
Pablo Hoffman
6e50f94406
engine: make it more explicit that we don't need to return the value of nextcall.schedule()
2011-07-25 10:47:54 -03:00
Pablo Hoffman
ea3bf6d95d
more core refactoring including moving engine next request call logic to a separate class
2011-07-25 10:46:00 -03:00
Pablo Hoffman
209ecdf471
updated settings.py for djangoitem tests to new django multi-db format
2011-07-25 00:49:54 -03:00
Pablo Hoffman
2ac08a713d
downloader: renamed SpiderInfo to Slot, for consistency with engine and scraper names
2011-07-22 02:06:10 -03:00
Pablo Hoffman
d6b83fee3e
scraper: renamed SpiderInfo to Slot, for consistency with engine names
2011-07-22 02:01:05 -03:00
Pablo Hoffman
f19442425a
forked UnicodeDammit from BeautifulSoup to explicitly disable usage of chardet library
2011-07-20 17:41:53 -03:00
Pablo Hoffman
7d18fe18e2
added missing import
2011-07-20 17:05:21 -03:00
Pablo Hoffman
0e008268e1
removed SimpledbStatsCollector from scrapy code, it was moved to https://github.com/scrapinghub/scaws
2011-07-20 10:38:16 -03:00
Pablo Hoffman
b6b0a54d9f
removed FAQ entry
2011-07-20 01:31:36 -03:00
Pablo Hoffman
cc6ef3beb2
engine: renamed slot.requests to slot.start_requests
2011-07-20 01:18:34 -03:00
