11 Commits

Author	SHA1	Message	Date
nyov	8c7997083f	lazy-loading for DownloadHandlers	2015-08-10 16:52:44 -03:00
nyov	b2fd9bdb63	lazy-load s3 boto	2015-08-01 03:43:03 +00:00
Daniel Graña	ebe889a663	Unquote request path before passing to FTPClient, it already escape paths	2015-05-23 20:50:30 -03:00
Julia Medina	d3f576a816	Move scrapy/spider.py to scrapy/spiders/__init__.py	2015-05-09 04:20:09 -03:00
immerrr	82b187f283	S3DownloadHandler: fix auth for requests with quoted paths/query params	2014-12-11 18:14:36 +03:00
tpeng	cd19382754	attemp to fix travis fails	2014-11-25 14:20:25 +01:00
tpeng	a69f042d10	add 2 more test cases and minor doc fixes	2014-11-19 15:31:07 +01:00
tpeng	fa84730e70	avoid download large response ... introduce DOWNLOAD_MAXSIZE and DOWNLOAD_WARNSIZE in settings and download_maxsize/download_warnsize in spider/request meta, so downloader stop downloading as soon as the received data exceed the limit. also check the twsisted response's length in advance to stop downloading as early as possible.	2014-11-12 12:28:02 +01:00
Julia Medina	870438e5f4	Update tests utils, fixing get_crawler and removing docrawl	2014-08-12 14:02:56 -03:00
Daniel Graña	521303e27f	fix ftp test cases	2014-08-01 02:10:57 -03:00
Daniel Graña	242c08553f	move tests under root dir	2014-07-30 19:45:08 -03:00