BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-25 22:04:05 +00:00
Code Releases Wiki Activity
5,603 Commits 39 Branches 119 Tags
Commit Graph

41 Commits

Author SHA1 Message Date
Paul Tremberth
cd979ace40 Add HTTPS tests with non-hostname-maching server certificate 2016-04-20 14:42:03 +02:00
Paul Tremberth
ecddc093a4 Explicitly call Twisted transport stopProducing() on HTTP/1.1 timeouts 2016-02-24 23:04:31 +01:00
Paul Tremberth
6174192564 Add "Content-Length: 0" for body-less HTTP/1.1 POST requests 
GH-823 was fixed only for HTTP/1.0 (in GH-1089)
 2016-02-20 23:27:04 +01:00
Konstantin Lopuhin
d1ecb8cd38 Fix S3TestCase for precise env: we reraise TypeError as NotConfigured in this case 2016-02-15 19:59:48 +03:00
Konstantin Lopuhin
77ebb13684 fix assertRaises for precise env 2016-02-15 19:59:48 +03:00
Konstantin Lopuhin
19b2910ad1 Fix assert_aws_environ: check for botocore with boto fallback on PY2 2016-02-15 19:59:47 +03:00
Konstantin Lopuhin
bcb92b50dc check that no extra kwargs are silently discarded 2016-02-15 19:59:47 +03:00
Konstantin Lopuhin
d6bea3bf2e botocore not only does not allow passing our own Date header, but does not handle x-amz-date according to the spec 2016-02-15 19:59:47 +03:00
Konstantin Lopuhin
7748ee6bba mock date in s3 tests when using botocore 2016-02-15 19:59:47 +03:00
Konstantin Lopuhin
467553cc29 fix anon test: in this case we do no signing, just change the url 2016-02-15 19:59:47 +03:00
Konstantin Lopuhin
eaf3a239e4 using botocore for s3 request signing: proof of concept 2016-02-15 19:59:46 +03:00
Konstantin Lopuhin
1be90323c2 py3: properly skip s3 tests on py3 2016-01-24 23:44:56 +03:00
Daniel Graña
176610f910 optional_features has been removed 2016-01-19 15:34:26 -03:00
Daniel Graña
5ec4319885 Merge pull request #1358 from nyov/nyov/boto-anon-connection 
[MRG+1] Support anonymous S3DownloadHandler (boto) connections
 2016-01-19 13:35:23 -03:00
Konstantin Lopuhin
de98d8d006 move comment about test skipped on py3 into a proper place 2016-01-18 19:27:31 +03:00
Konstantin Lopuhin
7fdd3225b2 fix test skipping logic - this is (temporary) py2-only part 2016-01-18 19:09:09 +03:00
Konstantin Lopuhin
b940606b7e this is a test for TunnelingTCP4ClientEndpoint - move into Http11ProxyTestCase 2016-01-18 15:06:15 +03:00
Konstantin Lopuhin
0f527849f2 https proxy tunneling - add a test (not perfect, but covers all impl) and fix for py3 2016-01-18 14:44:04 +03:00
Konstantin Lopuhin
98c060d0b2 py3: fix http 1.1 test with https, and use self.host everywhere 2016-01-18 12:42:21 +03:00
Konstantin Lopuhin
04f69fd184 add https 1.1 downloader test - localhost is a valid DNS-ID 2016-01-18 12:37:46 +03:00
Konstantin Lopuhin
a4ca1668d8 add https test for http10 handler (no luck with testing https with http11 so far) 2016-01-15 14:20:19 +03:00
Konstantin Lopuhin
81a90c3af6 unskip part of test_download_gzip_response on py3, file a twisted issue for the remaining part 2016-01-14 18:47:06 +03:00
Konstantin Lopuhin
f46a9d595d skip ftp tests on py3 for now 2016-01-14 17:31:58 +03:00
Konstantin Lopuhin
4950f5988e py3: pass http proxy tests 2016-01-14 17:24:08 +03:00
Konstantin Lopuhin
c6f14a39de py3: fix http10 downloader - unicode host expected here 2016-01-14 16:50:16 +03:00
Konstantin Lopuhin
6b79fffa9a py3: pass all of HttpTestCase 2016-01-14 16:37:18 +03:00
Konstantin Lopuhin
3509378b8b py3: pass first http downloader test, simple crawler works now, yay! 2016-01-14 16:29:19 +03:00
Konstantin Lopuhin
32bb5b682a fix import of test_downloader_handlers.py: use @implementer, move failing on py3 imports into corresponding tests 2016-01-14 16:11:16 +03:00
nyov
ecbfe4bd66 drop deprecated "optional_features" set 2015-09-15 09:00:08 +00:00
nyov
1b4fd3a8df Support anonymous connections in S3DownloadHandler 
Also consider any unknown keyword args for S3DownloadHandler as
arguments to pass on to S3Connection (e.g. proxy settings).
 2015-08-14 22:12:12 +00:00
nyov
8c7997083f lazy-loading for DownloadHandlers 2015-08-10 16:52:44 -03:00
nyov
b2fd9bdb63 lazy-load s3 boto 2015-08-01 03:43:03 +00:00
Daniel Graña
ebe889a663 Unquote request path before passing to FTPClient, it already escape paths 2015-05-23 20:50:30 -03:00
Julia Medina
d3f576a816 Move scrapy/spider.py to scrapy/spiders/__init__.py 2015-05-09 04:20:09 -03:00
immerrr
82b187f283 S3DownloadHandler: fix auth for requests with quoted paths/query params 2014-12-11 18:14:36 +03:00
tpeng
cd19382754 attemp to fix travis fails 2014-11-25 14:20:25 +01:00
tpeng
a69f042d10 add 2 more test cases and minor doc fixes 2014-11-19 15:31:07 +01:00
tpeng
fa84730e70 avoid download large response 
introduce DOWNLOAD_MAXSIZE and DOWNLOAD_WARNSIZE in settings and
download_maxsize/download_warnsize in spider/request meta, so
downloader stop downloading as soon as the received data exceed the
limit. also check the twsisted response's length in advance to stop
downloading as early as possible.
 2014-11-12 12:28:02 +01:00
Julia Medina
870438e5f4 Update tests utils, fixing get_crawler and removing docrawl 2014-08-12 14:02:56 -03:00
Daniel Graña
521303e27f fix ftp test cases 2014-08-01 02:10:57 -03:00
Daniel Graña
242c08553f move tests under root dir 2014-07-30 19:45:08 -03:00