Konstantin Lopuhin
|
324f2c1a6f
|
common test_downloadermiddleware.py also passes now due to fixes in compression middleware
|
2016-01-18 16:45:22 +03:00 |
|
Konstantin Lopuhin
|
0b9336418e
|
py3: port compression downloader middleware and tests
|
2016-01-18 16:43:58 +03:00 |
|
Konstantin Lopuhin
|
6edd4dec33
|
Merge branch 'py3-http-downloaders' into py3-downloader-middleware
|
2016-01-18 15:51:26 +03:00 |
|
Konstantin Lopuhin
|
f876654a0c
|
Merge branch 'py3-webclient' into py3-http-downloaders
|
2016-01-18 15:50:58 +03:00 |
|
Konstantin Lopuhin
|
4946434582
|
revert most changes to this test, and clarify - it is valid only on py2, because urls are strictly unicode on py3
|
2016-01-18 15:23:01 +03:00 |
|
Konstantin Lopuhin
|
a2efd389b0
|
clarify: rename r_transform to response_transform
|
2016-01-18 15:09:54 +03:00 |
|
Konstantin Lopuhin
|
b940606b7e
|
this is a test for TunnelingTCP4ClientEndpoint - move into Http11ProxyTestCase
|
2016-01-18 15:06:15 +03:00 |
|
Konstantin Lopuhin
|
7af64e8fd2
|
py3: remove extra encoding/decoding of url: pass it as bytes only when required
|
2016-01-18 15:00:43 +03:00 |
|
Konstantin Lopuhin
|
0f527849f2
|
https proxy tunneling - add a test (not perfect, but covers all impl) and fix for py3
|
2016-01-18 14:44:04 +03:00 |
|
Konstantin Lopuhin
|
98c060d0b2
|
py3: fix http 1.1 test with https, and use self.host everywhere
|
2016-01-18 12:42:21 +03:00 |
|
Konstantin Lopuhin
|
04f69fd184
|
add https 1.1 downloader test - localhost is a valid DNS-ID
|
2016-01-18 12:37:46 +03:00 |
|
Konstantin Lopuhin
|
f403c88053
|
Merge branch 'py3-webclient' into py3-http-downloaders
|
2016-01-18 12:25:51 +03:00 |
|
Konstantin Lopuhin
|
f2e2ff5e1f
|
py3: webclient assumes that urls come from Request.url and are ascii-only
|
2016-01-18 12:20:01 +03:00 |
|
Konstantin Lopuhin
|
673df5e416
|
add webclient test - check that non-standart body encoding matches Content-Encoding header
|
2016-01-18 11:44:49 +03:00 |
|
Konstantin Lopuhin
|
d3c2b0cf7e
|
py3 webclient: I was mistaken about unicode body, revert conversion to bytes and fix HEAD response
|
2016-01-18 10:06:14 +03:00 |
|
Konstantin Lopuhin
|
7d44c5dcea
|
py3: unskip tests/test_downloadermiddleware_httpcache.py and scrapy/downloadermiddlewares/httpcache.py
|
2016-01-15 15:41:31 +03:00 |
|
Konstantin Lopuhin
|
085fdd6283
|
py3 fix for ignoring cache controls - map is not a list
|
2016-01-15 15:40:45 +03:00 |
|
Konstantin Lopuhin
|
b0648271d6
|
py3 fix for rfc1123_to_epoch - "except Exception" was hiding bytes/str error
|
2016-01-15 15:17:30 +03:00 |
|
Konstantin Lopuhin
|
8330776c21
|
fix error reporting in test: we can fail in process_request too, so result should always be defined
|
2016-01-15 15:16:12 +03:00 |
|
Konstantin Lopuhin
|
4398d95a02
|
skip this file on py3 again - it has one compression test, sould be done separately
|
2016-01-15 14:54:12 +03:00 |
|
Konstantin Lopuhin
|
ae6758ee97
|
Merge branch 'py3-http-downloaders' into py3-downloader-middleware
|
2016-01-15 14:39:27 +03:00 |
|
Konstantin Lopuhin
|
a4ca1668d8
|
add https test for http10 handler (no luck with testing https with http11 so far)
|
2016-01-15 14:20:19 +03:00 |
|
Konstantin Lopuhin
|
96fcf4cea4
|
add a check that byte url is not accepted in http.Response on py3
|
2016-01-15 13:27:28 +03:00 |
|
Konstantin Lopuhin
|
87849780bc
|
some py3 fixes for RFC2616Policy
|
2016-01-15 13:13:10 +03:00 |
|
Konstantin Lopuhin
|
ea0471e33a
|
py3: fix LeveldbCacheStorage - using bytes as keys and values in leveldb
|
2016-01-15 12:22:41 +03:00 |
|
Konstantin Lopuhin
|
ddc91dda27
|
py3: fix _BaseTest in httpcache
|
2016-01-15 11:49:28 +03:00 |
|
Konstantin Lopuhin
|
e7ed1fd70d
|
py3 compat in httpcache - headers are bytes
|
2016-01-15 11:48:07 +03:00 |
|
Konstantin Lopuhin
|
dbf6cc73d9
|
py3: add leveldb to py33 test env, fix anydbm module name on py3
|
2016-01-15 11:46:56 +03:00 |
|
Konstantin Lopuhin
|
94ab7bee6c
|
py3: body to bytes in tests, unskip test file
|
2016-01-15 11:26:01 +03:00 |
|
Konstantin Lopuhin
|
99f1f2ad1d
|
unskip tests and modules ported to py3
|
2016-01-14 19:00:48 +03:00 |
|
Konstantin Lopuhin
|
81a90c3af6
|
unskip part of test_download_gzip_response on py3, file a twisted issue for the remaining part
|
2016-01-14 18:47:06 +03:00 |
|
Konstantin Lopuhin
|
2aa6c92ffc
|
py3 fixes in tests.mockserver
|
2016-01-14 17:52:50 +03:00 |
|
Konstantin Lopuhin
|
f46a9d595d
|
skip ftp tests on py3 for now
|
2016-01-14 17:31:58 +03:00 |
|
Konstantin Lopuhin
|
4950f5988e
|
py3: pass http proxy tests
|
2016-01-14 17:24:08 +03:00 |
|
Konstantin Lopuhin
|
c6f14a39de
|
py3: fix http10 downloader - unicode host expected here
|
2016-01-14 16:50:16 +03:00 |
|
Konstantin Lopuhin
|
6b79fffa9a
|
py3: pass all of HttpTestCase
|
2016-01-14 16:37:18 +03:00 |
|
Konstantin Lopuhin
|
3509378b8b
|
py3: pass first http downloader test, simple crawler works now, yay!
|
2016-01-14 16:29:19 +03:00 |
|
Konstantin Lopuhin
|
32bb5b682a
|
fix import of test_downloader_handlers.py: use @implementer, move failing on py3 imports into corresponding tests
|
2016-01-14 16:11:16 +03:00 |
|
Konstantin Lopuhin
|
5c2241ccc7
|
py3: fix webclient tests after making ScrapyHTTPClientFactory use bytes as in twisted
|
2016-01-14 15:30:28 +03:00 |
|
Konstantin Lopuhin
|
8df35bcac6
|
rm note to self: to be discussed in PR
|
2016-01-14 14:52:44 +03:00 |
|
Konstantin Lopuhin
|
ac2cf191d1
|
py3: remove comments, utf-8 is fine here: as twisted ultimately uses urllib.parse.quote that assepts bytes and assumes utf-8
|
2016-01-14 14:50:58 +03:00 |
|
Konstantin Lopuhin
|
e5fb609438
|
make ScrapyHTTPClientFactory comply to twisted HTTPClientFactory protocol - use bytes (encoding are likely wrong at this stage)
|
2016-01-14 14:45:02 +03:00 |
|
Konstantin Lopuhin
|
6a412d2503
|
all tests pass in test_webclient.py on py3 - removing from py3-ignores
|
2016-01-14 13:48:48 +03:00 |
|
Konstantin Lopuhin
|
0178356178
|
py3 fix testRedirect: url is bytes here
|
2016-01-14 13:46:46 +03:00 |
|
Konstantin Lopuhin
|
30c7b4e4cc
|
py3 compat in test_timeoutTriggering cleanup
|
2016-01-14 13:44:14 +03:00 |
|
Konstantin Lopuhin
|
88f55312af
|
py3 fix in testFactoryInfo - factory attirbutes are bytes in twisted
|
2016-01-14 13:43:37 +03:00 |
|
Konstantin Lopuhin
|
b5f9bc8499
|
py3 test fixes in test_webclient - expect bytes as page body
|
2016-01-14 13:04:45 +03:00 |
|
Konstantin Lopuhin
|
ae4aa2c3b2
|
py3 test fix: putChild expects bytes as path
|
2016-01-14 13:04:10 +03:00 |
|
Konstantin Lopuhin
|
325b6af6c2
|
fix ScrapyHTTPPageGetterTests for py3 - we expect bytes here
|
2016-01-14 12:29:59 +03:00 |
|
Konstantin Lopuhin
|
945674eb8f
|
pass test_externalUnicodeInterference - the logic for py3 is clearly inverse of what was expected in this test, as scrapy Request url must be unicode
|
2016-01-14 12:25:54 +03:00 |
|