Paul Tremberth db40852892 Do not interpret non-ASCII bytes in "Location" and percent-encode them (#2322) ...

* Do not interpret non-ASCII bytes in "Location" and percent-encode them

Fixes GH-2321

The idea is to not guess the encoding of "Location" header value
and simply percent-encode non-ASCII bytes,
which should then be re-interpreted correctly by the remote website
in whatever encoding was used originally.

See https://tools.ietf.org/html/rfc3987#section-3.2

This is similar to the changes to safe_url_string in
https://github.com/scrapy/w3lib/pull/45

* Remove unused import

2016-10-19 23:26:12 -03:00

..

keys

Add HTTPS tests with non-hostname-maching server certificate

2016-04-20 14:42:03 +02:00

mocks

move tests under root dir

2014-07-30 19:45:08 -03:00

sample_data

Modifying existing gzip read failure recovery mechanism to patch read for broken archives

2016-08-11 18:21:42 +05:30

test_cmdline

Amend "settings" command to output JSON for dict settings

2016-01-29 21:12:03 +01:00

test_settings

Update tests for settings copy_to_dict()

2016-01-29 18:59:12 +01:00

test_spiderloader

TST improve CrawlerRunner tests

2015-10-30 20:24:37 +05:00

test_utils_misc

move tests under root dir

2014-07-30 19:45:08 -03:00

__init__.py

Forward coverage envvars needed to cover subprocess

2015-08-13 21:07:45 -03:00

mockserver.py

Add HTTPS tests with non-hostname-maching server certificate

2016-04-20 14:42:03 +02:00

py3-ignores.txt

py3: reviewed s3 downloader handlers

2016-02-15 19:59:48 +03:00

requirements-py3.txt

TST pin pytest-cov to 2.2.1; upgrade pytest

2016-07-06 18:29:49 +05:00

requirements.txt

TST pin pytest-cov to 2.2.1; upgrade pytest

2016-07-06 18:29:49 +05:00

spiders.py

py3 compat: use range, fixes CrawlTestCase.test_start_requests_bug_yielding

2016-01-21 17:52:12 +03:00

test_closespider.py

Use LogCapture in testfixtures package for tests

2015-04-22 17:24:40 -03:00

test_command_fetch.py

py3: port fetch and shell commands, and review + enable already passing test_closespider.py and tests/test_utils_template.py

2016-01-20 23:01:29 +03:00

test_command_parse.py

Moved parse command tests to its own file. Added some checks to check for logged errors.

2016-09-19 05:33:05 +02:00

test_command_shell.py

Add shell test for using scrapy.Request() directly without importing scrapy

2016-09-15 19:25:20 +02:00

test_command_version.py

Increase coverage of version command

2015-08-13 19:30:06 -03:00

test_commands.py

Moved parse command tests to its own file. Added some checks to check for logged errors.

2016-09-19 05:33:05 +02:00

test_contracts.py

Move scrapy/spider.py to scrapy/spiders/__init__.py

2015-05-09 04:20:09 -03:00

test_crawl.py

py3: pass CrawlTestCase.test_referer_header, fixing Echo resource in mockserver and json decoding in test

2016-01-21 17:52:12 +03:00

test_crawler.py

make "settings" argument optional for Crawler, CrawlerRunner and CrawlerProcess

2015-06-06 03:23:13 +05:00

test_dependencies.py

move tests under root dir

2014-07-30 19:45:08 -03:00

test_downloader_handlers.py

Merge pull request #2094 from redapple/dns-invalid-id

2016-07-13 10:48:08 +00:00

test_downloadermiddleware_ajaxcrawlable.py

PY3 fix AjacCrawlable middleware tests

2015-09-01 03:49:52 +05:00

test_downloadermiddleware_cookies.py

Do not propagate cookie log messages in tests so TopLevelFormatter does not rewrite them

2015-08-03 17:18:08 -03:00

test_downloadermiddleware_decompression.py

PY3 fix test downloadermiddleware decompression

2015-07-29 13:51:01 +00:00

test_downloadermiddleware_defaultheaders.py

PY3 fix DefaultHeadersMiddleware tests

2015-09-01 03:57:05 +05:00

test_downloadermiddleware_downloadtimeout.py

Move scrapy/spider.py to scrapy/spiders/__init__.py

2015-05-09 04:20:09 -03:00

test_downloadermiddleware_httpauth.py

PY3 fix HttpAuthMiddleware tests

2015-09-01 04:00:26 +05:00

test_downloadermiddleware_httpcache.py

fix error reporting in test: we can fail in process_request too, so result should always be defined

2016-01-15 15:16:12 +03:00

test_downloadermiddleware_httpcompression.py

[HttpCompressionMiddleware] Do not decompress binary/octet-stream responses

2016-06-20 16:37:00 +02:00

test_downloadermiddleware_httpproxy.py

adding configurable encoding for httpproxy authentication

2016-01-21 09:35:04 -05:00

test_downloadermiddleware_redirect.py

Do not interpret non-ASCII bytes in "Location" and percent-encode them (#2322)

2016-10-19 23:26:12 -03:00

test_downloadermiddleware_retry.py

py3 fix HttpProxy and Retry Middlewares

2016-01-21 09:35:04 -05:00

test_downloadermiddleware_robotstxt.py

Add test for already failed deferreds when downloading page in robots.txt middleware.

2016-01-27 12:56:42 -02:00

test_downloadermiddleware_stats.py

PY3 fix DownloaderStats middleware tests

2015-09-01 04:09:15 +05:00

test_downloadermiddleware_useragent.py

PY3 fix UserAgentMiddleware tests

2015-09-01 04:11:33 +05:00

test_downloadermiddleware.py

py3: body to bytes in tests, unskip test file

2016-01-15 11:26:01 +03:00

test_dupefilters.py

Merge pull request #1388 from scrapy/dupefilter-persist

2015-08-11 14:16:34 -03:00

test_engine.py

response.text. Fixes GH-1729.

2016-01-27 01:28:11 +05:00

test_exporters.py

Fixed bug on XMLItemExporter with non-string fields in items

2016-01-30 10:00:06 -02:00

test_feedexport.py

Feed exporter: start exporting only on first item

2016-09-17 01:36:56 +02:00

test_http_cookies.py

PY3: Implement some attributes of WrappedRequest required in Python 3

2016-02-07 14:19:27 +09:00

test_http_headers.py

normalize header values to bytes

2014-08-01 00:02:50 -03:00

test_http_request.py

Use newer w3lib.url.safe_url_string() and re-enable HTTP request tests

2016-04-12 00:33:25 +02:00

test_http_response.py

response.text. Fixes GH-1729.

2016-01-27 01:28:11 +05:00

test_item.py

item fields defined on fields class attribute fix

2015-05-13 15:51:23 -03:00

test_link.py

PY3 raise an exception if bytes are passed as url to Link constructor

2015-08-28 23:04:02 +05:00

test_linkextractors_deprecated.py

Add link extractor test for non-ASCII characters in query part of URL

2016-04-09 15:15:01 +02:00

test_linkextractors.py

TST split LinkExtractorTestCase.test_extraction into several methods; remove duplicated test

2015-08-31 00:49:38 +05:00

test_loader.py

split nested_loader into seperate methods

2015-09-04 13:15:48 -07:00

test_logformatter.py

PY3 port test_logformatter

2015-08-26 02:34:21 +05:00

test_mail.py

MailSender.send: allow passing a charset.

2016-02-04 19:33:44 +01:00

test_middleware.py

PY3 fix test middleware

2015-07-29 13:51:01 +00:00

test_pipeline_files.py

[image & file pipeline] loading setting for user classes

2016-09-15 09:39:16 +02:00

test_pipeline_images.py

[image & file pipeline] loading setting for user classes

2016-09-15 09:39:16 +02:00

test_pipeline_media.py

PY3 fix test pipeline media

2015-07-29 13:51:01 +00:00

test_proxy_connect.py

Use LogCapture in testfixtures package for tests

2015-04-22 17:24:40 -03:00

test_pydispatch_deprecated.py

add deprecation for pydispatch (thanks for the help @redapple)

2016-02-11 16:15:28 -02:00

test_responsetypes.py

Merge pull request #1503 from demelziraptor/amazon-json-response

2016-09-12 13:21:16 -03:00

test_selector_csstranslator.py

cleanup tests for selectors and translators

2015-08-11 14:09:34 -03:00

test_selector.py

Selector should not receive both response and text

2016-07-30 10:35:16 +03:00

test_spider.py

Fix SitemapSpider to extract sitemap urls from robots.txt properly

2016-02-06 23:54:07 +09:00

test_spidermiddleware_depth.py

Move scrapy/spider.py to scrapy/spiders/__init__.py

2015-05-09 04:20:09 -03:00

test_spidermiddleware_httperror.py

Move scrapy/spider.py to scrapy/spiders/__init__.py

2015-05-09 04:20:09 -03:00

test_spidermiddleware_offsite.py

tests+doc for subdomains in offsite middleware

2016-01-26 12:49:43 +02:00

test_spidermiddleware_referer.py

PY3 depth, offsite and referer spider middlewares; Crawler

2015-08-28 04:11:30 +05:00

test_spidermiddleware_urllength.py

Move scrapy/spider.py to scrapy/spiders/__init__.py

2015-05-09 04:20:09 -03:00

test_spiderstate.py

Add test for raised exception with SpiderState extension when no JOBDIR used

2016-01-26 16:33:24 +01:00

test_squeues.py

fix scrapy squeue tests after recent changes to queuelib

2015-09-09 15:55:55 -03:00

test_stats.py

enable test_stats and test_utils_log tests in Python 3

2015-08-26 01:58:59 +05:00

test_toplevel.py

drop deprecated "optional_features" set

2015-09-15 09:00:08 +00:00

test_urlparse_monkeypatches.py

move tests under root dir

2014-07-30 19:45:08 -03:00

test_utils_conf.py

Add backwards compatibility for build_component_list

2015-11-11 17:39:56 +01:00

test_utils_console.py

Add support for bpython console.

2015-08-21 01:12:58 +01:00

test_utils_datatypes.py

move tests under root dir

2014-07-30 19:45:08 -03:00

test_utils_defer.py

Improve python3 support for test_utils_defer.

2015-07-25 12:58:39 +02:00

test_utils_deprecate.py

Ignore ScrapyDeprecationWarning warnings properly.

2015-06-09 18:22:03 +03:00

test_utils_gz.py

Modifying existing gzip read failure recovery mechanism to patch read for broken archives

2016-08-11 18:21:42 +05:30

test_utils_http.py

move tests under root dir

2014-07-30 19:45:08 -03:00

test_utils_httpobj.py

move tests under root dir

2014-07-30 19:45:08 -03:00

test_utils_iterators.py

Refactored test code

2016-01-13 12:32:28 +01:00

test_utils_log.py

Replace FailureFormatter with direct exc_info conversions in log calls

2015-05-15 05:46:14 -03:00

test_utils_project.py

test abs path outside project as well

2016-09-30 15:07:58 -03:00

test_utils_python.py

Rename isbinarytext function to binary_is_text for clarity

2016-03-30 15:44:15 +00:00

test_utils_reqser.py

PY3 response bodies as bytes

2015-08-08 04:23:32 +05:00

test_utils_request.py

Do not fail representing non-http requests

2015-08-10 14:10:58 -03:00

test_utils_response.py

response_status_message should not fail on non-standard HTTP codes

2016-03-12 14:16:40 +01:00

test_utils_serialize.py

Add set serialization to ScrapyJSONEncoder

2016-07-22 17:20:03 -04:00

test_utils_signal.py

PY3 port signals

2015-07-25 18:04:55 +02:00

test_utils_sitemap.py

PY3 fix test_utils_sitemap.

2014-08-01 16:17:40 +06:00

test_utils_spider.py

Move scrapy/contrib/spiders to scrapy/spiders

2015-04-29 21:27:19 -03:00

test_utils_template.py

Test utils.template.render_templatefile()

2015-08-26 20:44:33 +03:00

test_utils_trackref.py

PY3: port scrapy.utils.trackref

2015-08-10 15:13:29 -03:00

test_utils_url.py

Use w3lib.url.canonicalize_url() from w3lib 1.15.0

2016-08-16 17:42:16 +05:30

test_webclient.py

revert most changes to this test, and clarify - it is valid only on py2, because urls are strictly unicode on py3

2016-01-18 15:23:01 +03:00