Paul Tremberth
db40852892
Do not interpret non-ASCII bytes in "Location" and percent-encode them ( #2322 )
...
* Do not interpret non-ASCII bytes in "Location" and percent-encode them
Fixes GH-2321
The idea is to not guess the encoding of "Location" header value
and simply percent-encode non-ASCII bytes,
which should then be re-interpreted correctly by the remote website
in whatever encoding was used originally.
See https://tools.ietf.org/html/rfc3987#section-3.2
This is similar to the changes to safe_url_string in
https://github.com/scrapy/w3lib/pull/45
* Remove unused import
2016-10-19 23:26:12 -03:00
Elias Dorneles
430e90f4b2
minor refactor on metarefresh redirect mware test
2015-09-13 12:29:41 -03:00
Elias Dorneles
81950f773d
made encoding conversion more explicit, added test for header with utf-8 encoding replicating what browsers do
2015-09-11 18:51:48 -03:00
Elias Dorneles
179c4588ca
adding test for latin1 location
2015-09-10 19:36:47 -03:00
Elias Dorneles
defa899135
PY3 port redirect middleware
2015-09-10 16:31:59 -03:00
Jakob de Maeyer
d164398a27
Fix RedirectMiddleware not honouring meta handle_httpstatus keys
2015-08-21 13:22:42 +02:00
Jakob de Maeyer
c153217bfb
Fix RedirectMiddleware test
2015-07-17 16:11:53 +02:00
Jakob de Maeyer
c908d31660
Make RedirectMiddleware respect Spider.handle_httpstatus_list
2015-07-16 12:50:26 +02:00
Julia Medina
d3f576a816
Move scrapy/spider.py to scrapy/spiders/__init__.py
2015-05-09 04:20:09 -03:00
Julia Medina
d7c444fefb
Move scrapy/contrib/downloadermiddleware to scrapy/downloadermiddlewares
2015-04-29 21:24:30 -03:00
Daniel Graña
ccde3317d7
Merge pull request #816 from Curita/api-cleanup
...
GSoC API cleanup
2014-09-01 21:55:36 -03:00
Rocio Aramberri
51b0bd281d
fix dont settings on meta behaviour, add docs and tests
2014-08-15 13:47:42 -07:00
Julia Medina
870438e5f4
Update tests utils, fixing get_crawler and removing docrawl
2014-08-12 14:02:56 -03:00
Daniel Graña
242c08553f
move tests under root dir
2014-07-30 19:45:08 -03:00
