Daniel Graña
25f609e2a3
Merge pull request #2675 from simongartz/png-p-to-jpg-conversion-fix
...
[MRG+1] Fixes conversion of transparent PNG with palette images to jpg #2452
2017-05-18 16:34:02 -03:00
Mikhail Korobov
532400f993
Merge pull request #2643 from harshasrinivas/set-retry-times-per-request
...
[MRG+1] Add feature to set RETRY_TIMES per request (#2642 )
2017-05-17 15:46:33 +05:00
Daniel Graña
73668ce407
Merge pull request #2721 from HarrisonGregg/feature-drop-from-response-field
...
[MRG+1] Allow dropping field in from_response formdata
2017-05-16 09:47:45 -03:00
Harrison Gregg
ffef828a8d
Add test for dropping fields in from_response request body
2017-05-15 12:25:32 -04:00
Paul Tremberth
dfe6d3d59a
Merge pull request #2456 from elacuesta/feed_export_beautify
...
[MRG+1] Feed exports: beautify JSON and XML
2017-05-12 18:12:48 +02:00
Eugenio Lacuesta
63b8caf5de
Feed exports: rewrite indentation test without .strip()
2017-05-09 11:58:53 -03:00
Daniel Graña
362d6f2d20
Merge pull request #2622 from rolando-contrib/download-maxsize-abort
...
[MRG+1] Abort connection earlier and avoid to buffer data when max size limit is reached
2017-05-04 11:44:51 -03:00
Paul Tremberth
6d14e392f1
Remove old test certificate+key
2017-04-27 23:35:01 +02:00
Paul Tremberth
e6ab8bc9a5
Change "localhost" test server certificate
2017-04-27 23:22:25 +02:00
Rolando Espinoza
2b34c6edff
Abort connection earlier and avoid to buffer data
...
A symptom of this issue was having the log message "Received (X) bytes
larger than download max size (Y)" several times printed, with increased
X values.
2017-04-27 13:28:16 -03:00
Paul Tremberth
c3d0f9b6c1
Add test for non-duplicated Content-Length: 0 for bodyless POST
2017-04-25 17:03:41 +02:00
Paul Tremberth
b1a0a6e258
Make mockserver runnable outside of tox
...
Add POST support for Echo resource
2017-04-25 17:01:54 +02:00
harshasrinivas
38e6857c95
Improvise the clarity of test cases
2017-03-23 19:45:04 +05:30
Simon Diviani Gartz
21d794d35a
Fixes conversion of transparent PNG with palette images to jpg #2452
2017-03-22 13:29:15 +01:00
Paul Tremberth
776129a951
Merge pull request #2649 from pawelmhm/logformatter-2647
...
[MRG+2] [logformatter] 'flags' format spec backward compatibility
2017-03-21 10:51:31 +01:00
harshasrinivas
0d9ebd6e1e
Update tests for max_retry_times
2017-03-19 06:15:46 +05:30
Pawel Miech
4345eaf1b6
[logformatter] backward compat comments
2017-03-17 08:11:20 +01:00
Mikhail Korobov
ec55799d5e
Merge pull request #2616 from redapple/mediapipeline-redirect-fix-continued
...
[MRG] Allow redirections in media files downloads
2017-03-16 15:31:28 +05:00
woxcab
a84652e775
Init tests are split by initializer' input
2017-03-15 12:39:48 +03:00
harshasrinivas
e321ac9931
Update unittests for max_retry_times
2017-03-15 04:12:32 +05:30
harshasrinivas
966bd49c42
Update unittest for meta['max_retry_times']
2017-03-14 16:23:47 +05:30
Pawel Miech
0f2a5cdb8e
[logformatter] 'flags' format spec backward compatibility
...
pass 'flags' kwarg to logger so that it is compatible with old
format of CRAWLEDMSG.
2017-03-13 15:16:38 +01:00
woxcab
fbb411a805
Allowed passing objects of Mapping class or its subclass to the CaselessDict initializer
2017-03-13 14:16:39 +03:00
Paul Tremberth
871134ee22
Refactor to also test FilesPipeline
2017-03-12 17:30:24 +01:00
Paul Tremberth
708f1b009b
Add integration tests for MEDIA_ALLOW_REDIRECTS
2017-03-10 21:36:33 +01:00
Paul Tremberth
7dcc86e61a
Add file listing resource + redirecting resource to MockServer
2017-03-10 21:35:25 +01:00
Mikhail Korobov
9c69e90056
Merge pull request #2632 from redapple/spider-loader-warn-or-fail
...
[MRG] Add SPIDER_LOADER_WARN_ONLY to toggle between spiderloader failure or warning
2017-03-09 23:01:27 +05:00
Paul Tremberth
c0cbaccb7b
Merge pull request #2581 from lopuhin/respect-custom-log-level
...
[MRG+1] Respect custom log level: fixes GH-1612
2017-03-09 12:47:20 +01:00
Paul Tremberth
9cfe9ae098
Do not use self.assertRaises() as context manager
2017-03-09 12:21:03 +01:00
Paul Tremberth
7be773e14a
Add SPIDER_LOADER_WARN_ONLY to toggle between spiderloader failure and warning
2017-03-07 17:40:40 +01:00
Mikhail Korobov
d3f8f3d38a
Merge pull request #2612 from redapple/dupe-spider-name-tests
...
[WIP] Add warning on duplicate spider name
2017-03-07 20:08:42 +05:00
Eugenio Lacuesta
c7bb2fa8ce
Feed exports: consistent and backwards compatible behaviour on indent
2017-03-07 11:56:00 -03:00
Eugenio Lacuesta
766b2c8453
Feed exports: enforce difference between None and 0 on indent
...
Also rename params and settings from "indent_width" to just "indent"
2017-03-07 11:56:00 -03:00
Eugenio Lacuesta
7e9153b38d
Feed exports: beautify JSON and XML
2017-03-07 11:56:00 -03:00
Paul Tremberth
0b9a18e1a1
Warn only once for all spiders
2017-03-07 15:41:17 +01:00
Paul Tremberth
978306a223
Fix dupe spider name warning string tests
2017-03-07 14:48:16 +01:00
Paul Tremberth
b6378c7ef6
Revert to using self.assert methods
2017-03-07 12:28:24 +01:00
Konstantin Lopuhin
6f55ca4643
Revert unneeded test_crawl changes
2017-03-07 14:20:52 +03:00
Paul Tremberth
11cdf58abe
Always decompress Content-Encoding: gzip at HttpCompression stage
...
Let SitemapSpider handle decoding of .xml.gz files if necessary
2017-03-07 11:02:46 +01:00
Paul Tremberth
e42b846a9f
Use body to chose response type after decompression content
2017-03-06 23:10:38 +01:00
Paul Tremberth
768f3155e5
Fix referrer policy from response headers and support explicit empty string
2017-03-06 16:20:37 +01:00
Paul Tremberth
0a17f9b55c
Merge pull request #2334 from ArturGaspar/data_uri
...
[MRG+1] data URI download handler.
2017-03-06 13:50:46 +01:00
Paul Tremberth
c68f99eed8
Refactor settings tests
2017-03-03 17:03:25 +01:00
Konstantin Lopuhin
ef04cfd237
Respect log settings in custom_settings: fixes GH-1612
...
A new root logger is installed when a crawler is created
if one was already installed before.
This allows to respect custom settings related to logging,
such as LOG_LEVEL, LOG_FILE, etc.
2017-03-03 18:01:11 +03:00
Paul Tremberth
f7e11b198e
Cleanup
2017-03-03 16:00:59 +01:00
Paul Tremberth
ecde166ee1
Refactor without MEDIA_HTTPSTATUS_LIST setting
2017-03-03 15:52:05 +01:00
Bernardas
11b31c9fbd
fix redirect change
2017-03-03 15:52:05 +01:00
Bernardas
3cef1cd451
adjust variable wording and redirect logic
2017-03-03 15:52:05 +01:00
Bernardas
6a42214716
add tests for media pipeline MEDIA_ALLOW_REDIRECTS and MEDIA_HTTPSTATUS_LIST settings
2017-03-03 15:52:05 +01:00
Mikhail Korobov
7e8453cf1e
Merge pull request #2306 from redapple/referrer-policy
...
[MRG] Referrer policies in RefererMiddleware
2017-03-03 04:05:13 +05:00
