Mikhail Korobov
78a4cd0f1c
PY3 fix HttpAuthMiddleware tests
2015-09-01 04:00:26 +05:00
Mikhail Korobov
179a4409fb
PY3 enable DownloadTimeoutMiddleware tests
2015-09-01 03:58:25 +05:00
Mikhail Korobov
3cf1911a92
PY3 fix DefaultHeadersMiddleware tests
2015-09-01 03:57:05 +05:00
Mikhail Korobov
45101829a5
PY3 fix AjacCrawlable middleware tests
2015-09-01 03:49:52 +05:00
Olaf Dietsche
e5f26078fa
Add test: submit form with button control
2015-08-31 19:57:45 +02:00
Mikhail Korobov
44bfcbcf0f
TST split LinkExtractorTestCase.test_extraction into several methods; remove duplicated test
2015-08-31 00:49:38 +05:00
Mikhail Korobov
f7052413e0
PY3 raise an exception if bytes are passed as url to Link constructor
2015-08-28 23:04:02 +05:00
Mikhail Korobov
9bfe6ece59
Merge branch 'master' into py3-linkextractors
...
Conflicts:
scrapy/linkextractors/lxmlhtml.py
tests/test_linkextractors.py
2015-08-28 04:53:32 +05:00
Mikhail Korobov
ff24cbbc47
PY3 depth, offsite and referer spider middlewares; Crawler
2015-08-28 04:11:30 +05:00
Mikhail Korobov
d5984bbea9
PY3 port scrapy.spiders
2015-08-28 04:11:30 +05:00
Mikhail Korobov
f2edbd05de
PY3 port LinkExtractor
...
* tests for other link extractors are moved to test_linkextractors_deprecated.py
* in Python 3 Link is converted to use native strings for urls
* minor cleanups
2015-08-28 04:11:30 +05:00
Mikhail Korobov
f46a450080
refactor test_linkextractors
...
* rename LinkExtractorTestCase to BaseSgmlLinkExtractorTestCase
* add BaseLinkExtractorTestCase link extractor tests can inherit from
and decouple it from SgmlLinkExtractor
* add an extra check for deny_extensions
* xfail test_restrict_xpaths_with_html_entities for LxmlLinkExtractor explicitly
2015-08-28 04:11:30 +05:00
Julia Medina
aa31811cfd
Merge pull request #1214 from rgtk/link-rel
...
[MRG+1] Support link rel attribute with multiple values
2015-08-27 19:57:54 -03:00
Rafał Gutkowski
cb3007c066
support link rel attribute with multiple values
2015-08-27 20:13:47 +02:00
Mikhail Korobov
8ecc4544b3
enable console tests in PY3
2015-08-27 21:37:24 +05:00
Mikhail Korobov
9616d91e4a
Merge pull request #1444 from cyberplant/bpython_support
...
[MRG +1] bpython support
2015-08-27 21:28:05 +05:00
Daniel Graña
71bd79e70f
Merge pull request #1212 from Digenis/test-fix-render_template_file
...
fix utils.template.render_templatefile() bug +test
2015-08-26 14:46:42 -03:00
Νικόλαος-Διγενής Καραγιάννης
f5c54072ec
Test utils.template.render_templatefile()
2015-08-26 20:44:33 +03:00
Mikhail Korobov
68a47ade46
PY3 port test_logformatter
2015-08-26 02:34:21 +05:00
Mikhail Korobov
7da769feb2
enable test_stats and test_utils_log tests in Python 3
2015-08-26 01:58:59 +05:00
Jakob de Maeyer
d164398a27
Fix RedirectMiddleware not honouring meta handle_httpstatus keys
2015-08-21 13:22:42 +02:00
nyov
509cc8d41e
Add support for bpython console.
...
Adds support for configuration of shells from scrapy.cfg
and SCRAPY_PYTHON_SHELL.
config snippet:
cat <<EOF >> ~/.scrapy.cfg
[settings]
# shell can be one of ipython, bpython or python;
# to be tried as the interactive python console
# (in above order, unless set here).
shell = python
EOF
(closes #270 , #1100 , #1301 )
2015-08-21 01:12:58 +01:00
Andrew Scorpil
de15fcdf33
[LinkExtractors] Ignore bogus links
...
(rebased the code for scrapy 1.0 and made a few code improvements --nyov)
2015-08-15 00:16:39 +00:00
nyov
1b4fd3a8df
Support anonymous connections in S3DownloadHandler
...
Also consider any unknown keyword args for S3DownloadHandler as
arguments to pass on to S3Connection (e.g. proxy settings).
2015-08-14 22:12:12 +00:00
Daniel Graña
27077d2b6c
Forward coverage envvars needed to cover subprocess
2015-08-13 21:07:45 -03:00
Daniel Graña
43fc2f23b8
Increase coverage of version command
2015-08-13 19:30:06 -03:00
Daniel Graña
108c3e63a5
Add coverage report trough codecov.io
2015-08-13 13:56:24 -03:00
Daniel Graña
15c1300d35
Merge pull request #1409 from eliasdorneles/migrate-parsel
...
[MRG+1] Migrating selectors to use parsel
2015-08-11 16:22:58 -03:00
Elias Dorneles
766c255152
upgrade parsel and add shim for deprecated selectorlist methods
2015-08-11 15:20:33 -03:00
Daniel Graña
457b97c13c
Merge pull request #1388 from scrapy/dupefilter-persist
...
Dupefilter persistence
2015-08-11 14:16:34 -03:00
Elias Dorneles
e50610bd3a
set base_url in kwargs to be fully backward compatible
2015-08-11 14:09:34 -03:00
Elias Dorneles
8ef5aa2ffc
using bytes for response body in tests
2015-08-11 14:09:34 -03:00
Elias Dorneles
3a03ef7c08
cleanup tests for selectors and translators
2015-08-11 14:09:34 -03:00
Elias Dorneles
12579b9afa
warning when ambiguous root arguments and minor cleanups
2015-08-11 14:09:34 -03:00
Elias Dorneles
2fe6d128f5
upgrade parsel and using promoted root attribute
2015-08-11 14:09:16 -03:00
Elias Dorneles
94c3a345b7
remove deprecated module lxmldocument
2015-08-11 14:09:00 -03:00
Elias Dorneles
17d7347a36
update minimal parsel version, add deprecated classes for csstranslator modules, fix test
2015-08-11 14:09:00 -03:00
Elias Dorneles
01d948f0fd
remove selector support for LxmlDocument DOM cache and add deprecation warning for _root argument
2015-08-11 14:08:43 -03:00
Elias Dorneles
ce21884a97
migrating scrapy Selector to use Parsel
2015-08-11 14:08:43 -03:00
Mikhail Korobov
fa123b335b
Merge pull request #1421 from dangra/nyov/lazyload-downloadhandlers
...
lazy-load downloadhandlers (continuation of #1357 )
2015-08-11 18:59:47 +05:00
Mikhail Korobov
fa3d84b050
Merge pull request #1420 from scrapy/py3-trackrefs
...
PY3: port scrapy.utils.trackref
2015-08-11 18:21:34 +05:00
nyov
8c7997083f
lazy-loading for DownloadHandlers
2015-08-10 16:52:44 -03:00
Daniel Graña
accd28cc71
PY3: port scrapy.utils.trackref
2015-08-10 15:13:29 -03:00
Daniel Graña
8d45b3c481
Do not fail representing non-http requests
2015-08-10 14:10:58 -03:00
Mikhail Korobov
56be610e6e
TST a test for --profile option
2015-08-08 04:54:47 +05:00
nyov
17b5e9fb86
PY3 response bodies as bytes
2015-08-08 04:23:32 +05:00
nyov
45d441d444
PY3 fix test loader
2015-08-08 04:23:16 +05:00
nyov
34eced0ee8
PY3 fix tests pipelines images
2015-08-08 04:23:06 +05:00
nyov
991197003b
PY3 fix tests pipelines files
2015-08-08 04:22:58 +05:00
nyov
6e762ce25c
PY3 renames (six types)
2015-08-08 04:21:58 +05:00
