Julia Medina
57f87b95d4
Merge pull request #1528 from scrapy/create-crawler
...
public Crawler.create_crawler method
2015-10-30 12:36:35 -03:00
Mikhail Korobov
a49c82ad62
TST improve CrawlerRunner tests
...
* use CrawlerRunner.create_crawler instead of get_crawler helper in test_crawl;
* add a test for loading spiders by name;
* add a test for passing Crawler objects instead of Spider objects;
* add a test for CrawlerRunner.join
2015-10-30 20:24:37 +05:00
Jakob de Maeyer
32ff4cc1d7
PEP8ify scrapy.settings tests
2015-10-29 19:52:19 +01:00
Jakob de Maeyer
51ca84c9b4
Add missing tests for scrapy.settings module
2015-10-29 17:34:18 +01:00
Jakob de Maeyer
8307c1212f
Add ExecutionEngine.close() method
2015-10-29 10:03:29 +01:00
Jakob de Maeyer
03f1720afb
Fix backwards-compatibility for users who explicitly set _BASE settings
2015-10-27 14:22:56 +01:00
Jakob de Maeyer
f249b309ab
Move scrapy.utils.conf.remove_none_values to s.u.python.without_none_values
2015-10-27 12:43:33 +01:00
Jakob de Maeyer
26586ef5a6
Deprecate _BASE settings, unify _BASE backwards-compatibility
2015-10-27 12:43:23 +01:00
Jakob de Maeyer
a769a1ef78
Introduce BaseSettings with full dictionary interface
2015-10-27 12:38:52 +01:00
Mikhail Korobov
215905bdb6
PY3 port bench, startproject, genspider, list and runspider commands
2015-10-12 17:59:49 +05:00
Νικόλαος-Διγενής Καραγιάννης
d66efb13ba
test xml nodename with dots
2015-10-07 14:43:47 +03:00
preetwinder
47c8e2ba78
Restructure tests for add_http_if_no_scheme function
2015-09-24 17:57:25 +00:00
preetwinder
8c629eee3e
adds docstring, tests and correction
2015-09-24 17:28:44 +00:00
preetwinder
c517951a48
add_scheme_if_missing for scrapy shell command
2015-09-24 17:28:44 +00:00
Mikhail Korobov
916141a08f
TST pin pytest to 2.7.3
2015-09-21 21:12:52 +05:00
Daniel Graña
51b1249033
Revert "test suite requires recent queuelib to pass because it depends on queuelib test suite"
...
This reverts commit a3390afc66134e77f98ae3bfae7bc23479ed8566.
2015-09-18 02:27:07 -03:00
Daniel Graña
a3390afc66
test suite requires recent queuelib to pass because it depends on queuelib test suite
2015-09-17 19:42:29 -03:00
Daniel Graña
3c596dcf46
Merge pull request #1467 from dacjames/master
...
add support for a nested loaders
2015-09-16 22:03:19 -03:00
Elias Dorneles
430e90f4b2
minor refactor on metarefresh redirect mware test
2015-09-13 12:29:41 -03:00
Elias Dorneles
81950f773d
made encoding conversion more explicit, added test for header with utf-8 encoding replicating what browsers do
2015-09-11 18:51:48 -03:00
Elias Dorneles
179c4588ca
adding test for latin1 location
2015-09-10 19:36:47 -03:00
Elias Dorneles
defa899135
PY3 port redirect middleware
2015-09-10 16:31:59 -03:00
Daniel Graña
9a64d8ff97
fix scrapy squeue tests after recent changes to queuelib
2015-09-09 15:55:55 -03:00
Daniel Collins
311d5cd495
split nested_loader into seperate methods
2015-09-04 13:15:48 -07:00
Daniel Collins
425e35ee90
removed commented code and add test for error handling
2015-09-04 13:15:48 -07:00
Daniel Collins
1f95af3c07
add support for a nested loaders
2015-09-04 13:15:48 -07:00
Daniel Graña
dd473145f2
Merge pull request #1473 from ArturGaspar/master
...
[MRG+1] Support for returning deferreds in middlewares
2015-09-02 13:04:26 -03:00
Artur Gaspar
dd83f6123d
Test for same response object in downloader middleware test.
2015-09-02 11:36:15 -03:00
Artur Gaspar
2748b38592
Test for not calling the download function when downloader middleware returns a response in process_request.
2015-09-02 11:15:37 -03:00
Artur Gaspar
1f4af9d81e
Test for robots.txt middleware for logger not being called when the request for robots.txt is ignored.
2015-09-02 10:33:53 -03:00
Artur Gaspar
668e5fd257
Test for robots.txt middleware for processing a request for which the robots.txt parser is ready.
2015-09-02 10:23:24 -03:00
Artur Gaspar
a6a629e707
Call actual error logger in robots.txt middleware tests.
2015-09-02 10:23:24 -03:00
Artur Gaspar
9ce9a293a6
Always check robots.txt before making another request in RobotsTxtMiddleware.
2015-09-02 10:23:24 -03:00
Daniel Graña
3919ad64c5
Merge pull request #1470 from scrapy/py3-middlewares
...
[MRG] port some downloader middlewares to Python 3
2015-09-02 00:17:37 -03:00
Olaf Dietsche
2c28b53cc0
Add tests with incomplete buttons
2015-09-01 12:35:49 +02:00
Mikhail Korobov
c44cafe4f5
PY3 fix UserAgentMiddleware tests
2015-09-01 04:11:33 +05:00
Mikhail Korobov
3a9c73bc5d
PY3 fix DownloaderStats middleware tests
2015-09-01 04:09:15 +05:00
Mikhail Korobov
78a4cd0f1c
PY3 fix HttpAuthMiddleware tests
2015-09-01 04:00:26 +05:00
Mikhail Korobov
179a4409fb
PY3 enable DownloadTimeoutMiddleware tests
2015-09-01 03:58:25 +05:00
Mikhail Korobov
3cf1911a92
PY3 fix DefaultHeadersMiddleware tests
2015-09-01 03:57:05 +05:00
Mikhail Korobov
45101829a5
PY3 fix AjacCrawlable middleware tests
2015-09-01 03:49:52 +05:00
Olaf Dietsche
e5f26078fa
Add test: submit form with button control
2015-08-31 19:57:45 +02:00
Mikhail Korobov
44bfcbcf0f
TST split LinkExtractorTestCase.test_extraction into several methods; remove duplicated test
2015-08-31 00:49:38 +05:00
Mikhail Korobov
f7052413e0
PY3 raise an exception if bytes are passed as url to Link constructor
2015-08-28 23:04:02 +05:00
Mikhail Korobov
9bfe6ece59
Merge branch 'master' into py3-linkextractors
...
Conflicts:
scrapy/linkextractors/lxmlhtml.py
tests/test_linkextractors.py
2015-08-28 04:53:32 +05:00
Mikhail Korobov
ff24cbbc47
PY3 depth, offsite and referer spider middlewares; Crawler
2015-08-28 04:11:30 +05:00
Mikhail Korobov
d5984bbea9
PY3 port scrapy.spiders
2015-08-28 04:11:30 +05:00
Mikhail Korobov
f2edbd05de
PY3 port LinkExtractor
...
* tests for other link extractors are moved to test_linkextractors_deprecated.py
* in Python 3 Link is converted to use native strings for urls
* minor cleanups
2015-08-28 04:11:30 +05:00
Mikhail Korobov
f46a450080
refactor test_linkextractors
...
* rename LinkExtractorTestCase to BaseSgmlLinkExtractorTestCase
* add BaseLinkExtractorTestCase link extractor tests can inherit from
and decouple it from SgmlLinkExtractor
* add an extra check for deny_extensions
* xfail test_restrict_xpaths_with_html_entities for LxmlLinkExtractor explicitly
2015-08-28 04:11:30 +05:00
Julia Medina
aa31811cfd
Merge pull request #1214 from rgtk/link-rel
...
[MRG+1] Support link rel attribute with multiple values
2015-08-27 19:57:54 -03:00
