Vostretsov Nikita
|
50c85e3114
|
Merge branch 'protego-default' of github.com:whalebot-helmsman/scrapy into protego-default
|
2019-09-16 14:24:31 +00:00 |
|
Vostretsov Nikita
|
5197b39e01
|
fix capitalization, remove commas
|
2019-09-16 14:24:25 +00:00 |
|
watsta
|
0b52fa6ca9
|
LogFormatter: Add the ability to skip log messages (#3987)
|
2019-09-16 14:12:04 +02:00 |
|
OmarFarrag
|
13735bcf34
|
Disallow media extensions unregistered with IANA (#3954)
Co-Authored-By: s-sanjay <sanjay537@gmail.com>
|
2019-09-16 14:04:06 +02:00 |
|
Eugenio Lacuesta
|
21ad8e20b9
|
Crawling rules: make link extractors optional
|
2019-09-13 17:32:42 -03:00 |
|
Eugenio Lacuesta
|
c5612f387b
|
Remove deprecated xlib module
|
2019-09-13 14:32:05 -03:00 |
|
Vostretsov Nikita
|
66145b4eaf
|
Update docs/topics/downloader-middleware.rst
Co-Authored-By: Mikhail Korobov <kmike84@gmail.com>
|
2019-09-12 18:51:00 +03:00 |
|
Vostretsov Nikita
|
171fa1cd10
|
documentation rework
|
2019-09-10 09:59:36 +00:00 |
|
Vostretsov Nikita
|
c7f2bdfdbe
|
add protego to install_requires
|
2019-09-10 08:58:52 +00:00 |
|
Vostretsov Nikita
|
6bd88711f2
|
update documentation
|
2019-09-10 08:55:37 +00:00 |
|
Vostretsov Nikita
|
db202487f0
|
newer version of protego and move up to top
|
2019-09-09 14:05:45 +00:00 |
|
Vostretsov Nikita
|
7b33fa58fa
|
Update requirements-py2.txt
Co-Authored-By: elacuesta <elacuesta@users.noreply.github.com>
|
2019-09-09 17:04:27 +03:00 |
|
Vostretsov Nikita
|
38828d3fd4
|
Update docs/topics/downloader-middleware.rst
Co-Authored-By: elacuesta <elacuesta@users.noreply.github.com>
|
2019-09-09 17:04:13 +03:00 |
|
Vostretsov Nikita
|
e418554c21
|
use proper equal
|
2019-09-09 08:12:32 +00:00 |
|
Vostretsov Nikita
|
7af8c76649
|
add pinned versions
|
2019-09-09 08:10:09 +00:00 |
|
Vostretsov Nikita
|
9578f49099
|
use protego as a default robots.txt parser
|
2019-09-09 07:36:55 +00:00 |
|
Mikhail Korobov
|
534de7395d
|
Merge pull request #3988 from elacuesta/contracts_cb_kwargs
CallbackKeywordArgumentsContract
|
2019-09-08 04:23:16 +05:00 |
|
Eugenio Lacuesta
|
b92b114633
|
[test] cb_kwargs contract
|
2019-08-31 02:44:09 -03:00 |
|
Eugenio Lacuesta
|
2061f2a382
|
[doc] cb_kwargs contract
|
2019-08-31 02:10:18 -03:00 |
|
Mikhail Korobov
|
d4b8bf18b2
|
Merge pull request #3993 from Gallaecio/documentation-coverage
Provide complete API documentation coverage of scrapy.extensions
|
2019-08-30 21:07:22 +05:00 |
|
Adrián Chaves
|
2828cb769f
|
Provide complete API documentation coverage of scrapy.extensions
|
2019-08-30 14:29:15 +02:00 |
|
Marc Hernández
|
ace2df3d14
|
Fix JSONRequest naming (#3982)
|
2019-08-30 11:03:44 +02:00 |
|
Eugenio Lacuesta
|
eb0bd2daef
|
Revert backward-incompatible change (contract priorities)
|
2019-08-29 14:01:13 -03:00 |
|
Eugenio Lacuesta
|
97a7d775f7
|
Aplly suggestions by @victor-torres
|
2019-08-29 10:51:16 -03:00 |
|
Eugenio Lacuesta
|
b6b76df057
|
CallbackKeywordArgumentsContract
|
2019-08-29 03:29:24 -03:00 |
|
Mikhail Korobov
|
ede91478e5
|
Merge pull request #3966 from anubhavp28/robotstxt_useragent
Adds ROBOTSTXT_USER_AGENT setting
|
2019-08-28 22:00:17 +05:00 |
|
Mikhail Korobov
|
93d4b0b0d7
|
Merge pull request #3973 from Gallaecio/documentation-coverage
Provide complete API documentation coverage of scrapy.exporters
|
2019-08-28 21:59:10 +05:00 |
|
Mikhail Korobov
|
b00f81c52b
|
Merge pull request #3978 from anubhavp28/doc-link-fix
Fixes a link in docs
|
2019-08-28 21:58:25 +05:00 |
|
Anubhav Patel
|
77c8ab2e62
|
makes suggested changes
|
2019-08-27 18:44:08 +05:30 |
|
Anubhav Patel
|
ad824a264b
|
fixes a link in doc
|
2019-08-27 18:30:11 +05:30 |
|
Anubhav Patel
|
3a7b949d6d
|
Adds integration with Protego robots.txt parser (#3935)
|
2019-08-27 09:41:31 +02:00 |
|
elacuesta
|
3abe7e6e6d
|
Add Bug report and Feature request templates (#3471)
|
2019-08-26 09:35:44 +02:00 |
|
Adrián Chaves
|
0fa384e80d
|
Provide complete API documentation coverage of scrapy.exporters
|
2019-08-22 20:10:42 +02:00 |
|
Mikhail Korobov
|
56948c446c
|
Merge pull request #3442 from wRAR/ciphers
Support for overriding OpenSSL ciphers
|
2019-08-19 23:15:09 +05:00 |
|
Anubhav Patel
|
00fe05e536
|
adds ROBOTSTXT_USER_AGENT setting
|
2019-08-19 09:24:16 +05:30 |
|
Tobias Hernstig
|
2b0de0606c
|
Fix merge conflicts
|
2019-08-15 18:54:28 +02:00 |
|
Tobias Hernstig
|
50c4cafe0c
|
Update documentation for logging manually
Usage of basicConfig() together with crawlerRunner is not recommended.
Update documentation to highlight this fact.
Closes #2149, #2352, #3146
|
2019-08-15 18:46:00 +02:00 |
|
Andrey Rakhmatullin
|
aaa5229e5d
|
Fixes and improvements for DOWNLOADER_CLIENT_TLS_CIPHERS.
|
2019-08-13 16:56:26 +05:00 |
|
Andrey Rakhmatullin
|
9a8edf2bf1
|
Tests for setting SSL ciphers.
|
2019-08-13 16:53:19 +05:00 |
|
Andrey Rakhmatullin
|
ce281d890d
|
Documentation for DOWNLOADER_CLIENT_TLS_CIPHERS.
|
2019-08-13 16:53:19 +05:00 |
|
Andrey Rakhmatullin
|
3384db92b4
|
Add support for setting SSL ciphers.
|
2019-08-13 16:53:19 +05:00 |
|
Mikhail Korobov
|
a95de71d8e
|
Merge pull request #3950 from elacuesta/version_updates
Remove obsolete version checks
|
2019-08-12 22:51:00 +05:00 |
|
Mikhail Korobov
|
2f0c46e762
|
Merge pull request #3946 from elacuesta/simplify_versions
[MRG+1] Simplify version reporting
|
2019-08-10 14:25:59 +05:00 |
|
Eugenio Lacuesta
|
26fb28b20f
|
PEP8-ify HTTP/1.1 downloader handler
Signed-off-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
|
2019-08-09 00:56:24 -03:00 |
|
Eugenio Lacuesta
|
d5dcc5eaef
|
Import twisted.web.client.URI directly
|
2019-08-09 00:30:58 -03:00 |
|
Eugenio Lacuesta
|
d3737d869b
|
Remove check for Twisted>=14.0
|
2019-08-09 00:21:43 -03:00 |
|
Eugenio Lacuesta
|
e17c9a48fd
|
Remove check for Twisted>=15.0.0
16.0.0 is currently the minimum supported version
|
2019-08-08 23:59:17 -03:00 |
|
Eugenio Lacuesta
|
d92f1b1858
|
Simplify import + assignment
|
2019-08-08 23:53:35 -03:00 |
|
Eugenio Lacuesta
|
b404941e0d
|
Remove import check for service_identity
service_identity.exceptions.CertificateError is available in the current minimum version (16.0.0)
|
2019-08-08 23:18:47 -03:00 |
|
Eugenio Lacuesta
|
3164543ed1
|
Remove fallback ScrapyClientContextFactory class (used in Twisted < 14.0.0)
16.0.0 is currently the minimum supported version
|
2019-08-08 23:15:03 -03:00 |
|