BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-23 18:44:05 +00:00
Code Releases Wiki Activity
7,310 Commits 39 Branches 119 Tags
Commit Graph

7310 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Vostretsov Nikita
db202487f0 newer version of protego and move up to top 2019-09-09 14:05:45 +00:00
Vostretsov Nikita
7b33fa58fa
Update requirements-py2.txt 
Co-Authored-By: elacuesta <elacuesta@users.noreply.github.com>
 2019-09-09 17:04:27 +03:00
Vostretsov Nikita
38828d3fd4
Update docs/topics/downloader-middleware.rst 
Co-Authored-By: elacuesta <elacuesta@users.noreply.github.com>
 2019-09-09 17:04:13 +03:00
Vostretsov Nikita
e418554c21 use proper equal 2019-09-09 08:12:32 +00:00
Vostretsov Nikita
7af8c76649 add pinned versions 2019-09-09 08:10:09 +00:00
Vostretsov Nikita
9578f49099 use protego as a default robots.txt parser 2019-09-09 07:36:55 +00:00
Mikhail Korobov
534de7395d
Merge pull request #3988 from elacuesta/contracts_cb_kwargs 
CallbackKeywordArgumentsContract
 2019-09-08 04:23:16 +05:00
Eugenio Lacuesta
b92b114633
[test] cb_kwargs contract 2019-08-31 02:44:09 -03:00
Eugenio Lacuesta
2061f2a382
[doc] cb_kwargs contract 2019-08-31 02:10:18 -03:00
Mikhail Korobov
d4b8bf18b2
Merge pull request #3993 from Gallaecio/documentation-coverage 
Provide complete API documentation coverage of scrapy.extensions
 2019-08-30 21:07:22 +05:00
Adrián Chaves
2828cb769f Provide complete API documentation coverage of scrapy.extensions 2019-08-30 14:29:15 +02:00
Marc Hernández
ace2df3d14 Fix JSONRequest naming (#3982) 2019-08-30 11:03:44 +02:00
Eugenio Lacuesta
eb0bd2daef
Revert backward-incompatible change (contract priorities) 2019-08-29 14:01:13 -03:00
Eugenio Lacuesta
97a7d775f7
Aplly suggestions by @victor-torres 2019-08-29 10:51:16 -03:00
Eugenio Lacuesta
b6b76df057
CallbackKeywordArgumentsContract 2019-08-29 03:29:24 -03:00
Mikhail Korobov
ede91478e5
Merge pull request #3966 from anubhavp28/robotstxt_useragent 
Adds ROBOTSTXT_USER_AGENT setting
 2019-08-28 22:00:17 +05:00
Mikhail Korobov
93d4b0b0d7
Merge pull request #3973 from Gallaecio/documentation-coverage 
Provide complete API documentation coverage of scrapy.exporters
 2019-08-28 21:59:10 +05:00
Mikhail Korobov
b00f81c52b
Merge pull request #3978 from anubhavp28/doc-link-fix 
Fixes a link in docs
 2019-08-28 21:58:25 +05:00
Anubhav Patel
77c8ab2e62 makes suggested changes 2019-08-27 18:44:08 +05:30
Anubhav Patel
ad824a264b fixes a link in doc 2019-08-27 18:30:11 +05:30
Anubhav Patel
3a7b949d6d Adds integration with Protego robots.txt parser (#3935) 2019-08-27 09:41:31 +02:00
elacuesta
3abe7e6e6d Add Bug report and Feature request templates (#3471) 2019-08-26 09:35:44 +02:00
Adrián Chaves
0fa384e80d Provide complete API documentation coverage of scrapy.exporters 2019-08-22 20:10:42 +02:00
Mikhail Korobov
56948c446c
Merge pull request #3442 from wRAR/ciphers 
Support for overriding OpenSSL ciphers
 2019-08-19 23:15:09 +05:00
Anubhav Patel
00fe05e536 adds ROBOTSTXT_USER_AGENT setting 2019-08-19 09:24:16 +05:30
Andrey Rakhmatullin
aaa5229e5d Fixes and improvements for DOWNLOADER_CLIENT_TLS_CIPHERS. 2019-08-13 16:56:26 +05:00
Andrey Rakhmatullin
9a8edf2bf1 Tests for setting SSL ciphers. 2019-08-13 16:53:19 +05:00
Andrey Rakhmatullin
ce281d890d Documentation for DOWNLOADER_CLIENT_TLS_CIPHERS. 2019-08-13 16:53:19 +05:00
Andrey Rakhmatullin
3384db92b4 Add support for setting SSL ciphers. 2019-08-13 16:53:19 +05:00
Mikhail Korobov
a95de71d8e
Merge pull request #3950 from elacuesta/version_updates 
Remove obsolete version checks
 2019-08-12 22:51:00 +05:00
Mikhail Korobov
2f0c46e762
Merge pull request #3946 from elacuesta/simplify_versions 
[MRG+1] Simplify version reporting
 2019-08-10 14:25:59 +05:00
Eugenio Lacuesta
26fb28b20f
PEP8-ify HTTP/1.1 downloader handler 
Signed-off-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
 2019-08-09 00:56:24 -03:00
Eugenio Lacuesta
d5dcc5eaef
Import twisted.web.client.URI directly 2019-08-09 00:30:58 -03:00
Eugenio Lacuesta
d3737d869b
Remove check for Twisted>=14.0 2019-08-09 00:21:43 -03:00
Eugenio Lacuesta
e17c9a48fd
Remove check for Twisted>=15.0.0 
16.0.0 is currently the minimum supported version
 2019-08-08 23:59:17 -03:00
Eugenio Lacuesta
d92f1b1858
Simplify import + assignment 2019-08-08 23:53:35 -03:00
Eugenio Lacuesta
b404941e0d
Remove import check for service_identity 
service_identity.exceptions.CertificateError is available in the current minimum version (16.0.0)
 2019-08-08 23:18:47 -03:00
Eugenio Lacuesta
3164543ed1
Remove fallback ScrapyClientContextFactory class (used in Twisted < 14.0.0) 
16.0.0 is currently the minimum supported version
 2019-08-08 23:15:03 -03:00
Eugenio Lacuesta
a940a80f58
Remove check for pyOpenSSL>=0.16 
16.2.0 is currently the minimum supported version
 2019-08-08 23:05:52 -03:00
Eugenio Lacuesta
fa9a9033f0
Remove check for Twisted>=14.0.0 
16.0.0 is currently the minimum supported version
 2019-08-08 22:58:59 -03:00
Eugenio Lacuesta
da385b56b1
Move get_openssl_version function to scrapy.utils.ssl 2019-08-08 12:44:23 -03:00
Shivam Sandbhor
3040f77468 [MRG+1] Update project.py removed one 'hack', seems irrelevant. (#3910) 
* Update project.py removed one 'hack', seems irrelevant.

As mentioned by @Gallaecio in issue #3871, the 'hack'  is cleared. I also  double checked whether the environment variable "SCRAPY_PICKLED_SETTINGS_TO_OVERRIDE" was ever set in our codebase and it turns out we didn't set it or used it anywhere else.So I guess the 'hack' was not used in the current version. Also the name of this environment variable rather doesn't suggest it was  a boolean(it is used in  an 'if' condition which has perplexed me )

* Update project.py

* Update project.py

How about this?

* Update project.py

* Update project.py

* Update scrapy/utils/project.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update scrapy/utils/project.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update project.py
 2019-08-08 16:58:22 +05:00
Mikhail Korobov
73d1b4b748
Merge pull request #3939 from Gallaecio/improve-scrapes-contract 
Report all missing fields when a scrapes contract fails
 2019-08-08 13:49:09 +05:00
Mikhail Korobov
b4556d6508
Merge pull request #3941 from starrify/ftp-storage-uri-unquote 
[MRG+1] Added: Properly handling quoted passwords in FEED_URI for FTP
 2019-08-08 13:38:17 +05:00
Adrián Chaves
9119798a5c Add test coverage for contract failures involving multiple missing fields 2019-08-08 09:52:03 +02:00
Marc Hernández
d76b6944c9 Create Request from curl command (#3862) 2019-08-08 09:43:42 +02:00
Eugenio Lacuesta
595c995ee6
Simplify version reporting 2019-08-07 15:38:04 -03:00
elacuesta
5dbeece8da [MRG+1] Drop py34 support - Update CI envs (#3892) 
* Drop py34 support

* Travis experiments

* More Travis experiments

* Bump Twisted version for py35+ (stretch)

* Remove Debian build

* Remove pinned lxml for Py34

* Fix merge error

* Remove unused tox env

* Add environment with pinned versions for py36

* Bump minimum Twisted version in py27; Envs with pinned versions for py27 and py35

* Add botocore as extra dep for py27 tests

* Update requirements-py2.txt

* Add botocore and Pillow as extra dependencies
 2019-08-07 12:36:52 +05:00
Pengyu Chen
7b755a41a1
Added: Properly handling quoted passwords in FEED_URI for FTP 2019-08-06 15:19:57 +01:00
Adrián Chaves
bff335cf7f Improve the error message in contract failures due to multiple missing fields 2019-08-05 15:47:58 +02:00