BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-23 09:24:04 +00:00
Code Releases Wiki Activity
7,523 Commits 39 Branches 119 Tags
Commit Graph

7523 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Eugenio Lacuesta
a940a80f58
Remove check for pyOpenSSL>=0.16 
16.2.0 is currently the minimum supported version
 2019-08-08 23:05:52 -03:00
Eugenio Lacuesta
fa9a9033f0
Remove check for Twisted>=14.0.0 
16.0.0 is currently the minimum supported version
 2019-08-08 22:58:59 -03:00
Eugenio Lacuesta
da385b56b1
Move get_openssl_version function to scrapy.utils.ssl 2019-08-08 12:44:23 -03:00
Shivam Sandbhor
3040f77468 [MRG+1] Update project.py removed one 'hack', seems irrelevant. (#3910) 
* Update project.py removed one 'hack', seems irrelevant.

As mentioned by @Gallaecio in issue #3871, the 'hack'  is cleared. I also  double checked whether the environment variable "SCRAPY_PICKLED_SETTINGS_TO_OVERRIDE" was ever set in our codebase and it turns out we didn't set it or used it anywhere else.So I guess the 'hack' was not used in the current version. Also the name of this environment variable rather doesn't suggest it was  a boolean(it is used in  an 'if' condition which has perplexed me )

* Update project.py

* Update project.py

How about this?

* Update project.py

* Update project.py

* Update scrapy/utils/project.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update scrapy/utils/project.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update project.py
 2019-08-08 16:58:22 +05:00
Mikhail Korobov
73d1b4b748
Merge pull request #3939 from Gallaecio/improve-scrapes-contract 
Report all missing fields when a scrapes contract fails
 2019-08-08 13:49:09 +05:00
Mikhail Korobov
b4556d6508
Merge pull request #3941 from starrify/ftp-storage-uri-unquote 
[MRG+1] Added: Properly handling quoted passwords in FEED_URI for FTP
 2019-08-08 13:38:17 +05:00
Adrián Chaves
9119798a5c Add test coverage for contract failures involving multiple missing fields 2019-08-08 09:52:03 +02:00
Marc Hernández
d76b6944c9 Create Request from curl command (#3862) 2019-08-08 09:43:42 +02:00
Eugenio Lacuesta
595c995ee6
Simplify version reporting 2019-08-07 15:38:04 -03:00
elacuesta
5dbeece8da [MRG+1] Drop py34 support - Update CI envs (#3892) 
* Drop py34 support

* Travis experiments

* More Travis experiments

* Bump Twisted version for py35+ (stretch)

* Remove Debian build

* Remove pinned lxml for Py34

* Fix merge error

* Remove unused tox env

* Add environment with pinned versions for py36

* Bump minimum Twisted version in py27; Envs with pinned versions for py27 and py35

* Add botocore as extra dep for py27 tests

* Update requirements-py2.txt

* Add botocore and Pillow as extra dependencies
 2019-08-07 12:36:52 +05:00
Pengyu Chen
7b755a41a1
Added: Properly handling quoted passwords in FEED_URI for FTP 2019-08-06 15:19:57 +01:00
Adrián Chaves
bff335cf7f Improve the error message in contract failures due to multiple missing fields 2019-08-05 15:47:58 +02:00
tpeng
a8621bbc29 show all the missing field when scrapes contract fails 2019-08-05 15:27:31 +02:00
Daniel Graña
eef1732374
Merge pull request #3934 from anubhavp28/typo-fix 
[MRG+1] Fixes typo
 2019-08-05 09:39:21 -03:00
Shivam Sandbhor
18d0affc01
Update reactor.py, updated 'if' sequencing , possibly eliminating a bug if portrange=None 
This should be the proper ordering.This is the explanation.
  If 'not portrange' is True ,it is guaranteed that `not hasattr(portrange, '__iter__')`  is also True  the converse of this is not always true.(for example, consider portrange=None, for such case we were executing the logic for `not hasattr(portrange, '__iter__')` . ).Such case is eliminated by this PR.
 2019-08-05 16:53:35 +05:30
Anubhav Patel
9a4cd94244 fixes typo 2019-08-03 22:46:06 +05:30
Anubhav Patel
8e813953bd [MRG+1] [GSoC 2019] Interface for robots.txt parsers (#3796) 
Make the robots.txt parser configurable through the new ROBOTSTXT_PARSER setting, support the Reppy and Robotexclusionrulesparser parsers, and allow implementing custom robots.txt parsers.
 2019-08-02 09:43:29 +02:00
Adrián Chaves
a12e8251e0 Cover Scrapy 1.7.3 in the release notes 2019-08-01 17:10:31 +02:00
sbs2001
783d61d32a [MRG+1] Update _monkeypatches.py (#3907) 
* Update _monkeypatches.py

The workarounds are not required assuming the bugs regarding urlparse are absent in  Python versions >2.7. We already exit the program if Python  version<2.7 in the __init__.py(line 17).The monkeypatches are deployed after this check at line 27  in  the __init__.py .

* Update _monkeypatches.py

Added the second workaround.

* Update _monkeypatches.py

* Update _monkeypatches.py
 2019-08-01 13:41:27 +05:00
Mikhail Korobov
cdf7889ada
Merge pull request #3923 from scrapy/pin-build-environment 
Pin Travis-ci build environment to previous default: Trusty
 2019-07-31 20:07:44 +05:00
Renne Rocha
a25e09ecdd Added constrain on lxml version based on Python version 2019-07-30 23:24:41 -03:00
Daniel Graña
7333fc02aa Pin Travis-ci build environment to previous default: Trusty 
Travis-ci changed the default build environment to Xenial as explained in https://blog.travis-ci.com/2019-04-15-xenial-default-build-environment
This causes builds meant for Debian Jessie to break as noted by @wRAR in https://github.com/scrapy/scrapy/issues/3917#issuecomment-516426389

This change pins the environment to known working ubuntu trusty distribution prior to dropping Jessie support and upgrade to Xenial as base.

Closes #1369
 2019-07-30 23:16:11 -03:00
Daniel Graña
b01d012b5a
Merge pull request #3920 from wRAR/fix-tls-logging 
Fix memory handling and error handling in utils.ssl.get_temp_key_info.
 2019-07-30 22:15:26 -03:00
Andrey Rakhmatullin
f21dc24a26 Fix memory handling and error handling in utils.ssl.get_temp_key_info. 2019-07-30 18:16:12 +05:00
Adrián Chaves
06c093f43f
Merge pull request #3905 from lucywang000/0.001 
s3 file store persist_file should accept all supported headers
 2019-07-29 19:02:14 +02:00
Adrián Chaves
04bca6af7c
Merge pull request #3894 from KristobalJunta/fix_retry_docs 
fix default RETRY_HTTP_CODES value in docs
 2019-07-29 18:20:55 +02:00
Lucy Wang
7551689c75 s3 file store should accept all supported headers 2019-07-26 09:07:29 +08:00
Mikhail Korobov
9c514b976f
Merge pull request #3450 from wRAR/tls-logging 
Log cipher, certificate and temp key info on establishing an SSL connection
 2019-07-24 01:45:40 +05:00
Mikhail Korobov
c679aefe85
Merge pull request #3660 from anubhavp28/logFormatter-doc-patch 
[MRG+1] docs for scrapy.logformatter
 2019-07-24 01:40:56 +05:00
Adrián Chaves
7843101f9a Cover Scrapy 1.7.2 in the release notes 2019-07-23 13:35:27 +02:00
Adrián Chaves
bc8672c366
Merge pull request #3896 from elacuesta/fix_configparser_import 
Fix ConfigParser import in py2
 2019-07-23 11:59:53 +02:00
Eugenio Lacuesta
7e622af4e5
Fix ConfigParser import in py2 2019-07-22 14:53:17 -03:00
Kristobal Junta
43d5b5a524 fix default RETRY_HTTP_CODES value in docs 2019-07-22 10:24:14 +03:00
Adrián Chaves
b8a43011e7 Cover Scrapy 1.7.1 in the release notes 2019-07-19 18:17:24 +02:00
Andrey Rakhmatullin
c6453800cd Remove an unneeded if. 2019-07-18 22:17:39 +05:00
Andrey Rakhmatullin
95dd2df7b5 Drop an unused import. 2019-07-18 20:51:26 +05:00
Andrey Rakhmatullin
42743fd9dd Move tls_verbose_logging extraction from __init__ to from_settings. 2019-07-18 20:49:25 +05:00
Adrián Chaves
ae4eab9843 Cover the 1.7.1 PyPI repackaging in the release notes 2019-07-18 17:02:48 +02:00
Adrián Chaves
4e23d70dd3 Bump version: 1.6.0 → 1.7.0 1.7.0 2019-07-18 15:38:25 +02:00
Adrián Chaves
a94b5bef3a Write the 1.7 release notes and cover dropping Python 2 support in the upcoming 2.0 2019-07-18 15:37:17 +02:00
Mikhail Korobov
44eb21aa51
Merge pull request #3882 from MagdalenaDeschner/master 
add instructions about how to define output file when running scrapy …
 2019-07-18 18:12:57 +05:00
Deschner, Magdalena
c44d49b238 minor PEP8 style changes 2019-07-17 13:13:52 +02:00
Maram Sumanth
0d51f9cc27 [MRG+1] Wrong value of log_count/INFO in stats (#3643) 
* Update statscollectors.py

* Update statscollectors.py

* Update statscollectors.py

* Update crawler.py

* Update crawler.py

* corrected tests

* Update test_utils_log.py

* Update crawler.py

* Update crawler.py

* interchanged order

* correced
 2019-07-17 14:36:49 +05:00
Mikhail Korobov
b2c013feca
Merge pull request #3878 from elacuesta/mergedict_to_chainmap 
Deprecate scrapy.utils.datatypes.MergeDict in favor of collections.ChainMap
 2019-07-17 14:31:50 +05:00
Deschner, Magdalena
6660020ebb remove detailed description about individual settings 2019-07-17 11:30:02 +02:00
Eugenio Lacuesta
d7074d86d2
Change condition to raise deprecation warning 2019-07-16 14:13:45 -03:00
Mikhail Korobov
377d8a7be7
Merge pull request #3877 from elacuesta/tests_deprecation_warnings 
Prevent deprecation warnings
 2019-07-16 21:58:44 +05:00
Deschner, Magdalena
e892a484e8 add instructions about how to define output file when running scrapy from script instead of cmd 2019-07-16 13:53:56 +02:00
Eugenio Lacuesta
7e3a602d56
Suggested changes 2019-07-15 12:37:09 -03:00
Eugenio Lacuesta
d5a2a70329
Fix import 2019-07-13 23:47:41 -03:00