BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-22 23:24:11 +00:00
Code Releases Wiki Activity
7,533 Commits 39 Branches 119 Tags
Commit Graph

87 Commits

Author SHA1 Message Date
Mikhail Korobov
298d709e16
Merge pull request #4162 from Gallaecio/bandit 
Add bandit to CI
 2019-11-14 23:40:01 +05:00
Adrián Chaves
5ee5508cc3 Have CI record the 10 slowest tests 2019-11-14 15:42:34 +01:00
Adrián Chaves
b8ef12cd47 Add bandit to CI 2019-11-14 12:11:28 +01:00
Adrián Chaves
1df5755699 Set the bases for testing code examples from the documentation 2019-11-08 16:00:10 +01:00
Marc Hernández
e8b1e46e85 Add pytest-flake8 (#3945) 2019-11-07 14:05:01 +01:00
Andrey Rahmatullin
229e722a03 Initial Python 2 removal (#4091) 2019-10-31 10:46:02 +01:00
Roy
74909030a5 [tox.ini] Removing obsolete py37 extra deps enviornment 
https://github.com/scrapy/scrapy/issues/4085
 2019-10-28 12:52:36 +00:00
Roy
3d4317bfe4 [tox.ini] Added python 3.8 fields 
https://github.com/scrapy/scrapy/issues/4085
 2019-10-21 18:32:30 +01:00
Vostretsov Nikita
e418554c21 use proper equal 2019-09-09 08:12:32 +00:00
Vostretsov Nikita
7af8c76649 add pinned versions 2019-09-09 08:10:09 +00:00
Vostretsov Nikita
9578f49099 use protego as a default robots.txt parser 2019-09-09 07:36:55 +00:00
Anubhav Patel
3a7b949d6d Adds integration with Protego robots.txt parser (#3935) 2019-08-27 09:41:31 +02:00
elacuesta
5dbeece8da [MRG+1] Drop py34 support - Update CI envs (#3892) 
* Drop py34 support

* Travis experiments

* More Travis experiments

* Bump Twisted version for py35+ (stretch)

* Remove Debian build

* Remove pinned lxml for Py34

* Fix merge error

* Remove unused tox env

* Add environment with pinned versions for py36

* Bump minimum Twisted version in py27; Envs with pinned versions for py27 and py35

* Add botocore as extra dep for py27 tests

* Update requirements-py2.txt

* Add botocore and Pillow as extra dependencies
 2019-08-07 12:36:52 +05:00
Anubhav Patel
8e813953bd [MRG+1] [GSoC 2019] Interface for robots.txt parsers (#3796) 
Make the robots.txt parser configurable through the new ROBOTSTXT_PARSER setting, support the Reppy and Robotexclusionrulesparser parsers, and allow implementing custom robots.txt parsers.
 2019-08-02 09:43:29 +02:00
Adrián Chaves
c7b5ad0e20 Add a Sphinx extension to generate documentation coverage information 2019-06-04 15:03:25 +02:00
Daniel Graña
8ed6beb7f9 Needs to be installed within tox env 2018-12-20 19:39:29 -03:00
Konstantin Lopuhin
1d2e2735fc Merge branch 'master' of https://github.com/patiences/scrapy into patiences-master 2018-07-09 11:59:22 +03:00
Mikhail Korobov
f11d65f7d6 TST make it clear which requirements are Python 2-only 
* rename requirements.txt to requirements-py2.txt, to make it clear they are Python 2-only
* make requirements-py3.txt consistent with requirements-py2.txt
 2018-06-29 18:34:11 +05:00
Mikhail Korobov
45f67eb64d TST exclude lxml==4.2.2 from tests, as it doesn't play well with Pillow 2018-06-28 01:02:30 +05:00
Vostretsov Nikita
c5ddfddb7e blacklist twisted version with regression in constraints file 2018-05-17 08:53:42 +00:00
Patience Shyu
aca2655c12 [WIP] Run tests for Python 3.7 2018-03-02 14:57:39 +01:00
Konstantin Lopuhin
f71df6f9ad Run tests for PyPy3 2017-12-25 14:48:06 +03:00
Hugo
44623687ab Drop support for EOL Python 3.3 2017-12-19 17:59:05 +02:00
Aditya
97d047a055 Fix link for Tox 2017-10-28 16:48:41 +05:30
rhoboro
e5d4364b2a Add tests for GCS Storage 2017-09-13 16:24:16 +09:00
Paul Tremberth
1a18587d41 Jessi toxenv: Add cryptography as per https://packages.debian.org/jessie/python-cryptography 2017-07-24 19:30:08 +02:00
harshasrinivas
4ec07ae764 Create docs/requirements.txt 2017-03-20 22:21:08 +05:30
Mikhail Korobov
de65ad3fb1 TST replace Ubuntu 12.04 tox environment with 14.04 2017-02-13 18:44:39 +05:00
Paul Tremberth
70f260d353 Don't run coverage stats when on PyPy 2017-02-01 15:19:17 +01:00
Paul Tremberth
a214731471 Add Python 3.6 tox env + Travis CI build for it 2017-01-06 15:38:35 +01:00
Paul Tremberth
22e870e955 Add Debian Jessie test env 2016-09-01 10:19:49 +02:00
Paul Tremberth
1075587dbd Add support for Sphinx 1.4 
See http://www.sphinx-doc.org/en/stable/changes.html#release-1-4-released-mar-28-2016

sphinx_rtd_theme has become optional, needs to be added to reqs

https://github.com/sphinx-doc/sphinx/pull/2320 changes node entries tuples
to 5 values instead of 4

`sh` syntax highlighting added very locally in selectors.rst
because of this warning/error with Sphinx 1.4:

```
Warning, treated as error:
/home/paul/src/scrapy/docs/topics/selectors.rst:743:
WARNING: Could not lex literal_block as "python". Highlighting skipped.
```
 2016-03-30 14:40:52 +02:00
Konstantin Lopuhin
32cd8c9165 add direct test for S3FilesStore 2016-02-15 19:59:48 +03:00
Konstantin Lopuhin
408bc1580b Pass env variables required for running tests against real s3 via tox. 2016-02-15 19:59:47 +03:00
Konstantin Lopuhin
c3fec83e7e use botocore by default, boto is still used in "precise" env 2016-02-15 19:59:47 +03:00
Paul Tremberth
a93d49a64c Add Python 3.5 tox env and Python 3.4 tests in Travis CI 2016-01-13 13:09:16 +01:00
Mikhail Korobov
669be0a2b0 TST don't use broken Pillow version in tests 2015-10-06 17:24:52 +05:00
Daniel Graña
cb9577a6d4 Do not be verbose with coverage report by default 2015-08-13 19:02:36 -03:00
Daniel Graña
108c3e63a5 Add coverage report trough codecov.io 2015-08-13 13:56:24 -03:00
Mikhail Korobov
75d3d03382 add service_identity to scrapy install_requires 2015-08-11 20:14:22 +05:00
nyov
4d41cc0dc4 PY3 split requirements into files 2015-08-08 04:19:37 +05:00
Julia Medina
ec54942925 Create separate testenvs to build docs and check links 2015-05-14 20:54:26 -03:00
Julia Medina
a555778013 Build docs in a temporary folder to avoid caching 2015-05-14 20:53:25 -03:00
Mikhail Korobov
fbb1078f58 Merge pull request #1060 from Curita/python-logging 
[MRG+1] Python logging
 2015-04-29 23:20:34 +05:00
Daniel Graña
5eb098a939 Merge pull request #1168 from scrapy/service-identity 
install service_identity package in tests to prevent warnings
 2015-04-28 23:48:58 -03:00
Julia Medina
c2d716807a Use LogCapture in testfixtures package for tests 
This allows to remove `get_testlog` helper, `flushLoggedErrors` from
twisted.trial.unittest.TestCase and Twisted log observers created for
each test on conftest.py.
 2015-04-22 17:24:40 -03:00
Julia Medina
ffc60910aa Remove djangoitem since we moved it to scrapy/scrapy-djangoitem 2015-04-19 13:09:25 -03:00
Mikhail Korobov
271f7f54d5 TST install service_identity package in tests to prevent warnings 
Also, Twisted version is bumped for Python 3.x tests, just in case.

service_identity is not added to Scrapy requirements because Scrapy
supports older Twisted / PyOpenSSL versions which don't use it.
 2015-04-16 22:54:51 +05:00
Berker Peksag
31e5f164d4 Import unittest.mock if available. 
mock is in the stdlib since Python 3.3.
 2015-03-08 15:03:04 +02:00
Daniel Graña
c05e99a4f4 oops, restore Pillow from precise test requirements 2014-09-10 12:21:08 -03:00