BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-23 21:24:20 +00:00
Code Releases Wiki Activity
6,779 Commits 39 Branches 119 Tags
Commit Graph

6779 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Daniel Graña
19ad94105f pywin32 is required to run tests under windows 2018-08-15 01:33:38 -03:00
Daniel Graña
dd75297e3f Run Appveyor CI for master and release branches only, but also PRs 2018-08-15 01:33:38 -03:00
Daniel Graña
4c53957f5b Skip leveldb tests on windows 2018-08-15 01:33:38 -03:00
Daniel Graña
1d25c98eb3 Add appveyor.yml 2018-08-15 01:33:38 -03:00
Stas Glubokiy
b4b1e48343 Add ability to use FormRequest in contracts 2018-08-11 22:18:43 +03:00
Stas Glubokiy
8fc017d345 Add dont_filter to ContractsManager requests 2018-08-11 19:25:33 +03:00
Stas Glubokiy
76220e8733 Use inspect.getmembers in tested_methods_from_spidercls 2018-08-11 18:49:12 +03:00
Stas Glubokiy
8bc536d88b Merge branch 'master' of github.com:StasDeep/scrapy into feature/issue-3364 2018-08-11 18:34:37 +03:00
Stas Glubokiy
ebbde57eca Add custom contracts tests 2018-08-11 18:21:07 +03:00
Stas Glubokiy
fb7d4cbce3 Add error handling in contracts 2018-08-11 16:08:26 +03:00
Konstantin Lopuhin
8a4e51a19b
Merge pull request #3371 from StasDeep/fix/issue-3370 
[MRG+1] Fix contract errback (#3370)
 2018-08-10 17:27:45 +03:00
Stas Glubokiy
16dad81715 Fix contract errback 2018-08-09 21:07:25 +03:00
Konstantin Lopuhin
c8f3d07e86
Merge pull request #3367 from testingcan/doc-tutorial-update 
[MRG+1] Updated tutorial.rst to include more and up-to-date beginner resources
 2018-08-07 10:05:44 +03:00
Raphael Wuillemier
d3aa1e8666 Updated tutorial.rst to include more and up-to-date beginner resources 2018-08-06 17:40:31 +02:00
Konstantin Lopuhin
eb007bafab
Merge pull request #3359 from scrapy/unused-imports 
remove unused imports from scrapy/settings/__init__.py
 2018-08-02 22:22:40 +03:00
Konstantin Lopuhin
d90bdd5d86
Merge pull request #3358 from BurnzZ/fix-error-msg-feedexport-when-using-s3 
[MRG+1] provide better error message when disabling s3 exporter
 2018-08-02 22:21:23 +03:00
Daniel Graña
db714f5a07
Merge pull request #3283 from CCInCharge/issue3247 
Fix #3247: Allow scrapy.FormRequest.from_response method to handle duplicate keys
 2018-08-02 14:49:14 -03:00
Mikhail Korobov
c87a4f5c6f remove unused imports from scrapy/settings/__init__.py 
This is a follow-up to https://github.com/scrapy/scrapy/pull/3327
 2018-08-01 01:45:16 +05:00
Kevin Lloyd Bernal
980be4cb4b provide better error message when disabling s3 exporter 2018-07-31 19:05:04 +08:00
Daniel Graña
93afe18587
Merge pull request #2956 from elacuesta/dupefilter_from_crawler 
Add from_crawler support to dupefilters
 2018-07-26 17:02:31 -03:00
Eugenio Lacuesta
999341b60b Simplify dupefilter creation 2018-07-26 16:24:58 -03:00
Eugenio Lacuesta
9e14f8c7e4 Fix test for dupefilter 2018-07-26 16:24:58 -03:00
Eugenio Lacuesta
0089a4ab31 Add test for direct creation of dupefilter (no from_crawler/from_settings) 2018-07-26 16:24:58 -03:00
Eugenio Lacuesta
d306fe30ac Test dupefilter creation by the Scheduler 2018-07-26 16:24:58 -03:00
Eugenio Lacuesta
701cd2ff9d Add from_crawler support to dupefilters 2018-07-26 16:24:58 -03:00
CCInCharge
782f866572 Fix #3247: Allow scrapy.FormRequest.from_response method to handle duplicate keys 2018-07-25 21:46:42 -07:00
Mikhail Korobov
895df937a3
Merge pull request #3350 from granadoho/master 
make amendments to grammer
 2018-07-25 23:50:06 +03:00
Mikhail Korobov
733ac6327c
Merge pull request #3347 from korigod/doc_copyright-notice 
Doc: update copyright notice
 2018-07-25 23:49:08 +03:00
Daniel Graña
b6abd45926
Merge pull request #3342 from elacuesta/copy-request-flags 
[MRG+1] Include flags when copying requests
 2018-07-25 11:58:25 -03:00
Daniel Graña
732d7e1cce
Merge pull request #3348 from elacuesta/enhancement/alternate-feedexport-constructors 
[MRG+1] Add from_crawler constructor for feed exporters and storages
 2018-07-25 11:55:16 -03:00
Malcolm Granado Ho Yong Liang
48866457b3 make amendments to grammer 2018-07-25 14:38:37 +08:00
Eugenio Lacuesta
917c1fde68 Merge branch 'master' into enhancement/alternate-feedexport-constructors 2018-07-20 12:09:50 -03:00
Eugenio Lacuesta
98d74d1083 Requested changes 2018-07-20 12:08:49 -03:00
Andrei Korigodski
7020c3e452
Doc: update copyright notice 
The years are updated. The hyphen is replaced with an en dash.
 2018-07-20 14:46:57 +03:00
Eugenio Lacuesta
6a38fc39f8 Include flags when copying requests 2018-07-19 11:56:23 -03:00
Mikhail Korobov
258121db04
Merge pull request #3335 from scrapy/docs-retry-codes 
update default RETRY_HTTP_CODES on docs
 2018-07-18 19:33:50 +05:00
Valdir Stumm Junior
c61e8a617f
[doc] update default RETRY_HTTP_CODES 2018-07-13 11:55:16 -07:00
Mikhail Korobov
c86213317d 1.5.1 release notes 2018-07-12 02:17:08 +05:00
Daniel Graña
01e317e79b
Merge pull request #3327 from nyov/deprecations 
[MRG+1] Remove deprecated CrawlerSettings class and Settings attributes
 2018-07-11 12:01:49 -03:00
Daniel Graña
f8f5f463e6
Merge pull request #3253 from rpkilby/depth-stats 
[MRG+1] Update depth middleware stats (fixes #3245)
 2018-07-11 12:00:08 -03:00
Daniel Graña
9c6d265280
Merge pull request #3152 from nctl144/ftp_linkextractors 
[MRG+1] add ftp to the scheme list
 2018-07-11 11:50:02 -03:00
Mikhail Korobov
8d5320dcd2
Merge pull request #3326 from lopuhin/patiences-master 
Python 3.7 support
 2018-07-11 17:36:55 +05:00
Konstantin Lopuhin
9428a4a3aa More visible telnet conch message 
Capture traceback when trying to import required twisted modules,
print it in case telnet is enabled, and mention settings variable
that can be used to supress the message.
Thanks @kmike!
 2018-07-09 21:06:51 +03:00
nyov
4f6778aa73 Remove deprecated CrawlerSettings class and Settings attributes 2018-07-09 17:16:31 +00:00
Konstantin Lopuhin
92b504eae5 Fix telnet warnings in tests 
Disable telnet console if it's not available, else we'll get an extra
warning about failure to enable it, and tests will fail.
 2018-07-09 13:44:02 +03:00
Konstantin Lopuhin
b3cd12dc48 Try to get python3.7 by using xenial base and sudo 
See https://github.com/travis-ci/travis-ci/issues/9815#issuecomment-401756442
and https://github.com/travis-ci/travis-ci/issues/9815#issuecomment-402045581
 2018-07-09 13:12:47 +03:00
Konstantin Lopuhin
f4f39057cb Make csviter work on python 3.7 
PEP 479 does not allow for StopIteration in generators.  Instead,
handle it explicitly, also use a for loop which looks simpler.
 2018-07-09 12:46:45 +03:00
Konstantin Lopuhin
2773fe09e4 Make "docs" the last build, even though it still uses python3.6 for now 2018-07-09 12:36:58 +03:00
Konstantin Lopuhin
cf9399acc1 Use python 3.7 on travis 2018-07-09 12:26:56 +03:00
Konstantin Lopuhin
17e9914b8a Catch SyntaxError as well when importing manhole 
Also give a more detailed reason why telnet is not enabled (for the
future).
 2018-07-09 12:26:09 +03:00