BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-24 11:43:41 +00:00
Code Releases Wiki Activity
6,637 Commits 39 Branches 119 Tags
Commit Graph

6637 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Mikhail Korobov
3cf871c61f
Merge pull request #3281 from fbergen/gunzipperf 
[MRG+2] Improve gunzip performance for big files on Python 3
 2018-06-02 04:13:26 +05:00
Mikhail Korobov
847b50ce9c
Merge pull request #3201 from grammy-jiang/master 
[MRG+1] fix a mistake in topic spider-middleware.rst
 2018-06-02 01:09:59 +05:00
Fredrik Bergenlid
6a2d2c3b77 Improve gunzip performance for big files 2018-06-01 21:38:07 +02:00
Mikhail Korobov
13b15dc9a8
Merge pull request #3279 from lewoudar/patch-1 
Update spiders.rst
 2018-06-01 22:20:41 +05:00
Mikhail Korobov
f1d87ee09a
Merge pull request #3280 from cms-/patch-1 
Minor edits to contributing.rst
 2018-06-01 20:54:39 +05:00
Chris Slothouber
ecdd888ff4
Minor edits to contributing.rst 
Corrected minor grammatical issues and increased clarity of instructions.
 2018-06-01 09:25:34 -04:00
Kevin Tewouda
ffa7bede17
Update spiders.rst 
I changed URLs to :class:`~scrapy.http.Request` in start_urls explanation of the default spider
 2018-05-30 06:33:18 +02:00
Vostretsov Nikita
b364d27247 [MRG+1] Automatic port selection for servicies in unit tests (#3210) 
* ability to pass port as a parameter

* try to find free ports

* use environment variables to pass mock server address

* get mock server address from environment variables

* ability to select ports for proxy in runtime

* use common method for URLs from mock server

* https support

* get mock server address

* get mock address

* replace hand-written mechanism by kernel-based one

* use ephemeral ports in mockserver

* strip EOL from addresses

* use ephemeral port in proxy

* no need to restore environment as it is restored in tearDown

* decode bytes

* use mockserver address as a variable

* ability to pass address as variable

* per test-case mockserver

* use base class

* remove obsolete environment manipulation

* return usage of proxy for http cases

* common method for broking proxy auth credentials

* python version-independent url methods
 2018-05-23 21:25:50 +03:00
Konstantin Lopuhin
bac1e2d47d
Merge pull request #3263 from whalebot-helmsman/no_twisted_18_4_0 
[MRG+1] Blacklist twisted version with regression
 2018-05-17 13:21:42 +03:00
Vostretsov Nikita
c5ddfddb7e blacklist twisted version with regression in constraints file 2018-05-17 08:53:42 +00:00
Vostretsov Nikita
0d015e5c0f blacklist twisted version with regression 2018-05-16 09:36:07 +00:00
Konstantin Lopuhin
c4f096d3a5
Merge pull request #3224 from lucywang000/better-processors-doc 
[MRG+1] Improve document about functions as processors
 2018-04-26 09:36:39 +03:00
Konstantin Lopuhin
f36e1b53d4
Merge pull request #3199 from rhoboro/gcs_acl 
[MRG+1] FilesPipeline supports ACL for Google Cloud Storage
 2018-04-25 19:42:19 +03:00
Lucy Wang
57b0e6b695 improve document about functions as processors 2018-04-19 13:35:46 +08:00
rhoboro
6ef6585b5a update docs 2018-04-13 19:06:29 +09:00
rhoboro
560ee623fd set defalut value "" to FILES_STORE_GCS_ACL 2018-04-13 19:00:27 +09:00
rhoboro
464973489e Using bucket's default object ACL 2018-04-13 12:06:39 +09:00
Konstantin Lopuhin
da1256a9c8
Merge pull request #3189 from stav/master 
[MRG+1] Doc: update wording for COOKIES_ENABLED
 2018-04-05 11:42:49 +03:00
grammy-jiang
cb76b88331 fix a mistake in topic spider-middleware.rst 2018-04-04 05:56:05 -04:00
rhoboro
74a9c65290 update docs for support gcs acl 2018-04-03 18:20:37 +09:00
rhoboro
5254ac393b added test for gcs policy 2018-04-03 18:06:34 +09:00
rhoboro
8e8994c6b5 add acl support for gcs 2018-04-02 15:36:47 +09:00
Steven Almeroth
c6d20bdd82 Doc: update wording for COOKIES_ENABLED 2018-03-27 16:21:07 -04:00
Daniel Graña
6c3970e672
Merge pull request #3153 from virmht/new_bug 
[MRG+1] Fixed bug FormRequest.from_response() clickdata ignores input[type=image]
 2018-03-21 16:32:12 -03:00
Viral Mehta
dd064413a4 corrected syntax error in XPath 2018-03-19 19:28:41 +05:30
Viral Mehta
a5acc9373f Resolving Comments 2018-03-19 18:19:39 +05:30
Viral Mehta
e25e2afe17 Removed unnecessary print statements 2018-03-17 18:20:14 +05:30
Viral Mehta
ff5f717f7a Fixed formatting issues 2018-03-17 18:17:48 +05:30
Daniel Graña
6cc6bbb5fc
Merge pull request #3166 from lucywang000/catch-tls-certificate-error 
catch CertificateError in tls verification
 2018-03-14 11:47:58 -03:00
Lucy Wang
2c58da19a6 update docstring of ScrapyClientTLSOptions 2018-03-14 09:27:59 +08:00
Lucy Wang
1a2f0193a3 fix tests on jessie 2018-03-13 19:14:52 +08:00
siulkilulki
6a7cdf9a6c [MRG+1] Add 'flv' to ignored video extensions. (#3165) 2018-03-13 10:35:27 +03:00
Lucy Wang
d9e6c73fb3 revert wrong changes 2018-03-13 13:05:37 +08:00
Lucy Wang
e487100987 add a test case 2018-03-13 08:59:03 +08:00
Lucy Wang
13a74d77e2 catch CertificateError in tls verification 2018-03-12 22:25:19 +08:00
Viral Mehta
65744c2199 Corrected Test 2018-03-03 20:07:50 +05:30
Viral Mehta
d5b7ebcfdc Fixed bug FormRequest.from_response() clickdata ignores input[type=image] 2018-03-03 18:17:49 +05:30
NewUserHa
acd2b8d43b [MRG+1] Fix part of issue #3128 - None should not be a valid type for 'url' in Response.follow (#3131) 
* fix one issue of issue#3128

because @kmike posted: 'If url is '', Scrapy should follow the same page, this is an intended behavior.'

*  fix one issue of issue#3128

because @kmike posted: 'If url is '', Scrapy should follow the same page, this is an intended behavior.'
 2018-02-22 03:37:26 +05:00
Cathal Garvey
426da0ed07
Merge pull request #3127 from Anjalizi/patch-2 
Updated contributing.rst
 2018-02-16 18:09:48 +00:00
Anjali Jain
bbc2a3569f further edited 2018-02-16 23:33:10 +05:30
Anjali Jain
6954da1366
Updated contributing.rst 
Rectified grammatical errors
 2018-02-15 23:27:40 +05:30
Konstantin Lopuhin
f8c688e6f4
Merge pull request #3123 from scrapy/fix-docs-build 
fix docs building on CI
 2018-02-14 18:50:26 +03:00
Mikhail Korobov
dc0304fde1 fix docs building with recent sphinx: don't use deprecated sphinx options and imports 2018-02-13 19:47:41 +05:00
Daniel Graña
c56f7b3c8d
Merge pull request #3113 from WenbinZhang/master 
[MRG+1] Update robotstxt.py
 2018-02-08 18:22:56 -03:00
Daniel Graña
68e45d32e0
Merge pull request #3115 from scrapy/telnet-log-level 
[MRG+1] use INFO log level to show telnet host/port
 2018-02-08 18:22:36 -03:00
Konstantin Lopuhin
936dbc7bf6
Merge branch 'master' into master 2018-02-08 23:43:29 +03:00
Konstantin Lopuhin
8d2240d066
Merge pull request #3082 from elacuesta/pickle-requests 
[MRG+1] Do not serialize unpickable objects (py3)
 2018-02-08 23:39:37 +03:00
Eugenio Lacuesta
6edd4114c4 Clarify comment about Pyhton versions 2018-02-08 15:47:20 -03:00
Eugenio Lacuesta
0d87e77afe Bump parsel dependency 2018-02-08 15:03:57 -03:00
Eugenio Lacuesta
e4558cb27e Update test for unpickable objects 2018-02-08 15:03:57 -03:00