BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-25 19:04:21 +00:00
Code Releases Wiki Activity
5,226 Commits 39 Branches 119 Tags
Commit Graph

5226 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Konstantin Lopuhin
a4ca1668d8 add https test for http10 handler (no luck with testing https with http11 so far) 2016-01-15 14:20:19 +03:00
Konstantin Lopuhin
96fcf4cea4 add a check that byte url is not accepted in http.Response on py3 2016-01-15 13:27:28 +03:00
Konstantin Lopuhin
99f1f2ad1d unskip tests and modules ported to py3 2016-01-14 19:00:48 +03:00
Konstantin Lopuhin
81a90c3af6 unskip part of test_download_gzip_response on py3, file a twisted issue for the remaining part 2016-01-14 18:47:06 +03:00
Konstantin Lopuhin
2aa6c92ffc py3 fixes in tests.mockserver 2016-01-14 17:52:50 +03:00
Konstantin Lopuhin
f46a9d595d skip ftp tests on py3 for now 2016-01-14 17:31:58 +03:00
Konstantin Lopuhin
4950f5988e py3: pass http proxy tests 2016-01-14 17:24:08 +03:00
Konstantin Lopuhin
c6f14a39de py3: fix http10 downloader - unicode host expected here 2016-01-14 16:50:16 +03:00
Konstantin Lopuhin
6b79fffa9a py3: pass all of HttpTestCase 2016-01-14 16:37:18 +03:00
Konstantin Lopuhin
3509378b8b py3: pass first http downloader test, simple crawler works now, yay! 2016-01-14 16:29:19 +03:00
Konstantin Lopuhin
32bb5b682a fix import of test_downloader_handlers.py: use @implementer, move failing on py3 imports into corresponding tests 2016-01-14 16:11:16 +03:00
Konstantin Lopuhin
5c2241ccc7 py3: fix webclient tests after making ScrapyHTTPClientFactory use bytes as in twisted 2016-01-14 15:30:28 +03:00
Konstantin Lopuhin
8df35bcac6 rm note to self: to be discussed in PR 2016-01-14 14:52:44 +03:00
Konstantin Lopuhin
ac2cf191d1 py3: remove comments, utf-8 is fine here: as twisted ultimately uses urllib.parse.quote that assepts bytes and assumes utf-8 2016-01-14 14:50:58 +03:00
Konstantin Lopuhin
e5fb609438 make ScrapyHTTPClientFactory comply to twisted HTTPClientFactory protocol - use bytes (encoding are likely wrong at this stage) 2016-01-14 14:45:02 +03:00
Konstantin Lopuhin
6a412d2503 all tests pass in test_webclient.py on py3 - removing from py3-ignores 2016-01-14 13:48:48 +03:00
Konstantin Lopuhin
0178356178 py3 fix testRedirect: url is bytes here 2016-01-14 13:46:46 +03:00
Konstantin Lopuhin
30c7b4e4cc py3 compat in test_timeoutTriggering cleanup 2016-01-14 13:44:14 +03:00
Konstantin Lopuhin
88f55312af py3 fix in testFactoryInfo - factory attirbutes are bytes in twisted 2016-01-14 13:43:37 +03:00
Konstantin Lopuhin
b5f9bc8499 py3 test fixes in test_webclient - expect bytes as page body 2016-01-14 13:04:45 +03:00
Konstantin Lopuhin
ae4aa2c3b2 py3 test fix: putChild expects bytes as path 2016-01-14 13:04:10 +03:00
Konstantin Lopuhin
325b6af6c2 fix ScrapyHTTPPageGetterTests for py3 - we expect bytes here 2016-01-14 12:29:59 +03:00
Konstantin Lopuhin
945674eb8f pass test_externalUnicodeInterference - the logic for py3 is clearly inverse of what was expected in this test, as scrapy Request url must be unicode 2016-01-14 12:25:54 +03:00
Konstantin Lopuhin
1d5ab67183 pass test_getPageHead on py3 2016-01-14 12:04:26 +03:00
Konstantin Lopuhin
73ff87c1dc decode body from utf-8, as scrapy stores body as bytes, and twisted has already converted to unicode 2016-01-14 12:03:08 +03:00
Konstantin Lopuhin
9f2be23a39 webclient tests, py3: fix setUp, pass test_getPage 2016-01-14 11:42:23 +03:00
Konstantin Lopuhin
f3889b0bce py3 compat: encode delimiter, method and path in ScrapyHTTPPageGetter 2016-01-14 11:41:49 +03:00
Elias Dorneles
57f99fc34e Merge pull request #1644 from yarikoptic/master 
[MRG+1] BF: robustify _monkeypatches check for twisted - str() name first (Closes #1634)
 2016-01-13 08:59:48 -02:00
Mikhail Korobov
f01fd07642 Merge pull request #1661 from redapple/py3-iterators 
[MRG+1] PY3: port utils/iterators
 2016-01-13 10:03:02 +05:00
Paul Tremberth
d7d4ef67a6 Changes following comments 2016-01-12 11:08:49 +01:00
Paul Tremberth
d4872940db PY3: port utils/iterators 2016-01-07 10:51:27 +01:00
Elias Dorneles
95e8ff8ba1 Merge pull request #1660 from stummjr/patch-4 
[MRG+1] Update deprecated examples
 2016-01-06 10:43:08 -02:00
Valdir Stumm Jr
2abc9bc901 Update deprecated examples 
* update the scrapy.org example to deal with the new layout.
* replaced slashdot.org by reddit.com, because it seems that slashdot is blocking requests.
 2016-01-06 10:29:45 -02:00
Daniel Graña
1b435b2887 Add 1.0.4 release notes 2015-12-30 15:43:54 -03:00
Yaroslav Halchenko
4f49aab7c0 BF: robustify _monkeypatches check for twisted - str() name first (Closes: #1634) 
In my case, while running datalad tests using nose, scrapy was failing
since v was None
 2015-12-18 16:16:05 -05:00
Mikhail Korobov
a5db7f862c Merge pull request #1639 from seales/master 
Spelling fixes
 2015-12-14 23:33:14 +05:00
Mikhail Korobov
1bc243a876 Merge pull request #1638 from orangain/toc-captions 
[MRG+1] DOC: Add captions to toctrees which appear in sidebar
 2015-12-14 23:32:18 +05:00
seales
719b1353a7 Spelling fixes 2015-12-13 19:39:48 -08:00
orangain
4be4ef038e DOC: Add captions to toctrees which appear in sidebar 2015-12-12 16:21:00 +09:00
Daniel Graña
b5285c3f38 Merge pull request #1628 from rolando/conda-docs 
[MRG] Removed pywin32 from conda install instructions as it's already declared as dependency.
 2015-12-05 14:45:17 -03:00
Rolando Espinoza
f8ae99d18f DOC Removed pywin32 from install instructions as it's already declared as dependency. 2015-12-05 09:48:17 -04:00
Elias Dorneles
7d187735ff Merge pull request #1359 from nyov/nyov/drop-optional_features 
[MRG+1] drop deprecated "optional_features" set
 2015-12-04 14:08:44 -02:00
Daniel Graña
17aba44f16 Merge pull request #1565 from starrify/fix-1564 
[MRG+1] fixed: Issue #1564 (Incorrectly picked URL in `scrapy.linkextractors.regex.RegexLinkExtractor` when there is a `<base>` tag. )
 2015-12-04 10:31:13 -03:00
Daniel Graña
dc65026395 Merge pull request #1597 from Digenis/various_formreq_fixes 
[MRG+1] Various FormRequest tests+fixes
 2015-12-04 09:55:12 -03:00
Daniel Graña
3881eaff45 Merge pull request #1575 from palego/startproject-templates-override 
[MRG+1] Startproject templates override
 2015-12-04 09:54:48 -03:00
Daniel Graña
a97f28fbf0 Merge pull request #1598 from Digenis/disable_dupefilter_in_shell 
[MRG+1] Disable dupefilter in shell to prevent hanging.
 2015-12-04 09:51:56 -03:00
Daniel Graña
21f638b139 Merge pull request #1622 from sibiryakov/twisted-1550-fix 
[MRG+1] Ignoring xlib/tx folder, depending on Twisted version.
 2015-12-04 09:40:35 -03:00
Mikhail Korobov
015cd78940 Merge pull request #1618 from sibiryakov/engine-heartbeats 
Nextcall heartbeats in engine (event loop halting)
 2015-12-04 16:28:45 +05:00
Alexander Sibiryakov
c4d29ecaef Ignoring xlib/tx folder, depending on Twisted version. 2015-12-02 17:08:21 +01:00
Alexander Sibiryakov
5750ce92c0 nextcall repetitive calls (heartbeats). 2015-12-02 11:39:47 +01:00