BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-27 06:43:46 +00:00
Code Releases Wiki Activity
5,487 Commits 39 Branches 119 Tags
Commit Graph

5487 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Konstantin Lopuhin
6b79fffa9a py3: pass all of HttpTestCase 2016-01-14 16:37:18 +03:00
Konstantin Lopuhin
3509378b8b py3: pass first http downloader test, simple crawler works now, yay! 2016-01-14 16:29:19 +03:00
Konstantin Lopuhin
32bb5b682a fix import of test_downloader_handlers.py: use @implementer, move failing on py3 imports into corresponding tests 2016-01-14 16:11:16 +03:00
Konstantin Lopuhin
5c2241ccc7 py3: fix webclient tests after making ScrapyHTTPClientFactory use bytes as in twisted 2016-01-14 15:30:28 +03:00
Konstantin Lopuhin
8df35bcac6 rm note to self: to be discussed in PR 2016-01-14 14:52:44 +03:00
Konstantin Lopuhin
ac2cf191d1 py3: remove comments, utf-8 is fine here: as twisted ultimately uses urllib.parse.quote that assepts bytes and assumes utf-8 2016-01-14 14:50:58 +03:00
Konstantin Lopuhin
e5fb609438 make ScrapyHTTPClientFactory comply to twisted HTTPClientFactory protocol - use bytes (encoding are likely wrong at this stage) 2016-01-14 14:45:02 +03:00
Konstantin Lopuhin
6a412d2503 all tests pass in test_webclient.py on py3 - removing from py3-ignores 2016-01-14 13:48:48 +03:00
Konstantin Lopuhin
0178356178 py3 fix testRedirect: url is bytes here 2016-01-14 13:46:46 +03:00
Konstantin Lopuhin
30c7b4e4cc py3 compat in test_timeoutTriggering cleanup 2016-01-14 13:44:14 +03:00
Konstantin Lopuhin
88f55312af py3 fix in testFactoryInfo - factory attirbutes are bytes in twisted 2016-01-14 13:43:37 +03:00
Konstantin Lopuhin
b5f9bc8499 py3 test fixes in test_webclient - expect bytes as page body 2016-01-14 13:04:45 +03:00
Konstantin Lopuhin
ae4aa2c3b2 py3 test fix: putChild expects bytes as path 2016-01-14 13:04:10 +03:00
Paul Tremberth
85b0e6c9c7 Travis: run tox with Python 3.5 + add Python 3.5 tests 2016-01-14 10:50:51 +01:00
Konstantin Lopuhin
325b6af6c2 fix ScrapyHTTPPageGetterTests for py3 - we expect bytes here 2016-01-14 12:29:59 +03:00
Konstantin Lopuhin
945674eb8f pass test_externalUnicodeInterference - the logic for py3 is clearly inverse of what was expected in this test, as scrapy Request url must be unicode 2016-01-14 12:25:54 +03:00
Konstantin Lopuhin
1d5ab67183 pass test_getPageHead on py3 2016-01-14 12:04:26 +03:00
Konstantin Lopuhin
73ff87c1dc decode body from utf-8, as scrapy stores body as bytes, and twisted has already converted to unicode 2016-01-14 12:03:08 +03:00
Konstantin Lopuhin
9f2be23a39 webclient tests, py3: fix setUp, pass test_getPage 2016-01-14 11:42:23 +03:00
Konstantin Lopuhin
f3889b0bce py3 compat: encode delimiter, method and path in ScrapyHTTPPageGetter 2016-01-14 11:41:49 +03:00
Paul Tremberth
a93d49a64c Add Python 3.5 tox env and Python 3.4 tests in Travis CI 2016-01-13 13:09:16 +01:00
Paul Tremberth
1347015a80 Refactored test code 2016-01-13 12:32:28 +01:00
Paul Tremberth
d4c7d72b2b Add tests for input type in xmliter calls 2016-01-13 12:13:47 +01:00
Elias Dorneles
57f99fc34e Merge pull request #1644 from yarikoptic/master 
[MRG+1] BF: robustify _monkeypatches check for twisted - str() name first (Closes #1634)
 2016-01-13 08:59:48 -02:00
Paul Tremberth
9fad25f3d1 Use explicit Unicode and bytes for XML body in tests 2016-01-13 11:42:41 +01:00
Paul Tremberth
2f2c2e8096 Merge remote-tracking branch 'origin/master' into xmliter-unicode 2016-01-13 11:42:01 +01:00
Mikhail Korobov
f01fd07642 Merge pull request #1661 from redapple/py3-iterators 
[MRG+1] PY3: port utils/iterators
 2016-01-13 10:03:02 +05:00
Paul Tremberth
d7d4ef67a6 Changes following comments 2016-01-12 11:08:49 +01:00
Paul Tremberth
6ddd814738 Support unicode tags in xml iterators (fixes #1665) 2016-01-12 10:48:45 +01:00
Paul Tremberth
d4872940db PY3: port utils/iterators 2016-01-07 10:51:27 +01:00
Elias Dorneles
95e8ff8ba1 Merge pull request #1660 from stummjr/patch-4 
[MRG+1] Update deprecated examples
 2016-01-06 10:43:08 -02:00
Valdir Stumm Jr
2abc9bc901 Update deprecated examples 
* update the scrapy.org example to deal with the new layout.
* replaced slashdot.org by reddit.com, because it seems that slashdot is blocking requests.
 2016-01-06 10:29:45 -02:00
palego
d03d262f52 indentation 2016-01-04 10:00:13 +01:00
palego
c702c53015 change os.mknod() for open() 
os.mknod() is a privileged command on OS X, making the test fail
 2016-01-03 14:33:42 +01:00
Daniel Graña
1b435b2887 Add 1.0.4 release notes 2015-12-30 15:43:54 -03:00
Aron Bordin
f57121c77b show download warnsize once 2015-12-30 13:10:13 -02:00
Yaroslav Halchenko
4f49aab7c0 BF: robustify _monkeypatches check for twisted - str() name first (Closes: #1634) 
In my case, while running datalad tests using nose, scrapy was failing
since v was None
 2015-12-18 16:16:05 -05:00
orangain
bcce8d3d80 DOC: Update MetaRefreshMiddlware's setting variables 
* `REDIRECT_MAX_METAREFRESH_DELAY` has been deprecated and was
  renamed to `METAREFRESH_MAXDELAY`.
* Merge duplicate documents about `METAREFRESH_MAXDELAY` appeared both
  in the settings page and the downloader-middlewares page.
 2015-12-17 15:25:02 +09:00
Mikhail Korobov
a5db7f862c Merge pull request #1639 from seales/master 
Spelling fixes
 2015-12-14 23:33:14 +05:00
Mikhail Korobov
1bc243a876 Merge pull request #1638 from orangain/toc-captions 
[MRG+1] DOC: Add captions to toctrees which appear in sidebar
 2015-12-14 23:32:18 +05:00
seales
719b1353a7 Spelling fixes 2015-12-13 19:39:48 -08:00
orangain
4be4ef038e DOC: Add captions to toctrees which appear in sidebar 2015-12-12 16:21:00 +09:00
Daniel Graña
b5285c3f38 Merge pull request #1628 from rolando/conda-docs 
[MRG] Removed pywin32 from conda install instructions as it's already declared as dependency.
 2015-12-05 14:45:17 -03:00
Rolando Espinoza
f8ae99d18f DOC Removed pywin32 from install instructions as it's already declared as dependency. 2015-12-05 09:48:17 -04:00
Elias Dorneles
7d187735ff Merge pull request #1359 from nyov/nyov/drop-optional_features 
[MRG+1] drop deprecated "optional_features" set
 2015-12-04 14:08:44 -02:00
Daniel Graña
17aba44f16 Merge pull request #1565 from starrify/fix-1564 
[MRG+1] fixed: Issue #1564 (Incorrectly picked URL in `scrapy.linkextractors.regex.RegexLinkExtractor` when there is a `<base>` tag. )
 2015-12-04 10:31:13 -03:00
Daniel Graña
dc65026395 Merge pull request #1597 from Digenis/various_formreq_fixes 
[MRG+1] Various FormRequest tests+fixes
 2015-12-04 09:55:12 -03:00
Daniel Graña
3881eaff45 Merge pull request #1575 from palego/startproject-templates-override 
[MRG+1] Startproject templates override
 2015-12-04 09:54:48 -03:00
Daniel Graña
a97f28fbf0 Merge pull request #1598 from Digenis/disable_dupefilter_in_shell 
[MRG+1] Disable dupefilter in shell to prevent hanging.
 2015-12-04 09:51:56 -03:00
Daniel Graña
21f638b139 Merge pull request #1622 from sibiryakov/twisted-1550-fix 
[MRG+1] Ignoring xlib/tx folder, depending on Twisted version.
 2015-12-04 09:40:35 -03:00