Eugenio Lacuesta
a56540877c
Do not serialize unpickable objects (py3)
2018-02-08 15:03:57 -03:00
Konstantin Lopuhin
eff469245a
Merge pull request #3100 from scrapy/robots-stats
...
[MRG+1] more stats for RobotsTxtMiddleware
2018-02-08 12:01:36 +03:00
Mikhail Korobov
0c374c00fb
use INFO log level to show telnet host/port
2018-02-08 05:09:02 +05:00
Wenbin Zhang
4d5e5378bd
Update robotstxt.py
...
Add message to IgnoreRequest exception so that it can be detectedin the errbak method of a spider
2018-02-07 10:59:32 -05:00
Konstantin Lopuhin
fd02fa808c
Merge pull request #3096 from jesuslosada/fix-signal-names
...
[MRG+1] Fix OS signal names
2018-02-06 16:21:59 +03:00
Mikhail Korobov
6f264ab190
more stats for RobotsTxtMiddleware
2018-01-30 05:47:28 +05:00
Jesús Losada
c1916626c1
Fix OS signal names
2018-01-27 21:24:15 +00:00
Daniel Graña
8be28fe4ca
Merge pull request #3092 from scrapy/fix-tests-for-w3lib-1.19
...
Fix tests to account for changes in w3lib 1.19
2018-01-26 13:52:21 -03:00
Mikhail Korobov
ba15b63ed6
TST fix tests to account for changes in w3lib 1.19
2018-01-26 02:11:49 +05:00
Yash Sharma
1d1581266c
Changed some documentations ( #3089 )
...
DOC typo fix in defer_fail docstring
2018-01-26 01:12:17 +05:00
Mikhail Korobov
7c9e32213d
Merge pull request #3059 from jesuslosada/fix-typo
...
Fix typo in comment
2018-01-11 01:34:11 +05:00
Mikhail Korobov
786144e0c7
Merge pull request #3058 from jesuslosada/fix-link
...
Fix link in news.rst
2018-01-11 01:33:38 +05:00
Daniel Graña
b170e9fb96
Merge pull request #2609 from otobrglez/extending-s3-files-store
...
S3FilesStore support for other S3 providers (botocore options)
2018-01-07 19:16:11 -05:00
Jesús Losada
61c0b14782
Fix typo in comment
2018-01-01 16:03:55 +00:00
Jesús Losada
a0836b8fd9
Fix link in news.rst
2018-01-01 15:59:38 +00:00
Daniel Graña
2dee191374
Merge branch 'master' into extending-s3-files-store
2017-12-31 16:44:38 -03:00
Mikhail Korobov
aa83e159c9
Bump version: 1.4.0 → 1.5.0
2017-12-30 02:09:52 +05:00
Mikhail Korobov
d07fe11981
set release date
2017-12-30 02:09:41 +05:00
Daniel Graña
9b4d6a40a6
Merge pull request #3053 from scrapy/release-notes-1.5
...
Release notes for the upcoming 1.5.0 version
2017-12-29 11:57:04 -03:00
Mikhail Korobov
c107059ef8
DOC fix rst syntax
2017-12-29 07:07:43 +05:00
Mikhail Korobov
d4e5671d07
make release docs more readable, add highlights
2017-12-29 07:06:00 +05:00
Mikhail Korobov
45b0e1a0e4
DOC draft 1.5 release notes
2017-12-28 07:33:43 +05:00
Paul Tremberth
461f9daff5
Update release notes for upcoming 1.4.1 version
2017-12-28 05:50:18 +05:00
Raphael
a1cc5a63d3
Add mention to dont_merge_cookies in CookiesMiddlewares docs ( #2999 ) ( #3030 )
...
Add mention to dont_merge_cookies in CookiesMiddlewares docs (#2999 )
2017-12-27 17:54:17 -03:00
Daniel Graña
57d04aa960
Merge pull request #2767 from redapple/http-proxy-endpoint-key
...
[MRG+1] Use HTTP pool and proper endpoint key for ProxyAgent
2017-12-26 14:12:47 -03:00
Mikhail Korobov
930f6ed800
Merge pull request #3050 from lopuhin/pypy3
...
Add PyPy3 support
2017-12-25 18:50:58 +05:00
Konstantin Lopuhin
bb1f311891
Add PyPy3 support to faq and install doc
2017-12-25 15:46:05 +03:00
Konstantin Lopuhin
041308afe7
Fix get_func_args test for pypy3
...
These built-in functions are exposed as methods in PyPy3.
For scrapy this does not matter as:
1) they do not work for CPython at all
2) get_func_args is checked for presense of an argument in scrapy,
extra "self" does not matter.
But it still makes sense to leave these tests so that we know we
shouldn't use get_func_args for built-in functions/methods.
2017-12-25 14:48:10 +03:00
Konstantin Lopuhin
f71df6f9ad
Run tests for PyPy3
2017-12-25 14:48:06 +03:00
Mikhail Korobov
632f1cc073
Merge pull request #3049 from scrapy/trove-classifiers
...
[MRG+1] setup.py: mention that we support PyPy. See GH-2213.
2017-12-25 15:59:56 +05:00
Mikhail Korobov
9f9edeadfc
Merge pull request #3048 from lopuhin/pypy-install-docs
...
[MRG+1] Mention PyPy support, add PyPy to install docs
2017-12-25 15:32:50 +05:00
Mikhail Korobov
1058169f0e
setup.py: mention that we support PyPy. See GH-2213.
2017-12-25 15:31:07 +05:00
Konstantin Lopuhin
ea41114cf0
Mention PyPy support, add PyPy to install docs
2017-12-25 12:35:04 +03:00
Mikhail Korobov
bdc12f3994
Merge pull request #3045 from hugovk/rm-3.3
...
[MRG+1] Drop support for EOL Python 3.3
2017-12-20 21:08:13 +05:00
Hugo
cbcf80b98f
Fix typo
...
[CI skip]
2017-12-20 17:34:13 +02:00
Hugo
f11c21c6fc
Test on Python 3.4
2017-12-20 17:05:56 +02:00
Hugo
44623687ab
Drop support for EOL Python 3.3
2017-12-19 17:59:05 +02:00
Daniel Graña
86c322c3a8
Merge pull request #3038 from scrapy/update-contributing-docs
...
DOC update "Contributing" docs
2017-12-13 13:17:28 -03:00
Mikhail Korobov
9aa9dd8d45
DOC mention an easier way to track pull requests locally.
...
Thanks @eliasdorneles!
2017-12-12 19:17:00 +05:00
Mikhail Korobov
f716843a66
DOC update "Contributing" docs:
...
* suggest Stack Overflow for Scrapy usage questions;
* encourage users to submit test-only pull requests with reproducable examples;
* encourage users to pick up stalled pull requests;
* we don't use AUTHORS file as a main acknowledgement source;
* suggest using Sphinx autodocs extension
2017-12-12 16:19:43 +05:00
Konstantin Lopuhin
a21b800419
Merge pull request #3011 from Jane222/master
...
[MRG+1] Issues a warning when user puts a URL into allowed_domains (#2250 )
2017-12-08 15:45:16 +03:00
Jana Cavojska
22c68baf99
url_pattern is now being compiled before entering the loop
2017-12-07 18:38:29 +01:00
Daniel Graña
3cf0332ec3
Merge pull request #2957 from ScrapingLab/add_meta_json_to_parse_command
...
[MRG+1] Scrapy Command: add --meta/-m to the "parse" command to pass additional meta data into the request
2017-11-29 16:26:48 -03:00
Jana Cavojska
454d5e5733
checking for subclass of URLWarning instead of checking error message text when URL in allowed_domains
2017-11-26 20:07:04 +01:00
Jana Cavojska
8ec3b476b0
triggering a warning when user puts URL in allowed_domains now covered by test
2017-11-26 16:36:15 +01:00
IAlwaysBeCoding
6af323d7c8
Fix spelling mistake on scrapy parse command docs
...
Fixed spelling mistake from "will be pass" to "will be passed"
2017-11-26 00:24:52 +01:00
Konstantin Lopuhin
97dc527191
Merge pull request #3020 from Jesse-Bakker/add-fromcrawler-middleware-docs
...
[MRG+1] Added from_crawler to middleware docs
2017-11-24 17:56:57 +03:00
Mikhail Korobov
bd16951a05
Merge pull request #3013 from KosayJabre/patch-1
...
Separated import statements
2017-11-24 16:49:56 +05:00
Jesse Bakker
0b14cb44aa
Added from_crawler to middleware docs
2017-11-23 15:25:43 +01:00
Jana Cavojska
91ff194d1e
looping over allowed_domains directly instead of via index
2017-11-20 21:23:31 +01:00
