djunzu
dcb279bd6c
Add m4v extension to IGNORED_EXTENSIONS in LinkExtractor.
...
modified: scrapy/linkextractors/__init__.py
2017-09-17 16:09:22 -03:00
Renze Yu
088b80d41a
minor fix typo
2017-09-13 23:29:22 +08:00
rhoboro
ee166ec44f
Support for ImagesPipeline
2017-09-13 17:35:46 +09:00
rhoboro
e5d4364b2a
Add tests for GCS Storage
2017-09-13 16:24:16 +09:00
rhoboro
d71a063403
Support for Google Cloud Storage
2017-09-13 16:00:26 +09:00
Steven Almeroth
3637b75a67
[Doc] Update Response.body type
2017-09-12 17:41:47 -04:00
Iulian Onofrei
aab98080a0
Add option to disable automatic log handler install
2017-09-11 00:40:55 +03:00
Paul Tremberth
abaf466bb3
Print cryptography package version
2017-09-07 11:43:20 +02:00
cclauss
b8fabeed86
ur'string' not needed in Py 2, syntax error in Py3
...
This instance was missed in #2909 --> ur'Scrapy developers' --> u'Scrapy developers'
2017-09-04 14:29:23 +02:00
Mikhail Korobov
6213fa5175
Merge pull request #2909 from cclauss/patch-2
...
ur'string' not needed in Py 2, syntax error in Py3
2017-09-01 15:42:02 +05:00
cclauss
b702236082
ur'string' not needed in Py 2, syntax error in Py3
...
Convert `u'(.*)\:\d+\:\s\[(.*)\]\s(?:(.*)\sto\s(.*)|(.*))'`--> `u'(.*)\:\d+\:\s\[(.*)\]\s(?:(.*)\sto\s(.*)|(.*))'`to be compatible with both Python 2 and Python 3. See #2891
2017-09-01 11:56:09 +02:00
cclauss
9f16f040b6
ur'string' not needed in Py 2, syntax error in Py3
...
Convert `ur'Scrapy Documentation'`--> `u'Scrapy Documentation'`to be compatible with both Python 2 and Python 3. See #2891
2017-09-01 11:53:59 +02:00
Mikhail Korobov
65ac0b06df
Merge pull request #2894 from redapple/log-custom-overriden-settings
...
Move logging of overriden settings to Crawler init
2017-08-29 19:20:46 +05:00
Pablo Hoffman
a429d78019
update scrapinghub.com urls to use https
2017-08-24 16:03:36 -03:00
Paul Tremberth
7a35a1ad4a
Remove trailing bracket from components versions log
2017-08-23 17:54:14 +02:00
Paul Tremberth
1968a8ec02
Move logging of overriden settings to Crawler init
2017-08-23 15:08:10 +02:00
Mikhail Korobov
1ba77f0837
Merge pull request #2869 from cclauss/patch-3
...
# noqa to close #2836
2017-08-21 19:15:30 +05:00
Mikhail Korobov
885289f497
Merge pull request #2854 from jenya/2853-sitemap-follow-alternate-fix
...
[MRG+1] Follow alternate link for all types of sitemaps #2853
2017-08-21 19:15:03 +05:00
Mikhail Korobov
984c0c19ee
Merge pull request #2884 from iamminji/patch-1
...
fix typo
2017-08-16 16:36:00 +05:00
kim minji
1dcea6a9d4
fix typo
2017-08-16 18:07:52 +09:00
Daniel Graña
d5f3543d77
Merge pull request #2865 from kirankoduru/2831-explicit-msg-for-scrapy-parse-callback
...
[MRG+1] Explicit message for scrapy parse callback
2017-08-09 07:56:57 -03:00
Chomba Ng'ang'a
fd27cde24d
Update asserts to use more generic ones
2017-08-09 12:52:10 +02:00
Chomba Ng'ang'a
4ca61a2051
Update deprecated test aliases
...
- change ``failIf`` to ``assertFalse``
- change ``asertEquals`` to ``assertEqual``
- change ``assert_`` to ``assertTrue``
https://docs.python.org/2/library/unittest.html#deprecated-aliases
2017-08-09 12:52:10 +02:00
Kiran Koduru
2960c9b568
Use self.__class__.__name__ instead of showing generic Spider class name
2017-08-05 16:29:41 -04:00
Kiran Koduru
12409a0cf6
Fix broken encoding on text for py 3
2017-08-05 15:40:38 -04:00
kirankoduru
7adab61a7a
Added test for NotImplemented Spider.parse method
2017-08-05 15:40:38 -04:00
kirankoduru
be71f98e92
Explicit message for scrapy parse callback
...
The scrapy parse method raises a NotImplementedError when not defined,
but for new comers it can be hard to debug what might be going wrong.
Adding an explicit message for NotImplementedError will help new users.
2017-08-05 15:40:38 -04:00
Paul Tremberth
a68a8f8fdc
Merge remote-tracking branch 'origin/1.4'
2017-08-05 11:45:50 +02:00
cclauss
0a69a32b5a
Force Travis CI to test again
2017-08-04 14:35:43 +02:00
cclauss
c016a4309d
# noqa to close #2836
...
Marks #2836 as will not fix.
2017-08-04 01:44:23 +02:00
Paul Tremberth
0cb3085f84
Add test for alternate links
2017-08-03 16:55:09 +02:00
Eugene Vorobev
01ac883893
Follow alternate link for all types of sitemaps
2017-08-03 16:55:09 +02:00
david watson
71d5b7d75a
fix typo ( #2867 )
2017-08-01 14:49:22 -03:00
Andrei Petre
6e6b5cc29f
Use getfullargspec under the scenes for py3 to stop DeprecationWarning ( #2864 )
...
Use getfullargspec under the scenes for py3 to stop DeprecationWarning.
Closes #2862
2017-08-01 11:14:43 -03:00
Daniel Graña
19382c5675
Merge pull request #2755 from redapple/downloader-mdw-template
...
[MRG+1] Add template for a downloader middleware
2017-07-31 11:37:26 -03:00
simik-ru
a65fec050a
Small fix in description of startproject arguments
2017-07-31 13:03:49 +02:00
Paul Tremberth
aaaa4da7a4
Add template for a downloader middleware
2017-07-28 13:48:23 +02:00
Mikhail Korobov
9d34b2fe16
Merge pull request #2857 from redapple/scrapy-components-logs-startup
...
Log versions information at startup
2017-07-28 00:02:06 +05:00
Paul Tremberth
bf7ef3e4c3
Move methods to a new scrapy.utils.versions
2017-07-27 20:08:30 +02:00
Paul Tremberth
219c8aa0b6
Log versions information at startup
2017-07-27 17:30:30 +02:00
Daniel Graña
5d9bac789d
Merge pull request #2849 from cclauss/patch-2
...
[MRG+1] xrange() --> range() for Python 3
2017-07-26 15:59:37 -03:00
Daniel Graña
bbc56e6960
Merge pull request #2852 from starrify/new-http-codes-to-retry-2851
...
Added: HTTP status code 522/524 to retry.
2017-07-26 15:46:35 -03:00
Daniel Graña
881a5e3a57
Merge pull request #2847 from redapple/redirect-308
...
Handle HTTP 308 Permanent Redirect
2017-07-26 15:29:23 -03:00
Paul Tremberth
15a5c533fa
Add tests for HTTP 307 permanent redirects
2017-07-26 19:07:57 +02:00
Paul Tremberth
1fdc10684f
HTTP Cache: treat 308 as 301
2017-07-26 19:01:48 +02:00
Paul Tremberth
5dc9a88c34
Handle HTTP 308 Permanent Redirect
2017-07-26 19:01:48 +02:00
Pengyu Chen
11a1f970b7
Added: HTTP status code 522/524 to retry.
2017-07-26 16:11:13 +08:00
cclauss
33dfac5018
xrange() --> range() for Python 3
...
Either this PR or #2845 .
2017-07-24 22:06:17 +02:00
Mikhail Korobov
17bbd71433
Merge pull request #2812 from elacuesta/inspect_response_populate_spider
...
Populate spider variable when using shell.inspect_response
2017-07-25 00:28:14 +05:00
Daniel Graña
b6d036ede9
Merge pull request #2837 from dguo/patch-2
...
Fix a typo in the Items documentation
2017-07-24 15:55:30 -03:00
