BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-25 06:23:58 +00:00
Code Releases Wiki Activity
5,769 Commits 39 Branches 119 Tags
Commit Graph

5769 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Mikhail Korobov
a38a99e0e2 Merge pull request #1893 from redapple/sphinx-1.4 
Add support for Sphinx 1.4
 2016-03-30 19:55:48 +06:00
Paul Tremberth
1075587dbd Add support for Sphinx 1.4 
See http://www.sphinx-doc.org/en/stable/changes.html#release-1-4-released-mar-28-2016

sphinx_rtd_theme has become optional, needs to be added to reqs

https://github.com/sphinx-doc/sphinx/pull/2320 changes node entries tuples
to 5 values instead of 4

`sh` syntax highlighting added very locally in selectors.rst
because of this warning/error with Sphinx 1.4:

```
Warning, treated as error:
/home/paul/src/scrapy/docs/topics/selectors.rst:743:
WARNING: Could not lex literal_block as "python". Highlighting skipped.
```
 2016-03-30 14:40:52 +02:00
nanolab
a583e4d531 Update httpcache.py 
It checks cache directory modification time, but have to check file modification time.
 2016-03-30 10:57:48 +02:00
Lele
7082454f2a Changed sel. to response. for clarity 
Changed sel. to response. to comply with the rest of the examples in the same section, to avoid confusion.
 2016-03-28 05:27:15 +05:00
Konstantin Lopuhin
fc8cd45a48 Fix a race condition in the FilesPipeline 
Checksum calculation could happen simultaniously with
persisting the file in the store (which is done in a thread):
they operated on the same buf object.
Concretely this lead to a bug with S3FilesStore
when using botocore: the signature did not match because
the position in the buf was already at the end.
The fix is to move checksum calculation before passing buf
to the store.
 2016-03-27 21:56:47 +02:00
Konstantin Lopuhin
5045a4f168 Fix handling of meta=None in S3FilesStore.persist_file 2016-03-25 18:35:55 +03:00
Mikhail Korobov
4f335b5a01 DOC clarify Architecture docs 2016-03-25 17:03:41 +05:00
Mikhail Korobov
3ca977a8cb DOC improved Architecture overview 
* spiders don't have to work on specific domains;
* explain what to use Downloader middleware for
  and what to use Spider middleware for;
* Engine no longer locates spiders based on domains;
* "Spider middleware output direction" step was missing.

See also: GH-1569.
 2016-03-25 07:11:33 +05:00
pawelmhm
65c7c05060 response_status_message should not fail on non-standard HTTP codes 
utility is used in retry middleware and it was failing to handle non-standard HTTP codes.
Instead of raising exceptions when passing through to_native_str it should return
"Unknown status" message.
 2016-03-12 14:16:40 +01:00
Mikhail Korobov
ebef6d7c6d Merge pull request #1848 from aron-bordin/small_doc_style_fixes 
small doc style fixes
 2016-03-07 08:49:25 +05:00
Aron Bordin
2cfe9e424d small doc style fixes 2016-03-05 19:54:06 -03:00
Paul Tremberth
e122c569fe Merge pull request #1842 from nyov/nyov/docs 
[MRG+1] Update documentation links
 2016-03-04 11:50:26 +01:00
nyov
5876b9aa30 Update documentation links 2016-03-03 16:28:33 +00:00
Paul Tremberth
9f4fe5dc4a Merge pull request #1822 from nyov/nyov/scheduler 
[MRG+1] Allow core Scheduler priority queue customization
 2016-03-02 14:20:40 +01:00
Mikhail Korobov
6b2871dadd Merge pull request #1835 from djunzu/add_pps_to_IGNORED_EXTENSIONS 
[MRG+1] Add pps extension to IGNORED_EXTENSIONS
 2016-03-02 16:15:51 +05:00
djunzu
0e288d4a71 Add pps extension to IGNORED_EXTENSIONS 
modified:   scrapy/linkextractors/__init__.py
 2016-03-01 21:02:13 -03:00
nyov
2a6524ee3a Allow core Scheduler priority queue customization 2016-03-01 13:58:40 +00:00
Daniel Graña
b8fcb46e67 Merge pull request #1804 from redapple/enable-test-dwnld-timeout 
Re-enable HTTPS tests for download timeouts
 2016-03-01 10:44:24 -03:00
Daniel Graña
21da493109 Merge pull request #1828 from scrapy/py3-classifiers 
[MRG+1] declare Python 3 support in setup.py
 2016-03-01 10:34:36 -03:00
Daniel Graña
cf535fe840 Merge pull request #1827 from scrapy/proxy-auth-test 
[MRG+1] Extract a function to build CONNECT request; add tests for it.
 2016-03-01 10:34:24 -03:00
Mikhail Korobov
17d3bec699 declare Python 3 support in setup.py 2016-03-01 16:34:13 +05:00
Mikhail Korobov
94e28adfb7 Extract a function to build CONNECT request; add tests for it. See GH-1701 and GH-1808. 2016-03-01 16:29:12 +05:00
Mikhail Korobov
e8635cd03c Merge pull request #1826 from redapple/universal-wheels 
Build universal wheels
 2016-03-01 15:23:57 +05:00
Paul Tremberth
f3c7a509b9 Build universal wheels 2016-03-01 11:00:20 +01:00
Mikhail Korobov
16512a32ea Merge pull request #1825 from redapple/relnotes-s3policy 
[MRG+1] Update release notes about change of default S3 ACL policy to "private"
 2016-02-29 17:02:18 +05:00
Paul Tremberth
90c64a6d13 Update release notes about change of default S3 ACL policy to "private" 2016-02-29 12:27:25 +01:00
Paul Tremberth
74158611c8 Merge pull request #1818 from lagenar/master 
[MRG+1] Refactored SpiderLoader class constructor for easier subclassing
 2016-02-26 23:23:34 +01:00
Paul Tremberth
241ae9f22f Merge pull request #1820 from redapple/http-tls-settings 
[MRG+1] Document DOWNLOADER_* settings for HTTP/1.0 and TLS
 2016-02-26 23:20:33 +01:00
Paul Tremberth
709b4fa884 Update release notes about HTTPS downloader 2016-02-26 18:35:29 +01:00
Paul Tremberth
174f526709 Document DOWNLOADER_* settings for HTTP/1.0 and TLS 2016-02-26 18:23:57 +01:00
Lucas Moauro
1ce4c86c2b Refactored SpiderLoader class constructor for easier subclassing 2016-02-25 21:08:50 -03:00
Paul Tremberth
84dea19455 Update release notes 2016-02-25 00:59:15 +01:00
Paul Tremberth
c9e781357d Explicitly call Twisted transport stopProducing() on HTTP/1.0 timeouts 2016-02-24 23:04:31 +01:00
Paul Tremberth
ecddc093a4 Explicitly call Twisted transport stopProducing() on HTTP/1.1 timeouts 2016-02-24 23:04:31 +01:00
Mikhail Korobov
f7a48b0c6b Merge pull request #1794 from redapple/twisted-tls 
[MRG+1] Use best practices for TLS connections when using Twisted>=14.0
 2016-02-25 02:23:33 +05:00
Paul Tremberth
4e93501304 Merge pull request #1796 from lopuhin/s3-acl-private 
[MRG+1] Change default S3 ACL to "private" and allow customization via settings
 2016-02-24 22:18:43 +01:00
Paul Tremberth
0336c2504a Use context factory class name in warning message 2016-02-24 16:42:25 +01:00
Paul Tremberth
c29a1b98b3 Make warning message a bit nicer in logs 2016-02-24 16:01:18 +01:00
Konstantin Lopuhin
6137dd96d9 Fix documentation for S3_STORE_ACL (now settings.FILES_STORE_S3_ACL) settings: it has nothing to do with feed exporters. 2016-02-24 10:16:41 +03:00
Konstantin Lopuhin
164f300762 See #1778 - change default S3 ACL to "private" and allow customization via settings 2016-02-24 10:16:41 +03:00
Paul Tremberth
c9890d5f76 Add warning for context factories not accepting method param 2016-02-24 01:26:04 +01:00
Paul Tremberth
095495e99c Backward-compatibility for common Scrapy context factory patterns 2016-02-24 01:24:58 +01:00
Daniel Graña
513ba7a1fb Merge pull request #1800 from redapple/http11-post-content-length 
[MRG+1] Add "Content-Length: 0" for body-less HTTP/1.1 POST requests
 2016-02-23 15:00:33 -03:00
Elias Dorneles
329a77a3d0 Merge pull request #1808 from scrapy/revert-1701-cleanup-http11-tunneling 
Revert "Cleanup http11 tunneling connection after #1678"
 2016-02-23 13:59:15 -03:00
Elias Dorneles
10bcdb49b0 Merge pull request #1787 from scrapy/improve-errors 
[MRG+1] Better tracebacks
 2016-02-23 13:16:47 -03:00
Paul Tremberth
35fb630c4e Revert "[MRG+1] Cleanup http11 tunneling connection after #1678" 2016-02-23 16:28:48 +01:00
Mikhail Korobov
62a517163b Merge pull request #1801 from redapple/botocore-notconfigured-exception 
Fix SkipTest() message for botocore import test
 2016-02-21 12:33:48 +05:00
Paul Tremberth
ad4c1169c4 Pass exception directly to SkipTest() 
It prints the same as passing `str(e)`
 2016-02-21 01:16:46 +01:00
Paul Tremberth
e9bd328936 Fix SkipTest() message for botocore import test 2016-02-21 01:08:50 +01:00
Paul Tremberth
ac8f97c33b Fix typo in comment 2016-02-21 00:49:41 +01:00