BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-23 19:03:59 +00:00
Code Releases Wiki Activity
5,875 Commits 39 Branches 119 Tags
Commit Graph

5875 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Valdir Stumm Junior
7025d6656a document download_latency meta key 2016-11-14 13:06:18 -02:00
Mikhail Korobov
7e20725eb7 Merge pull request #2314 from redapple/tls-ciphers 
[MRG+1] Use OpenSSL default ciphers
 2016-10-17 11:44:28 +02:00
Paul Tremberth
dc1f9ad211 Merge pull request #2307 from eLRuLL/genspider-no-www-fix 
genspider: removing www. from starturl templates
 2016-10-12 15:32:18 +02:00
Raul Gallegos
118b42ab59 making start_urls a list in basic genspider template 2016-10-11 22:08:05 -05:00
Paul Tremberth
c3411373e8 Use OpenSSL default ciphers 
Twisted default TLS options restricts the ciphers list a bit -- "a secure default"
e38cc25a67/src/twisted/internet/_sslverify.py (L1861)

We want to be a bit more permissive with Scrapy
(at least as permissive as Scrapy 1.0 was, and which used a default OpenSSL Context)

See https://www.openssl.org/docs/manmaster/apps/ciphers.html#CIPHER_STRINGS

OpenSSL's 'DEFAULT' seems to be reasonable enough.

Fixes #2311
 2016-10-07 12:28:16 +02:00
Raul Gallegos
dd778892d0 genspider: removing www. from starturl templates 2016-10-05 12:16:45 -05:00
Paul Tremberth
3235bfeb1e Bump version: 1.2.0dev2 → 1.2.0 1.2.0 2016-10-03 13:04:11 +02:00
Paul Tremberth
95c6b9dffd Merge pull request #2280 from scrapy/release-notes-1.2 
Update release notes for upcoming 1.2.0
 2016-10-03 13:00:14 +02:00
Paul Tremberth
e2137d77ce Add release date for scrapy 1.2 2016-10-03 12:41:18 +02:00
Paul Tremberth
a433126301 Merge pull request #2294 from redapple/release-headers 
Change release section titles to have correct links in ToC
 2016-10-03 12:38:44 +02:00
Paul Tremberth
32c46453c3 Merge pull request #2300 from mineo/patch-3 
Remove duplicate colons from the feed export settings docs
 2016-10-03 12:34:28 +02:00
Wieland Hoffmann
e8edc6c2bb Remove duplicate colons from the feed export settings docs 2016-10-02 16:09:29 +02:00
Daniel Graña
5d38a8e2d9 Merge pull request #2298 from scrapy/update-1.2-relnotes-data-dir-util 
Update 1.2 release notes with data_path changes from #1581
 2016-09-30 15:23:55 -03:00
Daniel Graña
bca374d651 Merge pull request #1581 from scrapy/fix-util-function-to-work-outside-project-dir 
[MRG+1] Make data_path work when outside project (used by HttpCacheMiddleware and Deltafetch plugin)
 2016-09-30 15:23:34 -03:00
Elias Dorneles
2d932c173c test abs path outside project as well 2016-09-30 15:07:58 -03:00
Paul Tremberth
fed53c1e28 Merge pull request #2267 from scrapy/deprecate-ubuntu-packages 
[MRG+1] Deprecate official Ubuntu packages and update installation instructions
 2016-09-30 17:35:29 +02:00
Elias Dorneles
559b4edaec update release notes with changes from #1581 2016-09-30 11:42:50 -03:00
Paul Tremberth
ab3f27b502 Merge pull request #2296 from scrapy/architecture-overview-once-more-with-feeling 
[MRG+1] architecture docs: restore explanation loop to step 1 (see comment in 3ac3ac4)
 2016-09-30 16:30:37 +02:00
Elias Dorneles
8bafcf33a1 improve description of engine 2016-09-30 11:20:30 -03:00
Elias Dorneles
9c90d9515a update data_path dosctring 2016-09-29 19:11:17 -03:00
Elias Dorneles
e107212043 restore explanation loop to step 1 (see comment in 3ac3ac4) 2016-09-29 19:02:57 -03:00
Elias Dorneles
25bd3b3fea add .scrapy when outside spider too, add tests 2016-09-29 18:30:42 -03:00
Elias Dorneles
8e4947ef0d make utils.project.data_path return path unmodified when outside project 2016-09-29 10:17:35 -03:00
Paul Tremberth
fd0b6faecb Change release section titles to have correct links in ToC 2016-09-29 14:36:13 +02:00
Paul Tremberth
47ef202acd Merge pull request #2285 from redapple/relnotes-debian-jessie-baseline 
Mention new Debian Jessie baseline
 2016-09-29 14:25:53 +02:00
Paul Tremberth
c670e3cdf6 Move mention 2016-09-29 14:23:52 +02:00
Elias Dorneles
86eb865598 Merge pull request #2287 from pawelmhm/docs-2230 
[MRG+1][docs/item_pipeline] process_item returning Deferred docs
 2016-09-29 09:15:09 -03:00
Elias Dorneles
60559369b5 Merge pull request #2292 from stummjr/fix-example-section 
[MRG+1] Update examples section in documentation with quotes.toscrape spiders
 2016-09-29 09:11:45 -03:00
Paul Tremberth
7ba16b7360 Merge pull request #2293 from scrapy/architecture-overview-fix-reqs-vs-urls 
docs: update data flow description and image (fixes: #2278)
 2016-09-29 10:27:36 +02:00
Paul Tremberth
33d04684e5 Fix typo 2016-09-29 10:25:21 +02:00
Elias Dorneles
3ac3ac4d92 docs: update data flow description and image (fixes: #2278) 
This fixes the explanation to use Requests instead of URLs,
which is what actually happens, and is also consistent with the
new tutorial, which already explains how URLs become Request objects.

I've also changed the "loop", jumping from 9 to step 2.
 2016-09-28 16:38:45 -03:00
Valdir Stumm Junior
5680c61063 Doc: update examples section with quotes.toscrape.com spiders 2016-09-28 15:45:01 -03:00
Paul Tremberth
23abf2d31b Fix title underline 2016-09-28 18:31:27 +02:00
Paul Tremberth
8d130b295a Update installation guide 2016-09-28 17:09:17 +02:00
Paul Tremberth
c707c31345 Fix Debian number 2016-09-26 17:20:13 +02:00
pawelmhm
39dbd890b0 [docs/item_pipeline] process_item returning Deferred docs 
* quote url
* use hash of url as filename
 2016-09-24 08:36:09 +02:00
Paul Tremberth
bba31b545d Mention new Debian Jessie baseline 2016-09-23 17:33:08 +02:00
Paul Tremberth
7eeee6f616 Mention \uXXXX escapes in regards to FEED_EXPORT_ENCODING 2016-09-23 16:39:09 +02:00
Paul Tremberth
d52fbf2045 Merge remote-tracking branch 'origin/master' into release-notes-1.2 2016-09-23 16:34:06 +02:00
Paul Tremberth
d867268976 Merge pull request #2284 from redapple/release-notes-1.1.3-master 
Update release notes with 1.1.3 changes
 2016-09-23 16:25:52 +02:00
Paul Tremberth
80c1e5dc25 Set release date, fix typo and add tutorial improvement issue number 2016-09-23 16:11:50 +02:00
Paul Tremberth
a0f87d2f45 Update release notes for upcoming 1.1.3 bugfix release 2016-09-23 16:11:50 +02:00
Elias Dorneles
c2493c9452 Merge pull request #2282 from pawelmhm/docs-2230 
[docs] document that process_item can return Deferred
 2016-09-23 09:05:34 -03:00
Pawel Miech
b2bfd1e5c5 [docs] document that process_item can return Deferred 2016-09-23 10:36:03 +02:00
Paul Tremberth
a975a50558 Merge pull request #2252 from eliasdorneles/tutorial-upgrades 
[MRG+2] Tutorial: rewrite tutorial seeking to improve learning path
 2016-09-22 16:39:21 +02:00
Elias Dorneles
24bb91528a Merge pull request #2229 from ahlinc/fix_shell_completion 
[MRG+1] Fix completion in `scrapy shell` for new imports
 2016-09-22 11:15:30 -03:00
Elias Dorneles
f4a2208916 addressing review comments and other minor editing 2016-09-22 11:04:45 -03:00
Paul Tremberth
2e08a9b412 Merge pull request #2271 from redapple/mailsender-lists 
[MRG+1] Add note on "to" and "cc" as lists for sending emails
 2016-09-22 12:00:14 +02:00
Elias Dorneles
d636e5baa8 better description for start_requests expected return value 2016-09-21 18:54:12 -03:00
Elias Dorneles
32017a76f8 recommend learn python the hard way for beginners 2016-09-21 11:06:36 -03:00