BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-24 06:24:00 +00:00
Code Releases Wiki Activity
5,828 Commits 39 Branches 119 Tags
Commit Graph

5828 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Paul Tremberth
a975a50558 Merge pull request #2252 from eliasdorneles/tutorial-upgrades 
[MRG+2] Tutorial: rewrite tutorial seeking to improve learning path
 2016-09-22 16:39:21 +02:00
Elias Dorneles
24bb91528a Merge pull request #2229 from ahlinc/fix_shell_completion 
[MRG+1] Fix completion in `scrapy shell` for new imports
 2016-09-22 11:15:30 -03:00
Elias Dorneles
f4a2208916 addressing review comments and other minor editing 2016-09-22 11:04:45 -03:00
Paul Tremberth
2e08a9b412 Merge pull request #2271 from redapple/mailsender-lists 
[MRG+1] Add note on "to" and "cc" as lists for sending emails
 2016-09-22 12:00:14 +02:00
Elias Dorneles
d636e5baa8 better description for start_requests expected return value 2016-09-21 18:54:12 -03:00
Elias Dorneles
32017a76f8 recommend learn python the hard way for beginners 2016-09-21 11:06:36 -03:00
Elias Dorneles
38266cc949 recommend Dive into Python and Python tutorial instead of LPTHW for non-beginners 2016-09-21 11:02:24 -03:00
Elias Dorneles
c126c59361 address more review comments 2016-09-20 18:19:25 -03:00
Elias Dorneles
a876ea5bd2 minor grammar fix 2016-09-20 15:10:49 -03:00
Elias Dorneles
bc41fdf20e address review comments, add debug log to initial spider 2016-09-20 15:04:08 -03:00
Elias Dorneles
a19af5b164 Merge pull request #2273 from redapple/version-stability 
[MRG+1] Remove mention of odd-numbered versions for development releases
 2016-09-20 14:15:52 -03:00
Paul Tremberth
40293551b2 Remove mention of odd-numbered versions for development releases 
Fixes GH-1317
 2016-09-20 18:15:45 +02:00
Elias Dorneles
125b691102 more reviewing and editing, minor restructure, syntax fixes 2016-09-20 12:47:03 -03:00
Paul Tremberth
e59d79bf37 Add note on "to" and "cc" as lists for sending emails 
Fixes GH-2244
 2016-09-20 17:18:49 +02:00
Elias Dorneles
8975371a57 Merge branch 'master' into tutorial-upgrades 2016-09-20 09:45:05 -03:00
Elias Dorneles
f4f93c5c26 fix tox docs build, adjust title 2016-09-20 09:19:59 -03:00
Elias Dorneles
3fd947b30d Merge pull request #2269 from redapple/unserializable-warning 
Log warning when request cannot be serialized (instead of error)
 2016-09-20 09:00:58 -03:00
Paul Tremberth
a135dbaf19 Log warning when request cannot be serialized (instead of error) 
Fixes GH-2035
 2016-09-20 12:47:33 +02:00
Valdir Stumm Junior
fee07835f2 Completing the data extraction section 2016-09-19 19:19:44 -03:00
Elias Dorneles
2a409d1d95 [wip] changing introduction to scraping with selectors 2016-09-19 17:13:04 -03:00
Daniel Graña
eb49b459c1 Merge pull request #2212 from redapple/debian-jessie-baseline 
Add Debian Jessie test env
 2016-09-19 15:17:45 -03:00
Paul Tremberth
41cd9f401f Merge pull request #2243 from pawelmhm/image-pipeline-2198 
[MRG+1] [image & file pipeline] loading setting for user classes
 2016-09-19 18:43:52 +02:00
Elias Dorneles
063315258e Merge pull request #2202 from scrapy/doc-arch-overview2 
[MRG+1] DOC move Data Flow below the picture; add links to components
 2016-09-19 08:11:18 -03:00
Mikhail Korobov
490f6e08f3 Merge pull request #2239 from redapple/streamlogger-flush 
[MRG+1] Add flush() method to StreamLogger
 2016-09-19 14:44:45 +06:00
Mikhail Korobov
5657f6b8ef Merge pull request #2258 from redapple/feed-export-started 
[MRG+1] Feed exporter: start exporting only on first item
 2016-09-19 14:40:30 +06:00
Mikhail Korobov
552368727a Merge pull request #2225 from Tethik/parse_command_rules_fix 
[MRG+1] Two small fixes for when using the parse command and the '-r' flag (rules).
 2016-09-19 14:39:09 +06:00
Joakim Uddholm
8c38dde4e8 Moved parse command tests to its own file. Added some checks to check for logged errors. 2016-09-19 05:33:05 +02:00
Joakim Uddholm
88cf86f5f2 Merge pull request #1 from redapple/tethik_parse_command_rules_fix 
Add tests for crawl command non-default cases
 2016-09-19 00:51:36 +02:00
Paul Tremberth
48f6a065b8 Flush StreamLogger handlers 2016-09-17 15:25:45 +02:00
Paul Tremberth
27f88ad9cb Merge pull request #2260 from waynelovely/tutorial-fix-20160917-1 
Fix a dict key in the tutorial
 2016-09-17 14:19:21 +02:00
Wayne Lovely
cc8497abb1 Fix a dict key in the tutorial 2016-09-17 11:09:28 +00:00
Mikhail Korobov
992b2517b0 Merge pull request #2248 from redapple/scrapy-shell-import-scrapy 
[MRG+1] Make scrapy available in shell without explicit import statement
 2016-09-17 06:10:06 +06:00
Mikhail Korobov
91fcafde5e Merge pull request #2257 from scrapy/mention-stackoverflow 
Mentions stackoverflow as support channel (fixes #2255)
 2016-09-17 06:06:53 +06:00
Paul Tremberth
03ab077249 Feed exporter: start exporting only on first item 
Fixes GH-872
 2016-09-17 01:36:56 +02:00
Valdir Stumm Junior
233b98d642 include section describing spider arguments 2016-09-16 18:08:10 -03:00
Elias Dorneles
31545a9f84 tutorial: updating extracting data section to introduce CSS and XPath equally 2016-09-16 17:13:24 -03:00
Elias Dorneles
147e75602d update after review comments (thanks @stummjr) 2016-09-16 16:47:24 -03:00
Elias Dorneles
31260cf02f mentions stackoverflow as help channel (fixes #2255) 2016-09-16 16:05:36 -03:00
Elias Dorneles
de1a6ac677 Merge pull request #2249 from scrapy/fix-overview-spider 
[MRG+1] docs: update overview spider code to use toscrape.com and minor changes
 2016-09-16 16:00:23 -03:00
Elias Dorneles
21de617c77 mention that spiders need to subclass scrapy.Spider 2016-09-16 15:55:14 -03:00
Elias Dorneles
b2a5cddbb0 tutorial: update section about following links, expand examples 
adding an AuthorSpider to demonstrate further a different crawling
arrangement.
 2016-09-16 15:49:49 -03:00
Valdir Stumm Junior
0cd9dfcc85 small fixes on tutorial 2016-09-16 15:21:49 -03:00
Valdir Stumm Junior
0da497cf7a updates on the first section (our first spider) 2016-09-16 11:55:23 -03:00
Elias Dorneles
c508f40689 use harcoded URLs, remove item reference on second spider 2016-09-15 18:05:09 -03:00
Elias Dorneles
2427791287 tutorial: remove item class definition and present start_requests first 
This changes the tutorial, removing the step of creating an item class
and also starts by presenting the start_requests method instead of
start_urls.
 2016-09-15 17:46:31 -03:00
Elias Dorneles
75531e409e use better condition in example spider 2016-09-15 16:56:13 -03:00
Paul Tremberth
effaab867e Update shell help with availability of scrapy module 2016-09-15 21:37:15 +02:00
Elias Dorneles
1d159ae6f9 minor grammar fix 2016-09-15 15:37:03 -03:00
Elias Dorneles
18bd0b0886 docs: update overview spider code to use toscrape.com and minor changes 
So, this will replace the spider example code from the overview that
scrapes questions from StackOverflow by a spider scraping quotes (much
like the one in the tutorial), and upates the text around it to be
consistent.

There are also minor wording changes plus a small Sphinx/reST syntax fix
on the features list at the bottom (it was creating a definition list,
causing one line to be bold).
 2016-09-15 15:16:30 -03:00
Paul Tremberth
105163fece Make scrapy available in shell without explicit import statement 2016-09-15 19:26:53 +02:00