BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-25 18:04:11 +00:00
Code Releases Wiki Activity
5,341 Commits 39 Branches 119 Tags
Commit Graph

1183 Commits

Author SHA1 Message Date
Elias Dorneles
7135c4e694 some minor grammar fixes 2015-03-27 17:56:56 -03:00
Pablo Hoffman
bb4c922d85 Merge pull request #1081 from scrapy/dict-items 
Allow spiders to return dicts.
 2015-03-27 15:19:27 -03:00
Daniel Graña
55a23d102f Merge pull request #1086 from Curita/response-urljoin 
Add Response.urljoin() helper
 2015-03-27 15:17:54 -03:00
Julia Medina
f4e241a018 Merge pull request #1106 from eliasdorneles/overview-page-improvements 
[MRG+1] some improvements to overview page
 2015-03-27 15:16:33 -03:00
Mikhail Korobov
39085ae18f Merge pull request #1098 from nyov/nyov/userconfig 
[+1 MRG]look in ~/.config/scrapy.cfg for user config
 2015-03-27 02:11:35 +05:00
nyov
1134a9cab0 config: look in ~/.config/scrapy.cfg as well 2015-03-26 20:36:14 +00:00
Peter Bronez
475766c73a Converted sel.xpath() calls to response.xpath() in Extracting the data 2015-03-26 15:34:30 -04:00
Elias Dorneles
4dcecc98f9 moved example data to a better place 2015-03-26 15:45:17 -03:00
Elias Dorneles
7402e27230 fix community link 2015-03-26 15:35:31 -03:00
Elias Dorneles
729861c864 fixing indentation 2015-03-26 15:31:42 -03:00
Elias Dorneles
13d0ecde77 addressing more review comments, to avoid ambiguity on desired reading flow 2015-03-26 15:26:16 -03:00
Elias Dorneles
76e3bf1250 addressing comments from the review plus further editing 2015-03-26 14:26:20 -03:00
Elias Dorneles
8f4a268f37 added bit about async requests, improved phrasing 2015-03-26 12:14:56 -03:00
Elias Dorneles
32423d4a33 some improvements to overview page 2015-03-25 19:27:52 -03:00
Ramiro Morales
933dbc6be6 Oops 2015-03-25 18:33:17 -03:00
Ramiro Morales
ca2575001e Add missing callback arg in jobs topic example. 2015-03-25 18:32:20 -03:00
Mikhail Korobov
5ac91e4883 DOC remove Dynamic Creation of Item Classes section 
It was a hack, and dicts-as-items cover most use cases.

Dicts don't allow to attach metadata to fields,
but e.g. adding "_meta" key and removing it in a custom serializer
is no worse than creating classes dynamically.
 2015-03-23 18:11:35 +05:00
Julia Medina
cda3922507 Add Response.urljoin() helper 2015-03-19 19:07:52 -03:00
Pablo Hoffman
c81eefaf81 fix doc links 2015-03-19 17:42:48 -03:00
Mikhail Korobov
8ac397670f DOC move .. module: declaration to a proper place 2015-03-19 21:41:36 +05:00
Faisal Anees
643984e1b4 Updated architecture.rst 
Added http://krondo.com/blog/?page_id=1327 as a resource
 2015-03-18 23:55:22 -03:00
Mikhail Korobov
f16a33f34e DOC change structure of spider docs: 
* start with scrapy.Spider, then mention spider arguments,
  then describe generic spiders;
* change wording regarding start_urls/start_requests;
* show an example of start_requests vs start_urls;
* show an example of dicts as items;
* as defining Item is an optional step now, docs for Items are
  moved below Spider docs.
 2015-03-19 05:25:15 +05:00
Mikhail Korobov
817dbc6cbd DOC mention dicts in documentation; explain better what are Items for 2015-03-19 05:16:14 +05:00
Julia Medina
959aaad205 Document re_first 2015-03-18 21:11:08 -03:00
Mateusz Golewski
127c6c694a Fix extract_first() docs 2015-03-18 21:11:08 -03:00
Mateusz Golewski
012211accd Add docs for extract_first() 2015-03-18 21:11:08 -03:00
Daniel Graña
12eedd90d1 fix truncated 0.24.5 release notes. closes #1084 2015-03-18 20:46:28 -03:00
Julia Medina
4fb818a250 Run linkfix over current docs 2015-03-18 20:04:14 -03:00
Julia Medina
ff987fb5a5 Add linkfix rule to docs Makefile 2015-03-18 20:00:03 -03:00
Shadab Zafar
c05f5f175e Added linkfix script to docs/utils 
https://github.com/scrapy/scrapy/pull/1041#issuecomment-78143576
 2015-03-18 19:41:26 -03:00
Shadab Zafar
5a58d64131 Fix some redirection links in documentation 
Fixes #606
 2015-03-18 19:41:26 -03:00
Nicolás Alejandro Ramírez Quiros
ee82fe0e24 Merge pull request #1016 from SudShekhar/jsonProcessor 
[MRG+1] Added JmesSelect
 2015-03-18 08:11:25 -03:00
Mikhail Korobov
39635e5f55 Allow spiders to return dicts. See GH-1064. 2015-03-18 07:26:56 +05:00
Pablo Hoffman
934584a355 Merge pull request #1020 from jojje/gzip_http_cache 
[MRG+1] add gzip compression to filesystem http cache backend
 2015-03-17 14:32:06 -03:00
Pablo Hoffman
f924567591 Merge pull request #983 from ArturGaspar/linkextractor_css 
[MRG+1] CSS support in link extractors
 2015-03-17 01:07:47 -03:00
nramirezuy
c13e23641b httpcache dont_cache meta #19 #689 2015-03-16 11:50:04 -03:00
Mikhail Korobov
c1bf8365d5 Merge pull request #1022 from scrapy/docs-reorder 
DOC reorder topics
 2015-03-13 16:40:47 +05:00
Mikhail Korobov
baf5c59386 Merge pull request #1071 from eliasdorneles/updating-request-meta-special-keys 
updating list of Request.meta special keys
 2015-03-13 16:38:19 +05:00
Elias Dorneles
57a5ee0097 added example value to set for proxy meta key 2015-03-12 23:20:44 -03:00
Elias Dorneles
f7031c08ff updating list of Request.meta special keys 2015-03-10 22:29:07 -03:00
Daniel Graña
c9d7386a32 Add 0.24.5 release notes 2015-02-25 19:36:40 -02:00
Sudhanshu Shekhar
839ffba971 Added the first version of SelectJmes 
Utilizes jmespath. Also, added tests and documentation for the same.
 2015-02-24 22:59:01 +05:30
Mikhail Korobov
6419f76bbc Merge pull request #1029 from dangra/contribute-to-master 
DOC contribute to master branch
 2015-02-03 02:10:47 +05:00
Nicolás Alejandro Ramírez Quiros
8a3b9b6131 Merge pull request #1011 from SudShekhar/master 
Extension example fix to something that makes more sense
 2015-01-30 15:45:52 -02:00
Sudhanshu Shekhar
e42a1ac1a1 Reset items_scraped instead of item_count 
items_scraped is the counter that needs to be reset each time we have scraped a specific number of items in the code instead of item_count (which represents the specific number of items needed before a message is logged). Updating the source code to reflect this.
Removed some irrelevant words from the log message.
Signed-off-by: Sudhanshu Shekhar <sudshekhar02@gmail.com>
 2015-01-30 23:13:06 +05:30
Daniel Graña
f0bdc14522 Tentative attention message about what document to read for contributions 2015-01-29 15:40:12 -02:00
Mikhail Korobov
a07b4353d6 DOC reorder topics 
* duplicate "topics/commands" link is removed;
* Request/Response docs are moved to "Basic Concepts";
* settings docs are moved to "Basic Concepts";
* exceptions docs are moved to "Basic Concepts";
* "signals" and "exporters" docs are moved to "Extending Scrapy";
* "Reference" section is dropped because it is empty now.
 2015-01-22 22:58:10 +05:00
Jonas Tingeborn
bd5d99a2d2 add gzip compression to filesystem http cache backend 2015-01-21 20:18:11 +01:00
Capi Etheriel
4bc14da59e Updates documentation on dynamic item classes. 
Fixes #398
 2015-01-19 17:21:56 -02:00
Mikhail Korobov
283d6a5344 DOC a couple more references are fixed 2015-01-19 22:07:03 +05:00