Pablo Hoffman
65aa9ccc70
Merge pull request #1220 from eliasdorneles/building-settingslist-from-docs
...
Building settings list from docs
2015-05-13 14:43:19 -03:00
Elias Dorneles
5753e498bf
fixes referencing, and list only settings not documented in current document
2015-05-09 16:15:06 -03:00
Elias Dorneles
d1053d2a2f
automatic settings list for docs [WIP]
2015-05-04 02:17:19 -03:00
Julia Medina
b827097c91
Rename scrapy/statscol.py to scrapy/statscollectors.py
2015-04-29 21:36:40 -03:00
Julia Medina
54a4ce069c
Rename scrapy/dupefilter.py to scrapy/dupefilters.py
2015-04-29 21:27:19 -03:00
Julia Medina
9a3e3ba505
Move scrapy/contrib remaining top-level files to scrapy/extensions
2015-04-29 21:27:19 -03:00
Julia Medina
180272c092
Move scrapy/contrib/spidermiddleware to scrapy/spidermiddlewares
2015-04-29 21:26:35 -03:00
Julia Medina
d7c444fefb
Move scrapy/contrib/downloadermiddleware to scrapy/downloadermiddlewares
2015-04-29 21:24:30 -03:00
Julia Medina
ccdd8bfbcc
Parametrize log formatting strings
2015-04-22 17:24:40 -03:00
Mikhail Korobov
403e7c7c70
rename scrapy.spidermanager.SpiderManager to scrapy.spiderloader.SpiderLoader
2015-04-16 20:07:53 +05:00
Alexander Sibiryakov
85aa3c7596
Dns cache size and timeout options
2015-04-02 18:30:59 +02:00
Daniel Graña
27591b55fc
Merge pull request #1123 from sibiryakov/reactor-threadpool-size
...
[MRG+1] Reactor threadpool max size setting
2015-04-01 15:08:03 -03:00
Alexander Sibiryakov
e7b274edf3
Reformat to 80 characters per line.
2015-04-01 11:49:55 +02:00
Alexander Sibiryakov
94fceb4c15
Fixing underscore size.
2015-04-01 11:25:10 +02:00
Alexander Sibiryakov
5864d291d4
Setting documentation.
2015-03-31 11:10:56 +02:00
nyov
1134a9cab0
config: look in ~/.config/scrapy.cfg as well
2015-03-26 20:36:14 +00:00
Shadab Zafar
5a58d64131
Fix some redirection links in documentation
...
Fixes #606
2015-03-18 19:41:26 -03:00
Elias Dorneles
f7031c08ff
updating list of Request.meta special keys
2015-03-10 22:29:07 -03:00
tpeng
a69f042d10
add 2 more test cases and minor doc fixes
2014-11-19 15:31:07 +01:00
tpeng
fa84730e70
avoid download large response
...
introduce DOWNLOAD_MAXSIZE and DOWNLOAD_WARNSIZE in settings and
download_maxsize/download_warnsize in spider/request meta, so
downloader stop downloading as soon as the received data exceed the
limit. also check the twsisted response's length in advance to stop
downloading as early as possible.
2014-11-12 12:28:02 +01:00
Mikhail Korobov
7d68b084a4
DOC document download_timeout Request.meta key and download_timeout spider attribute.
2014-10-07 04:23:11 +06:00
Julia Medina
16e62e9c9b
Per-spider settings documentation
2014-09-01 21:56:57 -03:00
Daniel Graña
ccde3317d7
Merge pull request #816 from Curita/api-cleanup
...
GSoC API cleanup
2014-09-01 21:55:36 -03:00
Daniel Graña
a9292cfab7
jsonrpc webservice moved to https://github.com/scrapy/scrapy-jsonrpc repository
2014-08-15 23:28:13 -03:00
Julia Medina
d7038b2a13
SpiderManager interface cleanup
2014-08-12 14:02:55 -03:00
Julia Medina
8a3a6236d9
Settings topic updated
2014-06-10 11:26:50 -03:00
Alexander Chekunkov
fa5a67729b
DOWNLOADER setting
2014-06-02 13:11:02 +03:00
Alexander Chekunkov
1fba64d34e
DOWNLOADER setting
2014-06-02 13:05:22 +03:00
Alexander Chekunkov
4aa6179af7
added short RFPDupeFilter.request_fingerprint interface description
2014-04-26 15:46:53 +03:00
Alexander Chekunkov
baaa077725
added note about RFPDupeFilter.request_fingerprint overriding to the settings documentation
2014-04-26 15:43:12 +03:00
Julia Medina
b9e2aad874
Doc for disabling download handler
2014-03-12 23:21:33 -03:00
Pablo Hoffman
6d8b7b29ef
remove unused setting: DOWNLOADER_DEBUG
2014-02-27 12:16:05 -02:00
Paul Tremberth
41765ca18d
DupeFilter: add setting for verbose logging + stats counter for filtered requests
2014-02-17 13:42:42 +01:00
Rolando Espinoza
28f946b05f
DOC Use pipelines module name instead of pipieline following default project files.
2014-02-15 11:01:26 -04:00
Mikhail Korobov
9a999daa2a
DOWNLOAD_DELAY docs clarification:
...
* delay is enforced per website, not per spider;
* document download_delay attribute (it was previously documented only in FAQ about 999 error codes);
* document how CONCURRENT_REQUESTS_PER_IP affects download delays.
2013-12-28 06:30:34 +06:00
Pablo Hoffman
e8ee449a2a
Merge pull request #432 from darkrho/crawl-url
...
Removed URL reference in crawl command and .tld suffix in docs for spider names
2013-10-21 09:40:58 -07:00
Rolando Espinoza La fuente
34543c2b2e
DOCS removed .tld suffix for spider names for the sake of consistency.
2013-10-19 23:03:20 -04:00
Pablo Hoffman
12280c2a95
fix sphinx references in doc
2013-09-25 15:13:17 -03:00
Pablo Hoffman
fc388f4636
Make ITEM_PIPELINE setting a dict
...
This is for consistency with how spider and downloader middlewares are
defined. ITEM_PIPELINE_BASE was also added and both remain empty.
Backwards compatibility is kept (with a warning) with list-based
ITEM_PIPELINES.
2013-09-23 17:50:43 -03:00
Pablo Hoffman
22edc44c6c
doc: remove links to diveintopython.org, which is no longer available. closes #246
2013-02-14 11:09:40 -02:00
Chris Tilden
aae6aed4fb
fixes spelling errors in documentation
2013-01-22 14:52:18 -08:00
Daniel Graña
076ba40404
update DOWNLOADER_MIDDLEWARES_BASE setting documentation
2013-01-08 10:50:27 -02:00
Pablo Hoffman
7a7c5d1334
removed reference to global scrapy stats from settings doc
2012-11-03 17:05:01 -02:00
Pablo Hoffman
1f89eb59fe
fixed doc reference to topics-contracts
2012-10-09 16:02:12 -02:00
Pablo Hoffman
c380910b40
Merge pull request #167 from alexcepoi/sep-017
...
Spider contracts (SEP-017)
2012-09-28 13:57:07 -07:00
Pablo Hoffman
b46b5a6ef0
Documented AutoThrottle extension and added to extensions available by
...
default. Also deprecated concurrency and delay settings, in favour of
using the standard Scrapy ones.
2012-09-20 18:52:57 -03:00
Pablo Hoffman
c7f8219901
- removed scrapy.conf singleton from scrapy.log, scrapy.responsetypes,
...
scrapy.http.response.text, scrapy.selector
- fixed bug with scrapy.conf.settings backwards compatibility support
- added facility to notify (and provide some guidelines) about deprecated/obsolete settings
2012-09-19 03:03:34 -03:00
Alex Cepoi
bf8dc61fb7
SEP-017 contracts: pretty-printing and docs
2012-09-10 23:17:27 +02:00
Pablo Hoffman
babfc6e79b
Updated documentation after singleton removal changes.
...
Also removed some unused code and made some minor additional
refactoring.
2012-08-28 18:35:57 -03:00
Pablo Hoffman
27018fced7
changed default user agent to Scrapy/0.15 (+ http://scrapy.org ) and removed no longer needed BOT_VERSION setting
2012-03-23 13:45:21 -03:00
