BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-03-15 04:50:33 +00:00
Code Releases Wiki Activity
8,214 Commits 46 Branches 119 Tags
Commit Graph

242 Commits

Author SHA1 Message Date
Paul Tremberth
1c83108893 Clarify priority adjust settings docs 
Fixes #1593
 2016-01-26 19:24:11 +01:00
Mikhail Korobov
f30758c246 Enable robots.txt handling by default for new projects. Fixes GH-1668. 
For backwards compatibility reasons the default value is not changed.
 2016-01-26 17:47:46 +05:00
orangain
bcce8d3d80 DOC: Update MetaRefreshMiddlware's setting variables 
* `REDIRECT_MAX_METAREFRESH_DELAY` has been deprecated and was
  renamed to `METAREFRESH_MAXDELAY`.
* Merge duplicate documents about `METAREFRESH_MAXDELAY` appeared both
  in the settings page and the downloader-middlewares page.
 2015-12-17 15:25:02 +09:00
palego
2c251000d3 custom project templates 
allow override of TEMPLATES_DIR for startproject
copy full TEMPLATES_DIR/project tree
doc update
 2015-12-01 03:07:27 +01:00
Jakob de Maeyer
e66f649894 Bring back _BASE settings 2015-11-11 17:39:56 +01:00
Julia Medina
72eeead6db Merge pull request #1524 from Digenis/relocate_telnet_console 
[MRG+1] Relocate telnetconsole to extensions/
 2015-10-30 12:10:38 -03:00
Jakob de Maeyer
26586ef5a6 Deprecate _BASE settings, unify _BASE backwards-compatibility 2015-10-27 12:43:23 +01:00
Νικόλαος-Διγενής Καραγιάννης
d523c75f5e Relocate telnetconsole to extensions/ 2015-10-05 21:16:04 +03:00
Julia Medina
8b3ca4f250 Merge pull request #1302 from eliasdorneles/improving-access-settings-docs 
[MRG+1] Improvements for docs on how to access settings
 2015-07-03 00:56:32 -03:00
Daniel Graña
3fc4e0b319 Merge pull request #1282 from otherchirps/memusage-check-interval 
[MRG+1] Added MEMUSAGE_CHECK_INTERVAL_SECONDS to Memory usage extension options.
 2015-07-02 13:50:55 -03:00
Mikhail Korobov
d850238c22 add AUTOTHROTTLE_TARGET_CONCURRENCY option and expand AutoThrottle docs 2015-06-27 04:59:42 +05:00
Elias Dorneles
2de5c66058 improvements for docs on how to access settings 2015-06-15 13:07:55 -03:00
Chris Nilsson
0c532baf4c Removed typo, and clarified time unit of setting 2015-06-06 11:18:13 +10:00
Chris Nilsson
eae25a04d9 Added MEMUSAGE_CHECK_INTERVAL_SECONDS to Memory usage extension options. 
Kept the default as it was, at 60.0 seconds. But added a setting to
allow this to be changed as desired.
 2015-06-06 00:39:14 +10:00
Julia Medina
367ea81e71 Remove deprecated %z formatting from the default LOG_DATEFORMAT 2015-06-04 04:11:23 +08:00
Daniel Graña
1195f906d6 Merge pull request #1218 from Curita/move-base-to-packages 
Move base classes to their packages
 2015-05-14 18:28:17 -03:00
Pablo Hoffman
65aa9ccc70 Merge pull request #1220 from eliasdorneles/building-settingslist-from-docs 
Building settings list from docs
 2015-05-13 14:43:19 -03:00
Elias Dorneles
5753e498bf fixes referencing, and list only settings not documented in current document 2015-05-09 16:15:06 -03:00
Julia Medina
d3f576a816 Move scrapy/spider.py to scrapy/spiders/__init__.py 2015-05-09 04:20:09 -03:00
Elias Dorneles
d1053d2a2f automatic settings list for docs [WIP] 2015-05-04 02:17:19 -03:00
Julia Medina
b827097c91 Rename scrapy/statscol.py to scrapy/statscollectors.py 2015-04-29 21:36:40 -03:00
Julia Medina
54a4ce069c Rename scrapy/dupefilter.py to scrapy/dupefilters.py 2015-04-29 21:27:19 -03:00
Julia Medina
9a3e3ba505 Move scrapy/contrib remaining top-level files to scrapy/extensions 2015-04-29 21:27:19 -03:00
Julia Medina
180272c092 Move scrapy/contrib/spidermiddleware to scrapy/spidermiddlewares 2015-04-29 21:26:35 -03:00
Julia Medina
d7c444fefb Move scrapy/contrib/downloadermiddleware to scrapy/downloadermiddlewares 2015-04-29 21:24:30 -03:00
Julia Medina
ccdd8bfbcc Parametrize log formatting strings 2015-04-22 17:24:40 -03:00
Mikhail Korobov
403e7c7c70 rename scrapy.spidermanager.SpiderManager to scrapy.spiderloader.SpiderLoader 2015-04-16 20:07:53 +05:00
Alexander Sibiryakov
85aa3c7596 Dns cache size and timeout options 2015-04-02 18:30:59 +02:00
Daniel Graña
27591b55fc Merge pull request #1123 from sibiryakov/reactor-threadpool-size 
[MRG+1] Reactor threadpool max size setting
 2015-04-01 15:08:03 -03:00
Alexander Sibiryakov
e7b274edf3 Reformat to 80 characters per line. 2015-04-01 11:49:55 +02:00
Alexander Sibiryakov
94fceb4c15 Fixing underscore size. 2015-04-01 11:25:10 +02:00
Alexander Sibiryakov
5864d291d4 Setting documentation. 2015-03-31 11:10:56 +02:00
nyov
1134a9cab0 config: look in ~/.config/scrapy.cfg as well 2015-03-26 20:36:14 +00:00
Shadab Zafar
5a58d64131 Fix some redirection links in documentation 
Fixes #606
 2015-03-18 19:41:26 -03:00
Elias Dorneles
f7031c08ff updating list of Request.meta special keys 2015-03-10 22:29:07 -03:00
tpeng
a69f042d10 add 2 more test cases and minor doc fixes 2014-11-19 15:31:07 +01:00
tpeng
fa84730e70 avoid download large response 
introduce DOWNLOAD_MAXSIZE and DOWNLOAD_WARNSIZE in settings and
download_maxsize/download_warnsize in spider/request meta, so
downloader stop downloading as soon as the received data exceed the
limit. also check the twsisted response's length in advance to stop
downloading as early as possible.
 2014-11-12 12:28:02 +01:00
Mikhail Korobov
7d68b084a4 DOC document download_timeout Request.meta key and download_timeout spider attribute. 2014-10-07 04:23:11 +06:00
Julia Medina
16e62e9c9b Per-spider settings documentation 2014-09-01 21:56:57 -03:00
Daniel Graña
ccde3317d7 Merge pull request #816 from Curita/api-cleanup 
GSoC API cleanup
 2014-09-01 21:55:36 -03:00
Daniel Graña
a9292cfab7 jsonrpc webservice moved to https://github.com/scrapy/scrapy-jsonrpc repository 2014-08-15 23:28:13 -03:00
Julia Medina
d7038b2a13 SpiderManager interface cleanup 2014-08-12 14:02:55 -03:00
Julia Medina
8a3a6236d9 Settings topic updated 2014-06-10 11:26:50 -03:00
Alexander Chekunkov
fa5a67729b DOWNLOADER setting 2014-06-02 13:11:02 +03:00
Alexander Chekunkov
1fba64d34e DOWNLOADER setting 2014-06-02 13:05:22 +03:00
Alexander Chekunkov
4aa6179af7 added short RFPDupeFilter.request_fingerprint interface description 2014-04-26 15:46:53 +03:00
Alexander Chekunkov
baaa077725 added note about RFPDupeFilter.request_fingerprint overriding to the settings documentation 2014-04-26 15:43:12 +03:00
Julia Medina
b9e2aad874 Doc for disabling download handler 2014-03-12 23:21:33 -03:00
Pablo Hoffman
6d8b7b29ef remove unused setting: DOWNLOADER_DEBUG 2014-02-27 12:16:05 -02:00
Paul Tremberth
41765ca18d DupeFilter: add setting for verbose logging + stats counter for filtered requests 2014-02-17 13:42:42 +01:00