Paul Tremberth
44d8df2060
Add versionadded note for MEMUSAGE_CHECK_INTERVAL_SECONDS
2016-02-03 12:35:26 +01:00
Paul Tremberth
2b033eebce
Fix recently added HTTPCACHE_ settings versionadded notes
2016-02-03 12:34:44 +01:00
Paul Tremberth
142aa21737
Add AUTOTHROTTLE_TARGET_CONCURRENCY versionadded note
2016-02-03 12:33:10 +01:00
Paul Tremberth
c6f374f2eb
Merge remote-tracking branch 'origin/master' into 1550-shell_file-cont
2016-01-28 14:02:48 +01:00
Paul Tremberth
27fb200e45
Merge pull request #1727 from redapple/priority-adjust
...
[MRG+1] Clarify priority adjust settings docs
2016-01-27 18:52:39 +01:00
Mikhail Korobov
7ca9ae1976
DOC typo fix
2016-01-27 17:54:28 +05:00
Paul Tremberth
d999e3f7a7
More explicit description of DEPTH_PRIORITY
2016-01-27 12:57:03 +01:00
Paul Tremberth
6ed08d2332
Add note for DEPTH_PRIORITY
2016-01-27 11:53:29 +01:00
Mikhail Korobov
4bcbb77bcc
response.text. Fixes GH-1729.
2016-01-27 01:28:11 +05:00
Paul Tremberth
1c83108893
Clarify priority adjust settings docs
...
Fixes #1593
2016-01-26 19:24:11 +01:00
Paul Tremberth
bb1f4013a3
Rewrite warning about shell with local files as note
2016-01-26 17:23:28 +01:00
Mikhail Korobov
f30758c246
Enable robots.txt handling by default for new projects. Fixes GH-1668.
...
For backwards compatibility reasons the default value is not changed.
2016-01-26 17:47:46 +05:00
Νικόλαος-Διγενής Καραγιάννης
1cffa99e0d
tests+doc for subdomains in offsite middleware
2016-01-26 12:49:43 +02:00
Paul Tremberth
713e1eee9b
Update docs about local files support for "scrapy shell"
2016-01-26 10:44:38 +01:00
Raul Gallegos
a06a5f00f4
adding configurable encoding for httpproxy authentication
2016-01-21 09:35:04 -05:00
Elias Dorneles
d4c4ca8062
fix version number to appear new feature
2016-01-21 09:42:15 -02:00
Capi Etheriel
659715ecd9
implements FormRequest.from_response CSS support
2016-01-21 01:05:20 -02:00
Daniel Graña
b1e44436bc
Merge pull request #1642 from orangain/doc-metarefresh
...
[MRG+1] DOC: Update MetaRefreshMiddlware's setting variables
2016-01-19 13:33:50 -03:00
carlosp420
e15f361b05
fixed typo You -> you
2016-01-19 11:12:43 -05:00
Ralph Gutkowski
79147a61a7
Update stats.rst
2016-01-15 19:25:56 +01:00
Ralph Gutkowski
bb38400db5
Update Stats Collection documentation
...
`pages_crawled` value doesn't exist. Replace it with `downloader/response_count` in order to avoid confusion.
2016-01-15 19:00:58 +01:00
Valdir Stumm Jr
2abc9bc901
Update deprecated examples
...
* update the scrapy.org example to deal with the new layout.
* replaced slashdot.org by reddit.com, because it seems that slashdot is blocking requests.
2016-01-06 10:29:45 -02:00
orangain
bcce8d3d80
DOC: Update MetaRefreshMiddlware's setting variables
...
* `REDIRECT_MAX_METAREFRESH_DELAY` has been deprecated and was
renamed to `METAREFRESH_MAXDELAY`.
* Merge duplicate documents about `METAREFRESH_MAXDELAY` appeared both
in the settings page and the downloader-middlewares page.
2015-12-17 15:25:02 +09:00
palego
2c251000d3
custom project templates
...
allow override of TEMPLATES_DIR for startproject
copy full TEMPLATES_DIR/project tree
doc update
2015-12-01 03:07:27 +01:00
Julia Medina
48539af163
Merge pull request #1601 from mvj3/minor-documentation-fixes
...
Fixed minor grammar issues.
2015-11-16 14:58:07 -03:00
David Chen
0025d5a943
Fixed minor grammar issues.
2015-11-16 07:30:17 +08:00
Jakob de Maeyer
e66f649894
Bring back _BASE settings
2015-11-11 17:39:56 +01:00
Julia Medina
72eeead6db
Merge pull request #1524 from Digenis/relocate_telnet_console
...
[MRG+1] Relocate telnetconsole to extensions/
2015-10-30 12:10:38 -03:00
Jakob de Maeyer
bb6dee611c
Move Settings documentation to docstrings
2015-10-27 12:43:33 +01:00
Jakob de Maeyer
26586ef5a6
Deprecate _BASE settings, unify _BASE backwards-compatibility
2015-10-27 12:43:23 +01:00
Jakob de Maeyer
a769a1ef78
Introduce BaseSettings with full dictionary interface
2015-10-27 12:38:52 +01:00
Valdir Stumm Jr
d577c4702d
fixed a typo in the documentation.
2015-10-26 00:00:20 -02:00
Mikhail Korobov
1b6d60c251
DOC fix docs after GH-1289.
2015-10-09 01:26:09 +05:00
hoatle
2869cf8dde
fix another invalid xpath error
2015-10-07 16:03:43 +07:00
Hoat Le
4e66955411
fix ValueError: Invalid XPath: //div/[id="not-exists"]/text() on selectors.rst
...
>>> response.xpath('//div/[id="not-exists"]/text()').extract_first() is None
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/home/vagrant/.virtualenvs/scrapy/lib/python2.7/site-packages/scrapy/http/response/text.py", line 109, in xpath
return self.selector.xpath(query)
File "/home/vagrant/.virtualenvs/scrapy/lib/python2.7/site-packages/scrapy/selector/unified.py", line 100, in xpath
raise ValueError(msg if six.PY3 else msg.encode("unicode_escape"))
ValueError: Invalid XPath: //div/[id="not-exists"]/text()
2015-10-07 15:43:02 +07:00
Νικόλαος-Διγενής Καραγιάννης
d523c75f5e
Relocate telnetconsole to extensions/
2015-10-05 21:16:04 +03:00
smirecki
8379bea7ed
Typos corrections
...
I've made a few small corrections, some spelling changes and typo fixes.
I've tried to respect regional spelling differences and avoided proposing hyphenating compound words.
Please enter the commit message for your changes. Lines starting
2015-10-02 23:48:27 -04:00
Marius Gedminas
0620e76433
Fix list formatting
2015-09-29 03:33:30 +05:00
Marius Gedminas
eaad10facf
Typo
2015-09-28 14:44:15 +05:00
Daniel Graña
3c596dcf46
Merge pull request #1467 from dacjames/master
...
add support for a nested loaders
2015-09-16 22:03:19 -03:00
hy
14f7f22555
fix typos in downloader-middleware.rst and exceptions.rst, middlware -> middleware
2015-09-16 16:59:23 +08:00
Daniel Collins
036109e7de
updte nested loader documentation
2015-09-15 23:49:35 -07:00
Daniel Collins
88c92cb68b
provide documentation for nested loaders
2015-09-04 13:15:48 -07:00
Artur Gaspar
9ce9a293a6
Always check robots.txt before making another request in RobotsTxtMiddleware.
2015-09-02 10:23:24 -03:00
Artur Gaspar
ca83a0b028
Support for returning deferreds in downloader middleware methods.
2015-09-01 13:22:43 -03:00
David Tagatac
08162a15d8
minor: scrapy.Spider docs grammar
2015-08-27 17:37:16 -04:00
Mikhail Korobov
9616d91e4a
Merge pull request #1444 from cyberplant/bpython_support
...
[MRG +1] bpython support
2015-08-27 21:28:05 +05:00
Mikhail Korobov
cfae62f9cc
Merge pull request #1441 from aivarsk/fix-common-practices
...
Make common practices sample code match the comments
2015-08-23 17:36:09 +05:00
Jakob de Maeyer
d164398a27
Fix RedirectMiddleware not honouring meta handle_httpstatus keys
2015-08-21 13:22:42 +02:00
nyov
509cc8d41e
Add support for bpython console.
...
Adds support for configuration of shells from scrapy.cfg
and SCRAPY_PYTHON_SHELL.
config snippet:
cat <<EOF >> ~/.scrapy.cfg
[settings]
# shell can be one of ipython, bpython or python;
# to be tried as the interactive python console
# (in above order, unless set here).
shell = python
EOF
(closes #270 , #1100 , #1301 )
2015-08-21 01:12:58 +01:00
