Artur Gaspar
9ce9a293a6
Always check robots.txt before making another request in RobotsTxtMiddleware.
2015-09-02 10:23:24 -03:00
Artur Gaspar
ca83a0b028
Support for returning deferreds in downloader middleware methods.
2015-09-01 13:22:43 -03:00
Daniel Graña
b4e382c867
Merge pull request #1318 from yarikoptic/fix-docs-explicit-rtd-theme-path
...
[MRG+1] DOC(ENH): specify path to rtd theme explicitly
2015-08-31 10:42:02 -03:00
David Tagatac
08162a15d8
minor: scrapy.Spider docs grammar
2015-08-27 17:37:16 -04:00
Mikhail Korobov
9616d91e4a
Merge pull request #1444 from cyberplant/bpython_support
...
[MRG +1] bpython support
2015-08-27 21:28:05 +05:00
Rick
aa88fd797a
Small grammatical change
2015-08-26 15:01:50 -03:00
Mikhail Korobov
cfae62f9cc
Merge pull request #1441 from aivarsk/fix-common-practices
...
Make common practices sample code match the comments
2015-08-23 17:36:09 +05:00
Jakob de Maeyer
d164398a27
Fix RedirectMiddleware not honouring meta handle_httpstatus keys
2015-08-21 13:22:42 +02:00
nyov
509cc8d41e
Add support for bpython console.
...
Adds support for configuration of shells from scrapy.cfg
and SCRAPY_PYTHON_SHELL.
config snippet:
cat <<EOF >> ~/.scrapy.cfg
[settings]
# shell can be one of ipython, bpython or python;
# to be tried as the interactive python console
# (in above order, unless set here).
shell = python
EOF
(closes #270 , #1100 , #1301 )
2015-08-21 01:12:58 +01:00
Aivars Kalvāns
b8b1e8e544
Make common practices sample code match the comments
2015-08-19 16:54:10 +03:00
Daniel Graña
54f9837804
Document how to get coverage report
2015-08-13 19:02:51 -03:00
Daniel Graña
f67a808d9e
Add 1.0.3 release notes
2015-08-11 13:49:49 -03:00
Daniel Graña
57fafc702d
Add 1.0.2 release notes
2015-08-06 18:02:12 -03:00
Mikhail Lyundin
ea4199f8e1
Minor method name fix
2015-08-05 22:42:53 +03:00
Daniel Graña
5e6c492967
Merge pull request #1364 from jdemaeyer/enhancement/spider-handles-redirects
...
[MRG+1] Make RedirectMiddleware respect Spider.handle_httpstatus_list
2015-08-02 23:00:00 -03:00
David Tagatac
08123207c5
minor: scrapy.Spider grammar and clarity
2015-07-31 17:01:59 -04:00
Mikhail Korobov
b6eb3404a2
Merge pull request #1335 from jdemaeyer/fix/docs-scrapy-path
...
Make Sphinx autodoc use local, not system-wide Scrapy
2015-07-30 15:33:02 +05:00
Jakob de Maeyer
9c12a3f23a
Remove incorrect path for Sphinx autodoc
2015-07-30 12:12:08 +02:00
Andrew Murray
198be7d679
Fixed typos
2015-07-16 23:59:02 +10:00
Jakob de Maeyer
c908d31660
Make RedirectMiddleware respect Spider.handle_httpstatus_list
2015-07-16 12:50:26 +02:00
Julia Medina
d706310d8b
Merge pull request #1151 from marven/cache-control
...
[MRG+1] RFC2616 policy enhancements + tests
2015-07-11 08:06:20 -03:00
Nikola Pavlović
7bef61dbbd
Fix reference.
2015-07-10 02:35:27 +02:00
Jakob de Maeyer
d438075a47
Make Sphinx autodoc use local, not system-wide Scrapy
2015-07-03 16:51:53 +02:00
Julia Medina
8b3ca4f250
Merge pull request #1302 from eliasdorneles/improving-access-settings-docs
...
[MRG+1] Improvements for docs on how to access settings
2015-07-03 00:56:32 -03:00
Daniel Graña
3fc4e0b319
Merge pull request #1282 from otherchirps/memusage-check-interval
...
[MRG+1] Added MEMUSAGE_CHECK_INTERVAL_SECONDS to Memory usage extension options.
2015-07-02 13:50:55 -03:00
Daniel Graña
6fa3f2421c
Add 1.0.1 release notes
2015-07-01 01:49:44 -03:00
Mikhail Korobov
d850238c22
add AUTOTHROTTLE_TARGET_CONCURRENCY option and expand AutoThrottle docs
2015-06-27 04:59:42 +05:00
Mikhail Korobov
63317531f9
DOC fix authrottle docs
...
see https://github.com/scrapy/scrapy/pull/502/files#r8574692
2015-06-26 20:47:58 +05:00
Yaroslav Halchenko
5f3e1e9a16
ENH: make explicit build-ignore-errors to be used by linkfix
2015-06-25 21:30:35 -04:00
Yaroslav Halchenko
48582be9a6
DOC(ENH): specify path to rtd theme explicitly
2015-06-25 10:46:22 -04:00
Yaroslav Halchenko
49fe915798
BF: fail if docs failed to build
2015-06-25 10:12:56 -04:00
Marco DallaG
44c8ef2d47
DOC Bring Ubuntu and Archlinux outside of Windows subsection
...
In the installation guide, in the "Platform specific installation notes" section, Ubuntu and ArchLinux are currently subsections of Windows, which does not make sense imho.
This commit changes the section tree from:
Platform specific installation notes
- Windows
-- Archlinux
-- Ubuntu
To:
Platform specific installation notes
- Windows
- Ubuntu
- Archlinux
2015-06-24 11:44:48 +02:00
Pablo Hoffman
38e5bfb61c
remove version suffix from ubuntu package
2015-06-22 10:57:24 -03:00
Julia Medina
af97c373ea
Update release date for 1.0
...
(cherry picked from commit c89fa29c415f669fd42ad4bae88e754ed0194a4f)
2015-06-19 18:53:54 -03:00
Elias Dorneles
2de5c66058
improvements for docs on how to access settings
2015-06-15 13:07:55 -03:00
Julia Medina
f958fb9d62
Update release notes for 1.0.0rc3
2015-06-12 16:42:39 -03:00
Julia Medina
fa1c25c840
Merge pull request #1286 from scrapy/configure_logging
...
configure_logging: change the meaning of settings=None
2015-06-12 13:22:42 -03:00
Julia Medina
36bc912cdd
DOC indent additional docs for configure_logging
2015-06-12 13:00:31 -03:00
Bryan Crowe
6a4c475e87
Fix a couple typos
2015-06-11 19:47:30 -04:00
Daniel Graña
5bd0395be4
Merge pull request #1291 from scrapy/signalmanager-docstrings
...
DOC SignalManager docstrings. See GH-713.
2015-06-10 16:28:35 -03:00
Mikhail Korobov
6c9daf3a95
DOC remove unnecessary links; fix references in send_catch_log_deferred docstring
2015-06-10 01:44:19 +05:00
Mikhail Korobov
a611f8dd2d
DOC remove FailureFormatter mentions, stop copy-pasting configure_logging docstring
2015-06-09 22:57:18 +05:00
Mikhail Korobov
790c67b643
DOC spider_error doesn't support deferreds
2015-06-09 02:20:10 +05:00
Mikhail Korobov
1740fcf1a6
DOC SignalManager docstrings. See GH-713.
...
This change is not 100% backwards compatible because of *args changes.
Their usage was not documented, so we're not breaking public interface.
2015-06-08 21:05:58 +05:00
Mikhail Korobov
9a787893e3
(backwards-incompatible) allow to pass settings=None to configure_logging
...
* use explicit argument for disabling root handler;
* handle LOG_STDOUT even if install_root_handler is False
2015-06-08 19:54:18 +05:00
Chris Nilsson
0c532baf4c
Removed typo, and clarified time unit of setting
2015-06-06 11:18:13 +10:00
Mikhail Korobov
d047665c02
make "settings" argument optional for Crawler, CrawlerRunner and CrawlerProcess
2015-06-06 03:23:13 +05:00
Julia Medina
24d8a85269
Update release notes for 1.0.0rc2
...
(cherry picked from commit 6e61d54168cf471363be3e7e54d75ad544b9f6e1)
2015-06-05 17:11:40 -03:00
Chris Nilsson
eae25a04d9
Added MEMUSAGE_CHECK_INTERVAL_SECONDS to Memory usage extension options.
...
Kept the default as it was, at 60.0 seconds. But added a setting to
allow this to be changed as desired.
2015-06-06 00:39:14 +10:00
Julia Medina
367ea81e71
Remove deprecated %z formatting from the default LOG_DATEFORMAT
2015-06-04 04:11:23 +08:00
