Pablo Hoffman
9aefa242d5
Applied documentation patch provided by Lucian Ursu ( closes #207 )
2010-08-21 01:26:35 -03:00
Pablo Hoffman
f782245c5a
Removed obsolete files
2010-08-21 01:24:39 -03:00
Pablo Hoffman
1d3b9e2ca8
Scrapy shell refactoring
2010-08-20 11:26:14 -03:00
Pablo Hoffman
7858244dca
Scrapy shell: moved python console starting code to scrapy.utils.console and get rid of noisy console banners
2010-08-20 01:33:02 -03:00
Pablo Hoffman
30e2404d8f
updated FAQ entry to recommend using higher download delays
2010-08-19 17:59:52 -03:00
Pablo Hoffman
3d8151bb26
Added FAQ entry about response code 999
2010-08-19 16:51:51 -03:00
Pablo Hoffman
2ff5a83b7a
Added persistent execution queue (based on SQLite), and a new 'queue' command to control it. Closes #198
2010-08-19 02:55:52 -03:00
Pablo Hoffman
94ead94bf6
Improved documentation of Scrapy command-line tool
...
--HG--
rename : docs/topics/cmdline.rst => docs/topics/commands.rst
2010-08-19 00:04:52 -03:00
Pablo Hoffman
34554da201
Deprecated scrapy-ctl.py command in favour of simpler "scrapy" command. Closes #199 . Also updated documenation accordingly and added convenient scrapy.bat script for running from Windows.
...
--HG--
rename : debian/scrapy-ctl.1 => debian/scrapy.1
rename : docs/topics/scrapy-ctl.rst => docs/topics/cmdline.rst
2010-08-18 19:48:32 -03:00
Pablo Hoffman
a71521bfba
Default per-command settings are now specified in the default_settings attribute of the command object. Closes #201
2010-08-17 18:30:13 -03:00
Pablo Hoffman
ad3fd0afe8
fixed minor formatting issue with new feed exports doc
2010-08-17 14:37:59 -03:00
Pablo Hoffman
e741a807d2
Added new Feed exports extension with documentation and storage tests. Closes #197 .
...
Also deprecated File export pipeline (to be removed in Scrapy 0.11).
Still need to add tests for FeedExport main extension code.
2010-08-17 14:27:48 -03:00
Pablo Hoffman
3e3a66620b
Added support for returning deferreds from (some) signal handlers. Closes #193
2010-08-14 21:10:37 -03:00
Pablo Hoffman
1df2c17b78
updated old documentation references
2010-08-12 20:45:11 -03:00
Pablo Hoffman
43d47e5d9b
Some improvements to Item Pipeline ( closes #195 ):
...
* Made Item Pipeline Manager a subclass of scrapy.middleware.MiddlewareManager
* Added open_spider/close_spider methods with support for returning deferreds from them
* Inverted the process_item() arguments to be more friendly with deferred
callbacks (backwards compatibility kept through arguments introspection)
* Updated documentation with new methods and process_item() arguments change
2010-08-12 10:48:37 -03:00
Pablo Hoffman
9d38a99aa8
updated missing doc reference from previous commit
2010-08-10 17:47:04 -03:00
Pablo Hoffman
784722774b
moved scrapy.core.signals to scrapy.signals, keeping backwards compatibility
2010-08-10 17:40:53 -03:00
Pablo Hoffman
c359a34d7d
moved scrapy.core.exceptions to scrapy.exceptions, keeping backwards compatibility
...
--HG--
rename : scrapy/core/exceptions.py => scrapy/exceptions.py
2010-08-10 17:36:48 -03:00
Pablo Hoffman
b1c0280616
removed scheduler middleware doc, as scheduler middleware will be removed soon
2010-08-10 16:59:49 -03:00
Pablo Hoffman
c7d9f6e270
Added JSON item exporter with doc and unittests ( closes #192 ), and also:
...
* put all json exporters in scrapy.contrib.exporters and deprecated
scrapy.contrib.exporters.jsonlines to reduce module nesting
* use JSON exporter with EXPORT_FORMAT=json in file export pipeline
2010-08-07 15:52:59 -03:00
Pablo Hoffman
49851d7f55
Automated merge with http://hg.scrapy.org/scrapy-0.9
2010-08-02 17:20:55 -03:00
Pablo Hoffman
6c68e4ce15
fixed documentation typo
2010-08-02 17:20:13 -03:00
Pablo Hoffman
453e7bf38c
Scrapy logging refactoring ( closes #188 ):
...
* added Twisted log observer for Scrapy, with unittests
* use numeric values from Python logging module for log levels
* removed scrapy.log.exc() function - use scrapy.log.err() instead
* removed logmessage_received signal - write a (twisted) log observer instead
* dropped support for obsolete `domain` argument
* dropped support for old setting names: LOGLEVEL, LOGFILE (replaced by LOG_LEVEL, LOG_FILE)
* deprecated `component` argument
2010-08-02 08:49:14 -03:00
Ismael Carnales
e145ec686c
Replaced default spider manager (TwistedPluginSpiderManger) with a simpler one that doesn't depend on Twisted Plugins infrastructure.
2010-07-30 17:30:32 -03:00
Pablo Hoffman
e2290a5359
Some changes to Crawl spider:
...
* added process_request attribute to rules
* removed docstrings, since it duplicates documentation
2010-07-22 18:40:35 -03:00
Daniel Grana
3e013f564b
update docs for defaultheaders middleware and change spider attribute to match global setting name
2010-07-16 16:17:08 -03:00
Daniel Grana
6883a99c1e
Automated merge with ssh://hg.scrapy.org/scrapy-0.9
2010-07-16 14:56:00 -03:00
Pablo Hoffman
b91d40ba78
Fixed grammar error in doc (patch by stav) - closes #176
2010-07-16 11:34:18 -03:00
Pablo Hoffman
90a04f0530
Automated merge with http://hg.scrapy.org/scrapy-0.9
2010-07-13 19:47:55 -03:00
Pablo Hoffman
9e37ec4230
fixed documentation typo ( closes #151 )
2010-07-13 19:03:02 -03:00
Ping Yin
b3a65d3313
HTTPCACHE: Don't cache response with codes in HTTPCACHE_IGNORE_HTTP_CODES
2010-07-09 13:14:25 -03:00
Ismael Carnales
2571e1b7aa
docs: Some DjangoItem docs improvements, closes #134 . Thanks tn!
2010-06-27 09:09:54 -03:00
Pablo Hoffman
115e9f2162
Added FAQ entry about running Scrapy deployment.
2010-06-14 18:21:12 -03:00
Pablo Hoffman
ede1df4b4f
updated copyright year, and indentation space
2010-06-14 07:16:51 -03:00
Pablo Hoffman
bd16d1cd48
Added SMTP-AUTH support to scrapy.mail ( closes #149 )
2010-06-13 17:14:46 -03:00
Pablo Hoffman
6a33d6c4d0
* Added Scrapy Web Service with documentation and tests.
...
* Marked Web Console as deprecated.
* Removed Web Console documentation to discourage its use.
2010-06-09 13:46:22 -03:00
Pablo Hoffman
73305b1eb3
Added support for Requests without callbacks ( #166 ) - the Spider.parse() method
...
is used in those cases.
Also removed Request.deferred attribute.
2010-06-08 18:18:02 -03:00
Pablo Hoffman
38b5793152
Some changes to telnet console:
...
* moved module from scrapy.management.telnet to scrapy.telnet (to minimize
nested modules)
* added signal for updating telnet console variables (fixes #165 )
--HG--
rename : scrapy/management/telnet.py => scrapy/telnet.py
2010-06-02 17:49:18 -03:00
Pablo Hoffman
031eb1e5ed
removed no longer used SpiderScheduler (obsoleted by ExecutionQueue)
2010-05-28 17:27:15 -03:00
Ismael Carnales
a71dc295af
Some mail improvements and tests.
...
* Add mail_sent signal and use it in MailSender
* Add MAIL_DEBUG setting to not send mails when testing
* Add MailSender tests
2010-05-28 16:51:47 -03:00
Ping Yin
6059221716
Compose: stop process on None value by default
...
By doing this, we can use str.lower as a processor safely without
checking whether the given value is None.
By passing stop_on_none=False as keyword argument, this behaviour can be changed.
Signed-off-by: Ping Yin <pkufranky@gmail.com>
2010-04-08 10:59:47 +08:00
Ping Yin
15b879f845
ItemLoader: Update docs for {add,replace,get}_{value,xpath}
...
Signed-off-by: Ping Yin <pkufranky@gmail.com>
2010-05-18 17:54:25 +08:00
Pablo Hoffman
bfd9cb42e5
Automated merge with http://hg.scrapy.org/scrapy-0.8
2010-05-17 20:11:27 -03:00
Pablo Hoffman
076cdfd585
Added documentation about contributing to Scrapy
2010-05-17 20:10:46 -03:00
Pablo Hoffman
7a55158fed
fixed documentation bug (thanks rhill for reporting)
2010-05-11 11:25:03 -03:00
Steven Almeroth
5d03405cac
FormRequest.from_response doc fix. closes #155
...
--HG--
extra : rebase_source : d54979f6a15e5e997072dcbbc6d43b426189312b
2010-04-26 22:28:07 -03:00
Pablo Hoffman
2121a30c74
added note about installing Zope.Interface in windows platforms
2010-04-24 18:19:52 -03:00
Daniel Grana
6c12106803
Remove shpinx warning introduced by shorter title overline
2010-04-18 23:42:56 -03:00
Lucian Ursu
2f8c052484
#154 : Language fixes to the documentation
2010-04-18 23:39:54 -03:00
Pablo Hoffman
dfdac356af
added missing default values to file xporter doc
2010-04-02 02:49:18 -03:00
