Pablo Hoffman
a953efd8e5
Automated merge with http://hg.scrapy.org/scrapy-stable
2009-12-12 15:40:16 -02:00
Ismael Carnales
4ecc909bc1
Fix RobotsTxtMiddleware reference in doc
2009-12-04 15:37:24 -02:00
Ismael Carnales
07344666e2
Move webconsole extensions doc to webconsole topic
2009-12-01 10:47:11 -02:00
Ismael Carnales
e694c8ed02
Remove domain references in close spider extension doc
2009-11-30 11:38:56 -02:00
Ismael Carnales
12a7ff7312
Rename Close domain to close spider in extensions doc
2009-11-30 11:36:18 -02:00
Ismael Carnales
8d9cedd88b
Reorder signals doc to respect alphabetical order
2009-11-30 11:29:19 -02:00
Ismael Carnales
93cc3d2715
Correct param formatting in item pipelines doc
2009-11-30 11:04:15 -02:00
Pablo Hoffman
6084be3b2e
added iter_all() function to scrapy.util.trackref module and improved memory leaks documentation. also added a new FAQ antry about memory issues
2009-11-28 16:21:59 -02:00
Pablo Hoffman
dd662e09d8
some minor fixes to scheduler middleware doc
2009-11-19 12:23:54 -02:00
Pablo Hoffman
f4e93700bd
Automated merge with http://hg.scrapy.org/scrapy-stable/
2009-11-19 10:44:02 -02:00
Pablo Hoffman
c4f77c4da0
minor fixes to images doc (thanks amccloud)
2009-11-16 11:15:25 -02:00
Pablo Hoffman
0d6aee1f12
updated wrong documentation
2009-11-13 20:03:56 -02:00
Pablo Hoffman
aeab5370cb
StatsCollector: ported methods to receive spider instances ( closes #113 ), removed list_domains() method, added iter_spider_stats() method
2009-11-14 20:28:59 -02:00
Pablo Hoffman
c4c6e7c8cd
Automated merge with http://hg.scrapy.org/scrapy-stable/
2009-11-13 20:04:39 -02:00
Pablo Hoffman
07655d05ea
renamed REQUESTS_PER_SPIDER setting to CONCURRENT_REQUESTS_PER_SPIDER
2009-11-13 14:38:22 -02:00
Pablo Hoffman
564abd10ad
Refactored HttpCache middleware:
...
* simplified code
* performance improvements
* removed awkward/unused domain sectorization
* it can now receive Settings on constructor
* added unittests
* added documentation about filesystem storage structure
Also made scrapy.conf.Settings objects instantiable with a dict which is used to override default settings.
2009-11-13 14:25:47 -02:00
Pablo Hoffman
db7fec1fef
fixed doc typo
2009-11-12 12:17:39 -02:00
Pablo Hoffman
415dec4e16
made offsite middleware log messages when filtering out requests
2009-11-12 10:17:21 -02:00
Pablo Hoffman
49e39bf1ba
fixed typo
2009-11-06 16:49:48 -02:00
Pablo Hoffman
791f4932dd
added clarification about versioning and api stability
2009-11-06 16:28:51 -02:00
Pablo Hoffman
74d0e82dbe
renamed CloseDomain extension to CloseSpider, and renamed CLOSEDOMAIN_* settings to CLOSESPIDER_*
...
--HG--
rename : scrapy/contrib/closedomain.py => scrapy/contrib/closespider.py
2009-11-06 15:54:17 -02:00
Pablo Hoffman
919cd5b789
renamed setting CONCURRENT_DOMAINS to CONCURRENT_SPIDERS
2009-11-06 15:44:11 -02:00
Pablo Hoffman
d604dca96d
renamed setting REQUESTS_PER_DOMAIN to REQUESTS_PER_SPIDER
2009-11-06 15:42:11 -02:00
Pablo Hoffman
7728a23e99
Changed item pipeline API to pass spider references (instead of domain names) to process_item() method
2009-11-06 13:46:36 -02:00
Pablo Hoffman
a432c1ee40
updated logging doc to include new spider argument in log functions
2009-11-04 14:49:24 -02:00
Pablo Hoffman
97c322707a
* Renamed domain_{opened,closed,idle} signals to spider_{opened,closed,idle}
...
* Changed them to pass spider instances only (no domains) (refs #105 )
2009-11-03 00:39:02 -02:00
Pablo Hoffman
904cde6513
added clarification about new dont_click argument of FormRequest.from_response() method
2009-10-29 13:47:10 -02:00
Ismael Carnales
a244d23b89
added dont_click attr to FormRequest
2009-10-29 13:18:13 -02:00
Pablo Hoffman
b41c5b5d5b
fixed typo in intro/install doc (thanks phaithful)
2009-10-29 10:41:20 -02:00
Pablo Hoffman
9b5fef4f48
fixed typo in intro/install doc (thanks phaithful)
2009-10-28 09:34:31 -02:00
Pablo Hoffman
7296a7b889
added DEFAULT_RESPONSE_ENCODING setting
2009-10-21 16:13:41 -02:00
Pablo Hoffman
720bc166cf
updated new clickdata argument doc
2009-10-20 17:21:56 -02:00
Daniel Grana
6abb3c17ee
Improve FormRequest.from_response method to pass click data arguments to ClientForm library
2009-10-20 15:51:41 -02:00
Pablo Hoffman
2712d55cb9
Automated merge with http://hg.scrapy.org/scrapy-stable
2009-10-07 23:58:38 -02:00
Pablo Hoffman
bd481751d8
moved images pipeline documentation to stable doc
...
--HG--
rename : docs/experimental/images.rst => docs/topics/images.rst
2009-10-07 22:57:25 -02:00
Pablo Hoffman
b4d202a6b0
added note about memory usage extension not working on windows
2009-10-07 22:57:10 -02:00
Pablo Hoffman
937acd91d1
improved documentation of http proxy middleware
2009-10-07 21:00:34 -02:00
Pablo Hoffman
d03bf1f27f
Automated merge with http://hg.scrapy.org/scrapy-stable
2009-10-07 20:38:46 -02:00
Pablo Hoffman
37d9e015bb
minor fix to tutorial
2009-10-07 20:15:49 -02:00
Daniel Grana
bc64ca3e13
Add support to set http proxies per request, and obey enviroment variables http_proxy and no_proxy by default.
2009-10-05 04:10:22 -02:00
Daniel Grana
8aa7d153ae
rewrote of downloader handlers
...
* add REQUEST_HANDLERS setting with defaults for file, http and https schemes
* add documentation of new setting
* add unittests for all the builtin handlers
* remove unused getPage function
2009-10-05 04:10:22 -02:00
Pablo Hoffman
a0eec7eaf6
some typos fixes and updates to install doc
2009-09-29 09:44:02 -03:00
Ismael Carnales
1646482bef
reformatted installation guide
2009-09-29 08:41:34 -03:00
Pablo Hoffman
4a9d9282bc
removed obsolete scrapy architecture dia diagram
2009-09-28 23:54:01 -03:00
Ismael Carnales
5862ba7db7
modified doc to reflect the new spider callback return policy (lists not needed)
2009-09-22 11:25:40 -03:00
Ismael Carnales
802f918b69
removed obsolete doc static file
2009-09-22 11:25:38 -03:00
Pablo Hoffman
6e93872955
updated installation guide for using releases
2009-09-17 11:06:55 -03:00
Pablo Hoffman
132557dd14
some deployment changes in preparation for the 0.7.0 release candidate
2009-09-16 22:40:36 -03:00
Ismael Carnales
fd41f06056
added doc on how to enable an Item Pipeline component
2009-09-16 14:19:16 -03:00
Ismael Carnales
404e7e09d7
changed spider doc references in BaseSpider class
2009-09-16 14:10:11 -03:00
