Pablo Hoffman
f4e93700bd
Automated merge with http://hg.scrapy.org/scrapy-stable/
2009-11-19 10:44:02 -02:00
Pablo Hoffman
bf55a4708f
prevent 'import scrapy' from failing when twisted module is not available, also moved twisted 2.5.0 monkeypatch into a more specific module name
2009-11-19 10:41:36 -02:00
Pablo Hoffman
c4f77c4da0
minor fixes to images doc (thanks amccloud)
2009-11-16 11:15:25 -02:00
Pablo Hoffman
0d6aee1f12
updated wrong documentation
2009-11-13 20:03:56 -02:00
Daniel Grana
445d8cd9f0
delay next_request check after stoping a spider close to avoid 100% cpu usage loops in some cases
2009-11-06 22:02:12 -02:00
Pablo Hoffman
aeab5370cb
StatsCollector: ported methods to receive spider instances ( closes #113 ), removed list_domains() method, added iter_spider_stats() method
2009-11-14 20:28:59 -02:00
Pablo Hoffman
c4c6e7c8cd
Automated merge with http://hg.scrapy.org/scrapy-stable/
2009-11-13 20:04:39 -02:00
Pablo Hoffman
505dfe6f7e
fixed exception when running scrapy shell on a non-textual response ( fixes #116 )
2009-11-13 17:21:59 -02:00
Pablo Hoffman
f3e861c89e
replaced old reference to domain instead of spider
2009-11-13 14:44:03 -02:00
Pablo Hoffman
07655d05ea
renamed REQUESTS_PER_SPIDER setting to CONCURRENT_REQUESTS_PER_SPIDER
2009-11-13 14:38:22 -02:00
Pablo Hoffman
564abd10ad
Refactored HttpCache middleware:
...
* simplified code
* performance improvements
* removed awkward/unused domain sectorization
* it can now receive Settings on constructor
* added unittests
* added documentation about filesystem storage structure
Also made scrapy.conf.Settings objects instantiable with a dict which is used to override default settings.
2009-11-13 14:25:47 -02:00
Pablo Hoffman
db7fec1fef
fixed doc typo
2009-11-12 12:17:39 -02:00
Pablo Hoffman
415dec4e16
made offsite middleware log messages when filtering out requests
2009-11-12 10:17:21 -02:00
Pablo Hoffman
ee08d38ab6
removed deprecated SCRAPYSETTINGS_MODULE environment variable
2009-11-06 16:56:17 -02:00
Pablo Hoffman
49e39bf1ba
fixed typo
2009-11-06 16:49:48 -02:00
Pablo Hoffman
4a2a20489d
removed deprecated ScrapedItem (previously kept for backwards compatibility)
2009-11-06 16:39:15 -02:00
Pablo Hoffman
791f4932dd
added clarification about versioning and api stability
2009-11-06 16:28:51 -02:00
Pablo Hoffman
9bf4e87753
removed deprecated scrapy.xpath module (previously kept for backwards compatibility)
2009-11-06 16:20:38 -02:00
Pablo Hoffman
40646d3cd1
replaced remaining uses of log.msg() 'domain' argument to use 'spider' instead
2009-11-06 16:11:37 -02:00
Pablo Hoffman
fee1b751e5
deprecated 'domain' argument in log.msg()
2009-11-06 16:02:12 -02:00
Pablo Hoffman
74d0e82dbe
renamed CloseDomain extension to CloseSpider, and renamed CLOSEDOMAIN_* settings to CLOSESPIDER_*
...
--HG--
rename : scrapy/contrib/closedomain.py => scrapy/contrib/closespider.py
2009-11-06 15:54:17 -02:00
Pablo Hoffman
cff4592b64
changed label
2009-11-06 15:48:49 -02:00
Pablo Hoffman
919cd5b789
renamed setting CONCURRENT_DOMAINS to CONCURRENT_SPIDERS
2009-11-06 15:44:11 -02:00
Pablo Hoffman
d604dca96d
renamed setting REQUESTS_PER_DOMAIN to REQUESTS_PER_SPIDER
2009-11-06 15:42:11 -02:00
Pablo Hoffman
580d82468e
fixed images pipeline bug caused by recent api changes
2009-11-06 14:29:37 -02:00
Pablo Hoffman
e5cae1e6c9
renamed setting
2009-11-06 14:19:56 -02:00
Pablo Hoffman
7728a23e99
Changed item pipeline API to pass spider references (instead of domain names) to process_item() method
2009-11-06 13:46:36 -02:00
Pablo Hoffman
a432c1ee40
updated logging doc to include new spider argument in log functions
2009-11-04 14:49:24 -02:00
Pablo Hoffman
69058e3b50
fixed bug in log.msg when receiving new spider argument
2009-11-04 14:45:56 -02:00
Pablo Hoffman
55d584e478
added missing spider argument to log.msg call
2009-11-04 14:45:34 -02:00
Daniel Grana
87e322e568
fixes to lxml link extractors api and encoding handling
2009-11-04 12:39:58 -02:00
Pablo Hoffman
97c322707a
* Renamed domain_{opened,closed,idle} signals to spider_{opened,closed,idle}
...
* Changed them to pass spider instances only (no domains) (refs #105 )
2009-11-03 00:39:02 -02:00
Pablo Hoffman
a3f2933912
another minor change in stats test
2009-11-02 20:28:36 -02:00
Pablo Hoffman
65b6065323
minor change in stats test
2009-11-02 19:16:42 -02:00
Pablo Hoffman
4d4f5dca12
Automated merge with http://hg.scrapy.org/scrapy-stable
2009-10-31 14:37:46 -02:00
Pablo Hoffman
d5ae94df55
fixed bug when using log.start() with log level module constants instead of string names, and added regression tests
2009-10-31 14:36:38 -02:00
Pablo Hoffman
904cde6513
added clarification about new dont_click argument of FormRequest.from_response() method
2009-10-29 13:47:10 -02:00
Pablo Hoffman
77b8cfadc7
changed odd logic in if statement
2009-10-29 13:35:32 -02:00
Daniel Grana
b7a00cd2a1
return a generator in CrawlSpider
2009-10-29 13:24:30 -02:00
Ismael Carnales
a244d23b89
added dont_click attr to FormRequest
2009-10-29 13:18:13 -02:00
Pablo Hoffman
b41c5b5d5b
fixed typo in intro/install doc (thanks phaithful)
2009-10-29 10:41:20 -02:00
Pablo Hoffman
9b5fef4f48
fixed typo in intro/install doc (thanks phaithful)
2009-10-28 09:34:31 -02:00
Pablo Hoffman
7baff2914a
fixed bug caused when instantiating selectors with responses containing empty bodies
2009-10-21 16:37:30 -02:00
Pablo Hoffman
53ae45ad3d
restored previous try/except catch all blocks for Libxml2Document __del__() method
2009-10-21 16:25:49 -02:00
Pablo Hoffman
7296a7b889
added DEFAULT_RESPONSE_ENCODING setting
2009-10-21 16:13:41 -02:00
Daniel Grana
3d7a4c890e
fix get_meta_refresh bug raised for TextResponses without encoding
2009-10-21 13:57:06 -02:00
Daniel Grana
6405efa507
factorize redirectmw code that handles 302 and meta-refresh redirections
...
--HG--
extra : rebase_source : f78fee093ac076d4a0630982dc4bade16755fa3b
2009-10-21 12:46:18 -02:00
Ismael Carnales
5371d900a5
corrected the banner from scrapyctl webconsole module
2009-10-21 12:14:41 -02:00
Daniel Grana
0d637a6703
ensure meta refresh redirections use GET requests even if coming from POSTs
2009-10-21 11:35:36 -02:00
Pablo Hoffman
939e302b6e
make get_meta_refresh() function more robust and changed interface to return (int, str) as (interval, absolute_url)
2009-10-21 09:37:04 -02:00
