Daniel Graña
cdecc760ee
default httpcache to rfc2616 policy and improve storage and policy tests
2013-01-04 03:43:25 -02:00
Daniel Graña
0a5586fafd
move FilesystemCacheStorage to scrapy.contrib.httpcache
2013-01-03 09:23:18 -02:00
Pablo Hoffman
9f003a73da
Merge pull request #217 from Mimino666/item-processing-errormsg
...
Fixed error message formatting.
2013-01-02 21:04:40 -08:00
Michal Danilak
2cfbd13c17
Added "exclude" parameter testing to unittests.
2013-01-03 02:02:03 +01:00
Michal Danilak
035d1e99a9
Added model validation to DjangoItem.
2013-01-03 01:52:53 +01:00
Michal Danilak
8ea89b277c
Fixed error message formatting.
...
log.err() doesn't support cool formatting and when error occured, the message was:
"ERROR: Error processing %(item)s"
2013-01-02 22:55:04 +01:00
Pablo Hoffman
ea0967562e
Merge pull request #216 from Mimino666/shell-spider-option
...
Added --spider option to "shell" command.
2013-01-02 09:15:15 -08:00
Daniel Graña
a6ef76ed88
lint and improve images pipeline error logging
2013-01-02 14:14:37 -02:00
Michal Danilak
ea68250d77
Added --spider option to "shell" command.
2013-01-02 17:13:37 +01:00
Pablo Hoffman
e9c5b76242
Merge pull request #215 from kuyan/patch-1
...
Fixed typo
2013-01-02 06:21:32 -08:00
Natan L
d572f8945e
Fixed typo
...
'persitent' --> 'persistent'
2012-12-31 11:14:01 -08:00
Pablo Hoffman
1aa25cdff2
Merge pull request #214 from tonal/log-level-dropped-item
...
Make LogFormatter return the log level (and require it)
2012-12-29 19:21:54 -08:00
Alexandr N Zamaraev (aka tonal)
71b071ffeb
Log level return from LogFormatter methods
2012-12-29 11:43:41 +07:00
Pedro Faustino
cf5f0203b7
Instead of extending from HttpCachePolicy, following the same approach used for storage selection
2012-12-28 16:11:47 +01:00
Pablo Hoffman
ba257db6e2
Merge pull request #213 from tonal/forget-import
...
Remove firget imports
2012-12-28 06:42:40 -08:00
Pedro Faustino
492831fc6f
Merge branch 'master' of git://github.com/scrapy/scrapy into http-cache-middleware
2012-12-28 15:27:45 +01:00
Pedro Faustino
3e31d06872
Implement single HTTP cache policy
2012-12-28 13:28:35 +01:00
Pedro Faustino
63d0b9f8c8
Remove plural from stat key.
2012-12-28 11:06:04 +01:00
Alexandr N Zamaraev (aka tonal)
3d397de0e9
Possible set log-level in LogFormatter.dropped and LogFormatter.scraped
2012-12-28 13:40:27 +07:00
Alexandr N Zamaraev (aka tonal)
21de68b766
Remove firget imports
2012-12-28 13:01:39 +07:00
Hasnain Lakhani
93a1102189
Implemented policies for HTTP Cache
2012-12-26 16:29:48 -08:00
Pablo Hoffman
51b8feb4ce
fixed doc typos
2012-12-26 16:16:53 -02:00
Pablo Hoffman
1e2ee76df2
add documentation topics: Broad Crawls & Common Practies
2012-12-26 14:02:13 -02:00
Pedro Faustino
fdaa35f6e8
Updated the downloader middleware documentation to reflect changes introduced by the support for real HTTP caching.
2012-12-24 19:37:53 +01:00
Pedro Faustino
b2d3f4dd1b
Add cache storage support for real HTTP caching. Add real HTTP caching unit tests for both middleware and cache storage.
2012-12-24 16:15:04 +01:00
Pedro Faustino
0e435fb5f9
Add middleware support for the 'no-store' Cache-Control directive.
2012-12-24 13:27:13 +01:00
Pedro Faustino
bb55f39aed
Add middleware support for real HTTP caching. Add httpcache setting to allow either real or dummy HTTP caching (for backwards compatibility it's set to use dummy cache by default).
2012-12-24 01:52:45 +01:00
Pedro Faustino
e396509f5b
Removing plural from httpcache stats' value names.
2012-12-24 00:24:27 +01:00
Pablo Hoffman
dabb064293
fix bug in scrapy parse command when spider is not specified explicitly. closes #209
2012-12-20 11:59:07 -02:00
Pablo Hoffman
12475fccbe
Merge pull request #206 from dangra/downloader-enhancements
...
AutoThrottle and Downloader enhancements
2012-12-17 10:11:46 -08:00
Daniel Graña
22559833e7
core: drop download inactive slots and get slot key from meta
2012-12-17 16:09:01 -02:00
Daniel Graña
eb2d87259f
core: move download slot assignment post middleware evaluation
2012-12-17 16:09:01 -02:00
Daniel Graña
d7daf836d5
Altering delay is enough to auto throttle downloads
2012-12-17 16:08:49 -02:00
Daniel Graña
85c9881724
fix pylint and pep8 warnings
2012-12-14 12:27:53 -02:00
Pablo Hoffman
0bc5e4f9ec
Merge pull request #204 from LuanP/patch-1
...
Update docs/topics/commands.rst
2012-12-10 16:19:25 -08:00
Luan
5582ea28ec
Update docs/topics/commands.rst
...
A short change.
2012-12-10 15:16:02 -02:00
Daniel Graña
0cc138c010
Add 0.16.3 release notes
2012-12-07 18:55:15 -02:00
Daniel Graña
e676ad31d0
Support sending requests trough multiples slots in QPS exmaple spider
2012-12-05 17:06:38 -02:00
Daniel Graña
fd58de493c
Remove concurrency limitation when using download delays and still ensure inter-request delays are enforced
2012-12-05 15:08:34 -02:00
Daniel Graña
dcc0a540c0
QPS bench server and a useful spider spider to generate requests
2012-12-05 15:08:34 -02:00
Daniel Graña
a958fb11f0
Merge pull request #202 from saxicek/master
...
Image pipeline error improvement
2012-12-03 03:06:47 -08:00
Libor Nenadál
5f899aa4ec
add error details when image pipeline fails
2012-12-02 23:29:26 +01:00
Pablo Hoffman
9c9a18b3a3
Merge pull request #201 from alexcepoi/test-fixes-mac
...
improve mac os compatibility
2012-12-02 07:37:53 -08:00
Alex Cepoi
fc405e98aa
improve mac os compatibility
...
Highlights:
* FifoDiskQueue: mixing buffered version of seek with unbuffered version
of read causes problems
* BSD's find does not default to current directory
* gdbm needs to be closed before it can reopen the same file
* skip PIL tests if jpeg support is not available
2012-12-01 16:39:58 +01:00
Pablo Hoffman
b9a96147ed
setup.py: use README.rst to populate long_description
2012-11-25 22:22:33 -02:00
Pablo Hoffman
39274a2457
doc: removed obsolete references to ClientForm
2012-11-23 19:06:47 -02:00
Pablo Hoffman
8ca2ee3d60
Merge pull request #196 from stav/master
...
The default storage backend is now DbmCacheStorage
2012-11-22 13:02:15 -08:00
stav
99f164fc87
correct docs for default storage backend
2012-11-22 14:05:47 -06:00
Pablo Hoffman
1f0d167037
doc: removed broken proxyhub link from FAQ
2012-11-22 15:10:26 -02:00
Pablo Hoffman
0a00e0fd63
Merge pull request #195 from kalessin/floatdelay
...
download delay in autothrottle was being casted as int, should be float
2012-11-15 11:18:45 -08:00
