Pablo Hoffman
b7eeeff410
get rid of assertDictEqual (since it's python 2.7+ only)
2013-01-17 13:18:23 -02:00
Pablo Hoffman
c31441a273
revert default HTTP cache policy to dummy (instead of RFC2616)
2013-01-17 13:08:29 -02:00
Daniel Graña
897195186a
document new FormRequest parameter named formxpath that matches forms using xpath
2013-01-08 18:36:20 -02:00
Daniel Graña
7527ef97ba
Merge pull request #185 from notsobad/master
...
Added xpath support in FormRequest.from_response
2013-01-08 12:34:06 -08:00
Daniel Graña
75563b3f00
Add list of supported and missing RFC2616 caching features
2013-01-08 18:16:44 -02:00
Daniel Graña
3cbc4d0b94
django is an optional_features, its imports must not fail
2013-01-08 17:56:46 -02:00
Daniel Graña
d8a760bf57
Merge branch 'http-cache-middleware'
...
Conflicts:
scrapy/contrib/downloadermiddleware/httpcache.py
scrapy/contrib/httpcache.py
scrapy/tests/test_downloadermiddleware_httpcache.py
2013-01-08 17:34:48 -02:00
Daniel Graña
864a7aef87
More httpcache updates
...
* Change default cache policy to RFC2616
* Update HttpCacheMiddleware documentation
* Move policies to scrapy.contrib.httpcache
* remove a lint error for .has_key() usage in DBM storage backend
2013-01-08 17:26:32 -02:00
Daniel Graña
487299e068
TakeFirst doc says it returns first non-null/non-empty value, zero is a valid value. closes #59
2013-01-08 15:47:33 -02:00
Daniel Graña
672d09ea2e
add meta-refresh changes to release notes
2013-01-08 12:30:36 -02:00
Daniel Graña
9527c5819a
pep8ize settings
2013-01-08 11:48:36 -02:00
Daniel Graña
defc4f89b5
update metarefresh settings
2013-01-08 11:41:19 -02:00
Daniel Graña
6a2b23883a
Add MetaRefreshMiddleware docs
2013-01-08 11:25:38 -02:00
Daniel Graña
076ba40404
update DOWNLOADER_MIDDLEWARES_BASE setting documentation
2013-01-08 10:50:27 -02:00
Daniel Graña
71db7f1b25
Split redirection into status and metarefresh middlewares, also changes httpcompression priority. closes #78
2013-01-08 09:59:38 -02:00
Rolando Espinoza La fuente
fe5d0ce2e0
tests: added downloader middleware manager integration tests for gzipped redirection.
2013-01-08 09:59:38 -02:00
Pablo Hoffman
227a1d666b
add doc about disabling an extension. refs #132
2013-01-07 13:16:19 -02:00
Pedro Faustino
5d3a4d755f
Update downloader middleware documentation
2013-01-06 18:53:14 +00:00
Pedro Faustino
59dc71f394
Merge branch 'http-cache-middleware', remote-tracking branch 'dangra/http-cache-middleware' into http-cache-middleware
2013-01-06 17:53:25 +00:00
Pablo Hoffman
7f990a4af2
Merge pull request #221 from emschorsch/patch-1
...
Proposed Changes to DjangoItem documentation
2013-01-04 14:17:54 -08:00
Emanuel Schorsch
f9b130da12
Proposed Changes
...
I was very confused as to how you actually import DjangoItem.
I searched extensively on the internet looking for actual code so I could see how it worked.
I finally found http://blog.just2us.com/2012/07/setting-up-django-with-scrapy/ . It is much easier to understand with full files instead of code fragments.
I also edited where it says "we can see that the model is already saved" as I don't see how it's already saved.
2013-01-04 15:59:04 -05:00
Pablo Hoffman
acb7bad1ff
Merge pull request #218 from Mimino666/django-item-validation
...
Django item validation
2013-01-04 10:35:20 -08:00
Daniel Graña
3f03a2ca50
requests with no-cache set must force revalidation of cached responses
2013-01-04 04:20:45 -02:00
Daniel Graña
cdecc760ee
default httpcache to rfc2616 policy and improve storage and policy tests
2013-01-04 03:43:25 -02:00
Daniel Graña
0a5586fafd
move FilesystemCacheStorage to scrapy.contrib.httpcache
2013-01-03 09:23:18 -02:00
Pablo Hoffman
9f003a73da
Merge pull request #217 from Mimino666/item-processing-errormsg
...
Fixed error message formatting.
2013-01-02 21:04:40 -08:00
Michal Danilak
2cfbd13c17
Added "exclude" parameter testing to unittests.
2013-01-03 02:02:03 +01:00
Michal Danilak
035d1e99a9
Added model validation to DjangoItem.
2013-01-03 01:52:53 +01:00
Michal Danilak
8ea89b277c
Fixed error message formatting.
...
log.err() doesn't support cool formatting and when error occured, the message was:
"ERROR: Error processing %(item)s"
2013-01-02 22:55:04 +01:00
Pablo Hoffman
ea0967562e
Merge pull request #216 from Mimino666/shell-spider-option
...
Added --spider option to "shell" command.
2013-01-02 09:15:15 -08:00
Daniel Graña
a6ef76ed88
lint and improve images pipeline error logging
2013-01-02 14:14:37 -02:00
Michal Danilak
ea68250d77
Added --spider option to "shell" command.
2013-01-02 17:13:37 +01:00
Pablo Hoffman
e9c5b76242
Merge pull request #215 from kuyan/patch-1
...
Fixed typo
2013-01-02 06:21:32 -08:00
Natan L
d572f8945e
Fixed typo
...
'persitent' --> 'persistent'
2012-12-31 11:14:01 -08:00
Pablo Hoffman
1aa25cdff2
Merge pull request #214 from tonal/log-level-dropped-item
...
Make LogFormatter return the log level (and require it)
2012-12-29 19:21:54 -08:00
Alexandr N Zamaraev (aka tonal)
71b071ffeb
Log level return from LogFormatter methods
2012-12-29 11:43:41 +07:00
Pedro Faustino
cf5f0203b7
Instead of extending from HttpCachePolicy, following the same approach used for storage selection
2012-12-28 16:11:47 +01:00
Pablo Hoffman
ba257db6e2
Merge pull request #213 from tonal/forget-import
...
Remove firget imports
2012-12-28 06:42:40 -08:00
Pedro Faustino
492831fc6f
Merge branch 'master' of git://github.com/scrapy/scrapy into http-cache-middleware
2012-12-28 15:27:45 +01:00
Pedro Faustino
3e31d06872
Implement single HTTP cache policy
2012-12-28 13:28:35 +01:00
Pedro Faustino
63d0b9f8c8
Remove plural from stat key.
2012-12-28 11:06:04 +01:00
Alexandr N Zamaraev (aka tonal)
3d397de0e9
Possible set log-level in LogFormatter.dropped and LogFormatter.scraped
2012-12-28 13:40:27 +07:00
Alexandr N Zamaraev (aka tonal)
21de68b766
Remove firget imports
2012-12-28 13:01:39 +07:00
Hasnain Lakhani
93a1102189
Implemented policies for HTTP Cache
2012-12-26 16:29:48 -08:00
Pablo Hoffman
51b8feb4ce
fixed doc typos
2012-12-26 16:16:53 -02:00
Pablo Hoffman
1e2ee76df2
add documentation topics: Broad Crawls & Common Practies
2012-12-26 14:02:13 -02:00
Pedro Faustino
fdaa35f6e8
Updated the downloader middleware documentation to reflect changes introduced by the support for real HTTP caching.
2012-12-24 19:37:53 +01:00
Pedro Faustino
b2d3f4dd1b
Add cache storage support for real HTTP caching. Add real HTTP caching unit tests for both middleware and cache storage.
2012-12-24 16:15:04 +01:00
Pedro Faustino
0e435fb5f9
Add middleware support for the 'no-store' Cache-Control directive.
2012-12-24 13:27:13 +01:00
Pedro Faustino
bb55f39aed
Add middleware support for real HTTP caching. Add httpcache setting to allow either real or dummy HTTP caching (for backwards compatibility it's set to use dummy cache by default).
2012-12-24 01:52:45 +01:00
