Alex Cepoi
fc405e98aa
improve mac os compatibility
...
Highlights:
* FifoDiskQueue: mixing buffered version of seek with unbuffered version
of read causes problems
* BSD's find does not default to current directory
* gdbm needs to be closed before it can reopen the same file
* skip PIL tests if jpeg support is not available
2012-12-01 16:39:58 +01:00
Pablo Hoffman
b9a96147ed
setup.py: use README.rst to populate long_description
2012-11-25 22:22:33 -02:00
Pablo Hoffman
39274a2457
doc: removed obsolete references to ClientForm
2012-11-23 19:06:47 -02:00
Pablo Hoffman
8ca2ee3d60
Merge pull request #196 from stav/master
...
The default storage backend is now DbmCacheStorage
2012-11-22 13:02:15 -08:00
stav
99f164fc87
correct docs for default storage backend
2012-11-22 14:05:47 -06:00
Pablo Hoffman
1f0d167037
doc: removed broken proxyhub link from FAQ
2012-11-22 15:10:26 -02:00
Pablo Hoffman
0a00e0fd63
Merge pull request #195 from kalessin/floatdelay
...
download delay in autothrottle was being casted as int, should be float
2012-11-15 11:18:45 -08:00
Martin Olveyra
421dba98a7
download delay in autothrottle was being casted as int, should be float
2012-11-15 18:41:32 +00:00
Pablo Hoffman
3a60d42072
removed unused import
2012-11-15 16:29:07 -02:00
Pablo Hoffman
5ebaa40ed2
changed SSL version to use from SSLv23 to TLSv1. closes #194 but needs more testing against counter-effects
2012-11-15 16:28:39 -02:00
Pablo Hoffman
cba3967667
Merge pull request #192 from vkrest/master
...
Fix broken doc links.
2012-11-12 13:37:52 -08:00
vkrest
ac4c66f351
Fix broken doc links.
...
All generated template files contain broken link to doc files.
2012-11-12 22:43:42 +02:00
Pablo Hoffman
14179a16a1
Merge pull request #190 from pedrofaustino/master
...
Adding support to 'nofollow' in the SGML link extractors
2012-11-12 10:09:30 -08:00
Pedro Faustino
3d0e962cbb
The Link object has a 'nofollow' attribute. Adding support in the SGML link extractors.
2012-11-10 16:25:41 +01:00
Pablo Hoffman
79a8bb6b42
Merge pull request #189 from coagulant/patch-1
...
Fixed docs typo in SpiderOpenCloseLogging example
2012-11-10 06:37:54 -08:00
Ilya Baryshev
097aea04a4
Fixed docs typo in SpiderOpenCloseLogging example
2012-11-10 12:24:53 +04:00
Daniel Graña
da7e414fe9
Add 0.16.2 release notes
...
Conflicts:
docs/news.rst
2012-11-09 13:03:04 -02:00
Pablo Hoffman
db21bccf9a
added 0.18 to release notes and mention spider contracts
2012-11-07 16:02:18 -02:00
Pablo Hoffman
c01f81ab44
Merge pull request #177 from alexcepoi/contracts_fixes
...
Improve feedback on spider contracts run
2012-11-07 09:40:21 -08:00
Pablo Hoffman
aa0e02dc54
added open_in_browser to debugging doc
2012-11-04 19:58:06 -02:00
Pablo Hoffman
7a7c5d1334
removed reference to global scrapy stats from settings doc
2012-11-03 17:05:01 -02:00
Pablo Hoffman
626662f0f3
Fix SpiderState bug in Windows platforms
...
The spider state file was not opened in binary mode.
2012-11-03 16:57:20 -02:00
Alex Cepoi
0350920f1d
scrapy contracts: python2.6 compat
2012-10-28 18:23:21 +01:00
Alex Cepoi
c6fad05756
scrapy contracts verbose option
2012-10-28 17:53:38 +01:00
Alex Cepoi
94629a1f08
proper unittest-like output for scrapy contracts
2012-10-27 23:26:40 +02:00
Daniel Graña
c0542838d3
update news file with 0.16.1 release notes
2012-10-26 18:53:59 -02:00
Pablo Hoffman
b01be8619d
fixed LogStats extension, which got broken after a wrong merge before the 0.16 release
2012-10-25 15:24:53 -02:00
Pablo Hoffman
e8e5a62c20
better backwards compatibility for scrapy.conf.settings
...
Althought this backwards compatibility is more complex, it avoid modules
from failing when importing scrapy.conf, if they are not run through
"scrapy" command (such as when running tests on scrapy projects code).
2012-10-25 11:41:40 -02:00
Pablo Hoffman
8f4c879b58
extended documentation on how to access crawler stats from extensions
2012-10-25 11:28:23 -02:00
Pablo Hoffman
17d958469f
removed .hgtags (no longer needed now that scrapy uses git)
2012-10-24 17:59:00 -02:00
Daniel Graña
a9616f387e
fix dashes under rst headers
2012-10-18 20:14:30 -02:00
Daniel Graña
4852c6102d
set release date for 0.16.0 in news
2012-10-18 20:07:36 -02:00
Daniel Graña
d4b8e2633e
bumped version to 0.17.0
2012-10-18 17:11:33 -02:00
Pablo Hoffman
c3e91ba73e
updated scrapy_bash_completion
2012-10-12 20:02:03 -02:00
Pablo Hoffman
e9cef3ae69
better description for scrapy check command
2012-10-12 20:01:43 -02:00
Pablo Hoffman
a0e2b6e3ff
simplified backwards compatibility of scrapy.conf
2012-10-12 17:28:02 -02:00
Pablo Hoffman
9d7106219a
added deprecation warning when importing scrapy.conf module
2012-10-12 17:24:25 -02:00
Pablo Hoffman
b038bb8504
minor message update: spider stats -> Scrapy stats
2012-10-12 16:53:44 -02:00
Artem Bogomyagkov
1d5967cee6
httpcompression middleware improvement
2012-10-10 20:20:13 +03:00
Pablo Hoffman
1a905d62f5
removed scrapy.log.started attribute, and avoid checking if log has already been started (since it should be called once anyway)
2012-10-09 16:05:19 -02:00
Pablo Hoffman
1f89eb59fe
fixed doc reference to topics-contracts
2012-10-09 16:02:12 -02:00
Pablo Hoffman
9e4c6aba2f
updated release notes with new stats recently added
2012-10-09 12:50:42 -02:00
Pablo Hoffman
bd3a109299
Track number of log messages via Scrapy stats
...
Log messages count are grouped per LEVEL like:
'log_count/DEBUG': 8,
'log_count/ERROR': 1,
'log_count/INFO': 4,
This required passing the Crawler object to scrapy logging facility, so
the scrapy.log module underwent a few changes which should be backwards
compatible (or as backwards compatible as they could be).
2012-10-09 12:41:31 -02:00
Pablo Hoffman
1ae1a3ea52
don't reset scrapy stats when spider is opened
2012-10-09 12:40:51 -02:00
Pablo Hoffman
ceeea27d21
added response_received_count to scrapy stats
2012-10-09 12:39:30 -02:00
Pablo Hoffman
452595790e
fix test-scrapyd.sh after changes made to website
2012-10-08 15:30:19 -02:00
Daniel Graña
3c3c0c1f3f
notify #scrapy room of travis build results
2012-10-06 15:16:58 -03:00
Daniel Graña
05ce20dab3
more cases where response.meta is prefered
2012-10-06 15:16:58 -03:00
Pablo Hoffman
1e1cc76fb7
Merge pull request #179 from artem-dev/errback_fix
...
This fixes the case where failure.request wasn't available when the failure was generated in a spider middleware (for example, the HttpError middleware).
2012-10-06 08:47:26 -07:00
Artem Bogomyagkov
0256a34f97
fix for request's errback, allows to get failed requests correctly
2012-10-05 20:44:20 +03:00
