Daniel Graña
588a262b73
fix crawling tests under twisted pre 11.0.0
2013-08-27 14:05:37 -03:00
Daniel Graña
92826586d5
py26 can not format zero length fields {}
2013-08-27 12:12:50 -03:00
Daniel Graña
caa0f90263
test PotentiaDataLoss errors on unbound responses
2013-08-27 11:53:18 -03:00
Daniel Graña
6720acc1af
Treat responses without content-length or Transfer-Encoding as good responses
...
There is not a way to determine if responses without Content-Length or Transfer-Encoding are complete, this change treat them as good responses but flags them as "partial".
This is backout change only for this functionality of 3c64a989
2013-08-27 11:10:20 -03:00
Pablo Hoffman
da1f6d3144
remove unused imports and some assorted pylint-ing
2013-08-23 13:03:34 -03:00
Pablo Hoffman
19ff9ac4f9
url/body attributes of Request/Response objects are now immutable
2013-08-23 12:43:22 -03:00
Pablo Hoffman
86230c0ab8
added quantal & raring to support ubuntu releases
2013-08-22 21:49:55 -03:00
Lukasz Biedrycki
83bd151c57
backwards compatibility: image_key & image_downloaded
2013-08-22 15:13:59 +02:00
Lukasz Biedrycki
2b11e7da93
backwards compatibility: image_key
2013-08-22 14:44:04 +02:00
Lukasz Biedrycki
02d08722e7
typo
2013-08-22 14:33:28 +02:00
Lukasz Biedrycki
4c89d8b298
remarks
2013-08-22 12:26:56 +02:00
Daniel Graña
efb4f32d80
do no include ResponseFailed if http11 handler is not enabled
2013-08-21 19:04:12 -03:00
Daniel Graña
3c64a989ca
New HTTP client wraps connection losts in ResponseFailed exception. fix #373
2013-08-21 17:28:26 -03:00
Pablo Hoffman
071172cbd6
fix retry middleware which didn't retry certain connection errors after the upgrade to http1 client, closes GH-373
2013-08-21 14:32:33 -03:00
Pablo Hoffman
6e4ed8b004
Merge pull request #372 from kmike/xml-exporter-fix-for-2.7.4
...
fix XmlItemExporter in Python 2.7.4 and 2.7.5
2013-08-21 09:55:51 -07:00
Mikhail Korobov
a6e6ca06b9
fix XmlItemExporter in Python 2.7.4 and 2.7.5
2013-08-21 06:05:40 +06:00
Alex Cepoi
9c092536c5
remove redundant kwarg
2013-08-21 01:52:36 +02:00
Alex Cepoi
7bdeabc0ed
fix log intermixing in scrapy shell command
2013-08-21 01:42:03 +02:00
Alex Cepoi
902208ca58
fix some missing logs with new crawlerprocess
2013-08-20 14:02:55 +02:00
Lukasz Biedrycki
96077ccf48
typo
2013-08-20 11:29:55 +02:00
Alex Cepoi
a812def59e
port all scrapy commands to new CrawlerProcess
2013-08-19 23:40:23 +02:00
Alex Cepoi
a9a911fb09
default to multi crawler commands
...
compatibility for old single crawler process
deprecated command's `crawler` property
logs about settings only shown once
2013-08-19 22:43:29 +02:00
Mikhail Korobov
034ffae60f
Recommend Pillow instead of PIL. Closes GH-317.
2013-08-18 00:44:01 +06:00
Lukasz Biedrycki
45ff6ec28a
Test reorganization and new tests for Files and Images Pipelines, PEP8 changes in MediaPipeline
2013-08-16 18:47:56 +02:00
Lukasz Biedrycki
76ce8c5238
FilesPipeline which enalbes to download any files.
...
It has been extracted from ImagesPipelines.
ImagesPipeline is built on top of FilesPipeline and consist only with convert image and thumbnail generation logic.
2013-08-16 17:02:31 +02:00
Pablo Hoffman
f95b164d06
Merge pull request #367 from berendiwema/master
...
#327 - Support STARTTLS / SSL option in email sender
2013-08-14 07:17:30 -07:00
Berend Iwema
32b6364bcd
#327 - Support STARTTLS / SSL option in email sender
2013-08-14 12:59:01 +02:00
Pablo Hoffman
49952a45ae
add docstring to PythonItemExporter
2013-08-14 02:49:14 -03:00
Pablo Hoffman
c0b26e3d49
minor updates to 0.18 release notes
2013-08-14 01:39:44 -03:00
Daniel Graña
c6157d37e9
Merge pull request #366 from kalessin/exporter
...
added a python native classes item exporter
2013-08-13 06:55:27 -07:00
olveyra
456b6f2ef5
added a python native classes item exporter
2013-08-13 13:48:28 +00:00
Daniel Graña
b43b5f575e
adjust http11 pool size to per-domain concurrency
2013-08-12 22:43:40 -03:00
Pablo Hoffman
892386ee01
tox.ini: disable sitepackages on windows, as a compiler is often not available
2013-08-12 18:59:41 -03:00
Daniel Graña
ed5b9068d2
fix contributters list format
2013-08-12 11:26:16 -03:00
Daniel Graña
80e25c5980
bumped version to 0.19.0
2013-08-09 19:09:47 -03:00
Daniel Graña
a6693c9a5c
updated release notes and bumped version to 0.18.0
2013-08-09 19:02:28 -03:00
Pablo Hoffman
4dc76e7cca
fixed scrapy.utils.gz.gunzip() broken after changes from Python 2.7.3 to 2.7.4
2013-08-09 18:20:04 -03:00
Daniel Graña
29a12af1b0
Merge pull request #361 from hmsimha/master
...
Doc fixes: Minor typo (was fixed on 0.16 branch so I figured it could be on master as well) and an error in the overview doc.
2013-08-05 09:10:22 -07:00
Stefan Koch
915d7cf247
also fetch alternate URLs from sitemaps, see #360
2013-08-04 16:36:04 +02:00
Hart
c00c4d7148
correction to description of example XPath retrieval in overview doc
2013-08-03 17:08:58 -07:00
Hart
0ad01c34d4
fixed typo to parallel fix on 0.16 branch
2013-08-03 17:06:10 -07:00
Nicolás Alejandro Ramírez Quiros
c2a4046f14
Merge pull request #359 from rocioar/master
...
Added COMPRESSION_ENABLED setting
2013-08-01 10:43:47 -07:00
Rocio Aramberri
d227d530f6
Added COMPRESSION_ENABLED setting to enable or disable the HttpCompressionMiddleware
...
Added COMPRESSION_ENABLE setting to docs
Added COMPRESSION_ENABLED setting to default settings
2013-08-01 11:31:28 -03:00
arijitchakraborty
fb770852e8
Skipping cookie retrieval for non http requests
2013-07-22 11:33:26 -03:00
Mikhail Korobov
e1f0d0d45d
Merge pull request #352 from arijitchakraborty/bugfix/cookie_retrieval_for_hosts_with_port
...
Fixes for bug - Cookie retrieval for hosts with port + unittests for the...
2013-07-18 09:21:13 -07:00
arijitchakraborty
66ff34cf05
improving hostname extraction
2013-07-18 20:51:56 +05:30
arijitchakraborty
ebc136ddb0
Fixes for bug - Cookie retrieval for hosts with port + unittests for the fix
2013-07-18 19:42:58 +05:30
Daniel Graña
d7e11082f2
it is not possible to enforce an upper limit when latency is out of control
2013-07-18 11:07:42 -03:00
Daniel Graña
3eb5d5e931
Merge pull request #330 from nramirezuy/spider-from_crawler
...
use from_crawler method if available in spiders
2013-07-16 12:42:31 -07:00
Daniel Graña
d14541c5f1
Merge pull request #205 from joehillen/master
...
PyPy support
2013-07-16 12:34:17 -07:00
