BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-03-04 11:37:37 +00:00
Code Releases Wiki Activity
4,677 Commits 39 Branches 119 Tags
Commit Graph

4677 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
olveyra
456b6f2ef5 added a python native classes item exporter 2013-08-13 13:48:28 +00:00
Daniel Graña
b43b5f575e adjust http11 pool size to per-domain concurrency 2013-08-12 22:43:40 -03:00
Pablo Hoffman
892386ee01 tox.ini: disable sitepackages on windows, as a compiler is often not available 2013-08-12 18:59:41 -03:00
Daniel Graña
ed5b9068d2 fix contributters list format 2013-08-12 11:26:16 -03:00
Daniel Graña
80e25c5980 bumped version to 0.19.0 0.19.0 2013-08-09 19:09:47 -03:00
Daniel Graña
a6693c9a5c updated release notes and bumped version to 0.18.0 0.18.0 2013-08-09 19:02:28 -03:00
Pablo Hoffman
4dc76e7cca fixed scrapy.utils.gz.gunzip() broken after changes from Python 2.7.3 to 2.7.4 2013-08-09 18:20:04 -03:00
Daniel Graña
29a12af1b0 Merge pull request #361 from hmsimha/master 
Doc fixes: Minor typo (was fixed on 0.16 branch so I figured it could be on master as well) and an error in the overview doc.
 2013-08-05 09:10:22 -07:00
Stefan Koch
915d7cf247 also fetch alternate URLs from sitemaps, see #360 2013-08-04 16:36:04 +02:00
Hart
c00c4d7148 correction to description of example XPath retrieval in overview doc 2013-08-03 17:08:58 -07:00
Hart
0ad01c34d4 fixed typo to parallel fix on 0.16 branch 2013-08-03 17:06:10 -07:00
Nicolás Alejandro Ramírez Quiros
c2a4046f14 Merge pull request #359 from rocioar/master 
Added COMPRESSION_ENABLED setting
 2013-08-01 10:43:47 -07:00
Rocio Aramberri
d227d530f6 Added COMPRESSION_ENABLED setting to enable or disable the HttpCompressionMiddleware 
Added COMPRESSION_ENABLE setting to docs

Added COMPRESSION_ENABLED setting to default settings
 2013-08-01 11:31:28 -03:00
arijitchakraborty
fb770852e8 Skipping cookie retrieval for non http requests 2013-07-22 11:33:26 -03:00
Mikhail Korobov
e1f0d0d45d Merge pull request #352 from arijitchakraborty/bugfix/cookie_retrieval_for_hosts_with_port 
Fixes for bug - Cookie retrieval for hosts with port + unittests for the...
 2013-07-18 09:21:13 -07:00
arijitchakraborty
66ff34cf05 improving hostname extraction 2013-07-18 20:51:56 +05:30
arijitchakraborty
ebc136ddb0 Fixes for bug - Cookie retrieval for hosts with port + unittests for the fix 2013-07-18 19:42:58 +05:30
Daniel Graña
d7e11082f2 it is not possible to enforce an upper limit when latency is out of control 2013-07-18 11:07:42 -03:00
Daniel Graña
3eb5d5e931 Merge pull request #330 from nramirezuy/spider-from_crawler 
use from_crawler method if available in spiders
 2013-07-16 12:42:31 -07:00
Daniel Graña
d14541c5f1 Merge pull request #205 from joehillen/master 
PyPy support
 2013-07-16 12:34:17 -07:00
Daniel Graña
d51f1ff33d Merge pull request #350 from dellis23/master 
This pull request fixes #349
 2013-07-16 12:33:21 -07:00
Daniel Graña
d74b60051d take in count response latencies when testing download delays 2013-07-16 16:22:29 -03:00
Dan
1ca31244b0 Fixed ordering of super argument call. 2013-07-16 14:50:10 -04:00
Dan
e12b689c4f Updated documentation of spider arguments to include required super call. 2013-07-16 14:26:53 -04:00
Daniel Graña
891fa980d2 Merge pull request #341 from kmike/test-running 
Improved tox test running & fixed django tests
 2013-07-16 11:16:43 -07:00
Daniel Graña
66aa1331fc Twisted pre 10.2 (lucid) can not setup the required ftp server for tests 2013-07-16 00:49:35 -03:00
Daniel Graña
26c7ec7d9f Merge pull request #329 from kalessin/ftphandler 
added ftp handler
 2013-07-15 16:39:46 -07:00
Daniel Graña
f8c105d5c5 Merge branch 'gh-347'. thanks @chrisboo 2013-07-15 12:28:24 -03:00
Daniel Graña
9ff82d6695 use lxml parser with recover option to parse invalid sitemaps. closes #347 
thanks chrissboo for reporting the issue and proposing a fix.
 2013-07-15 12:26:16 -03:00
Mikhail Korobov
1a1c93fafe tiny FormRequest doc fix 2013-07-15 15:47:34 +06:00
Daniel Graña
aa7b529975 do not log Starting/Stopping factory messages 2013-07-10 12:35:15 -03:00
Mikhail Korobov
3fe2a32683 handle GET parameters for AJAX crawlable URLs: 2013-07-09 01:58:50 +06:00
Mikhail Korobov
8c555fd467 Enable Travis CI for pull requests and feature branches. Fix GH-340. 2013-07-09 01:50:52 +06:00
Mikhail Korobov
0d9f7843ba allow passing arguments to runtests script via tox 2013-07-09 00:41:13 +06:00
Mikhail Korobov
f29e5d0ca1 reuse runtests.sh and runtests.bat scripts in tox.ini 2013-07-09 00:38:00 +06:00
Mikhail Korobov
3a6dff3090 make tox test running more consistent with bin/runtests.sh script 2013-07-09 00:30:39 +06:00
Mikhail Korobov
2105ec58eb add django to test requirements and fix test_djangoitem for modern django versions 2013-07-09 00:28:39 +06:00
Mikhail Korobov
b98e80cbb8 use tox deps instead of "manual" pip install; add lucid and precise environments to tox.ini 2013-07-09 00:26:59 +06:00
Mikhail Korobov
35fbec2e63 allow passing arguments to tox 
e.g. "tox -- scrapy.tests.test_contrib_loader" runs test_contrib_loader for Python 2.6 and 2.7
 2013-07-08 22:51:03 +06:00
Alex Cepoi
f3d8b0802a disable logging for scrapy check 2013-07-08 17:40:03 +02:00
Alex Cepoi
919d7c4062 command with multiple crawlers, fix check command 2013-07-08 17:00:43 +02:00
Alex Cepoi
45d6d2044c fix a possible error in contracts post hook 2013-07-08 16:58:29 +02:00
Daniel Graña
6db259c075 Merge pull request #338 from kmike/patch-3 
DownloaderMiddleware docs fix
 2013-07-08 07:12:42 -07:00
Mikhail Korobov
ac2fadf3ab DownloaderMiddleware.process_response docs fix 
"returns an exception" -> "raises an exception"
 2013-07-08 19:41:58 +06:00
Pablo Hoffman
9a13a0fe74 Merge pull request #337 from kmike/patch-2 
improve docs for DownloaderMiddleware.process_response
 2013-07-08 06:21:49 -07:00
Mikhail Korobov
39e5da5f66 improve docs for DownloaderMiddleware.process_response 2013-07-08 19:17:29 +06:00
Pablo Hoffman
163592ee40 Merge pull request #334 from nramirezuy/feedexport-remove_multispider 
feedexport removed multispider support
 2013-07-03 14:26:49 -07:00
nramirezuy
1fc2bc3489 feedexport removed multispider support 2013-07-03 18:24:18 -03:00
Pablo Hoffman
42da57c448 Merge pull request #333 from nramirezuy/mediapipe-remove_multispider 
pipeline media removed multispider support
 2013-07-03 14:20:41 -07:00
Pablo Hoffman
6533115bbb Merge pull request #332 from nramirezuy/offsite-remove_multispider 
offsite multispider support removed
 2013-07-03 14:20:06 -07:00