Pablo Hoffman
4ac0eea55d
Merge pull request #316 from nramirezuy/closespider/errorcount
...
standarizes close_spider to use signals
2013-05-31 12:20:58 -07:00
nramirezuy
b61a4e719c
standarizes close_spider to use signals
2013-05-31 15:54:17 -03:00
Daniel Graña
b4fca90bba
merge 0.16.5 release notes
2013-05-30 18:49:00 -03:00
Daniel Graña
58dc16f48d
obey request method when scrapy deploy is redirected to a new endpoint
2013-05-30 18:34:59 -03:00
Pablo Hoffman
0effe139e5
Merge pull request #311 from nramirezuy/logstats-6017
...
removed multispider support from logstats
2013-05-30 08:47:33 -07:00
nramirezuy
14e382ae48
test_closespider fixed for python 2.6
2013-05-30 10:57:06 -03:00
Pablo Hoffman
461eace4af
Merge pull request #313 from nramirezuy/closespider-6017
...
drop multi-spider support from CloseSpider extension
2013-05-30 06:34:52 -07:00
nramirezuy
971ef5c78e
removed multispider support, test added
2013-05-28 17:39:37 -03:00
Pablo Hoffman
2bb2ba66f6
simplify scrapy.tests.mockserver
2013-05-28 15:12:38 -03:00
Pablo Hoffman
f35266e6a5
use os.pathsep
2013-05-28 15:10:05 -03:00
nramirezuy
96b8fb69be
removed multispider support from logstats
2013-05-28 14:45:02 -03:00
Pablo Hoffman
98aa3efea7
Merge pull request #312 from nramirezuy/cmd/bech/test
...
added test for bench command
2013-05-28 08:46:31 -07:00
nramirezuy
1ed5643eb2
added test for bench command
2013-05-28 12:42:50 -03:00
Pablo Hoffman
a4b5bfbb5e
Merge pull request #309 from amferraz/patch-1
...
Add FAQ entry referencing Request.meta usage
2013-05-27 09:07:48 -07:00
cacovsky
8007762890
Add FAQ entry referencing Request.meta usage
2013-05-27 13:02:17 -03:00
Nicolás Alejandro Ramírez Quiros
7874eef7af
Merge pull request #307 from DeaconDesperado/raise-format
...
Raise a usage error when an invalid or unrecognized output format is entered on Command line
2013-05-23 12:50:17 -07:00
Mark Grey
485a954571
raise on unrecognized format
2013-05-23 15:11:45 -04:00
Pablo Hoffman
845c64b89d
add benchmarking to 0.18 release notes
2013-05-17 10:38:42 -03:00
Pablo Hoffman
ca12886acb
update copyright notes
2013-05-16 15:05:52 -03:00
Pablo Hoffman
8e49fed918
minor improvements to benchmarking doc
2013-05-16 13:23:13 -03:00
Pablo Hoffman
76087e336a
add scrapy bench command for benchmarking, with documentation
2013-05-16 13:15:25 -03:00
Pablo Hoffman
5c40741d65
added context manager for mock server, moved test spiders into a separate module (scrapy.tests.spiders)
2013-05-16 13:01:02 -03:00
Pablo Hoffman
214bcdf3be
Merge pull request #301 from tpeng/fix_telnet_doc
...
change manager in telnet variable to crawler
2013-05-08 07:00:52 -07:00
tpeng
edbe525d19
change manager in telnet variable to crawler
2013-05-08 11:58:15 +08:00
Pablo Hoffman
99f8d5733a
added mock tests for retries (more to come)
2013-05-06 20:04:15 -03:00
Pablo Hoffman
a7b7c89216
improved logging of downloader errors, when they contain tracebacks
2013-05-06 20:03:54 -03:00
Pablo Hoffman
db195ee4df
added more methods to mockserver
2013-05-06 20:03:17 -03:00
Pablo Hoffman
c2561ca160
use get_testenv() shortcut instead of get_pythonpath()
2013-05-06 17:35:09 -03:00
Pablo Hoffman
14e7121674
make sure the mockserver forked from tests uses the current installation of scrapy, not the system one
2013-05-06 17:27:59 -03:00
Pablo Hoffman
cf4d4bc094
added mock server test for DOWNLOAD_TIMEOUT
2013-05-06 14:47:24 -03:00
Daniel Graña
495acba223
agents requires an instance of contextFactory
2013-05-06 12:56:13 -03:00
Daniel Graña
3e62ce35a0
cleanup http connection pool on engine stop
2013-05-06 12:56:13 -03:00
Daniel Graña
334ad71a2e
adapt for singletons removals
2013-05-06 12:56:13 -03:00
Daniel Graña
ba6545555f
empty bodies does not require a body producer
2013-05-06 12:56:13 -03:00
Daniel Graña
db232da068
close pool connections before finishing tests
2013-05-06 12:56:13 -03:00
Daniel Graña
b34606030d
remove duplicate context factory handling for non-ssl support in http1.0
2013-05-06 12:56:13 -03:00
Daniel Graña
0a26170086
move ssl context factory to its own module and implement a non-ssl version that warns about pyopenssl support
2013-05-06 12:52:58 -03:00
Daniel Graña
ab3407289c
enable persistent connections
2013-05-06 12:50:14 -03:00
Daniel Graña
e4fe7c63b0
add http connection pool and custom ssl context factory
2013-05-06 12:50:13 -03:00
Daniel Graña
a7a354f982
http11 cleanup
2013-05-06 12:45:27 -03:00
paul
ef03603869
Restore handling of HTTPS
2013-05-06 12:45:27 -03:00
paul
46341d5275
Renamed downloader to Http11DownloadHandler and some refactoring
...
Only for HTTP, not HTTPS
Test on expected body length instead of request method (HEAD case)
2013-05-06 12:45:27 -03:00
paul
4018d25a9b
Use twisted.web.client.Agent for download requests (use of HTTP/1.1)
...
Adds http11.HttpDownloadHandler in scrapy.core.downloader.handlers
2013-05-06 12:45:27 -03:00
Pablo Hoffman
66311db23e
mention crawlera in best practices, as a way to deal with bans
2013-05-04 18:20:23 -03:00
Pablo Hoffman
e1f4144391
removed obsolete (and broken) downloader method: is_idle()
2013-04-29 13:04:40 -03:00
Daniel Graña
2a1a4477d3
Merge pull request #297 from scrapy/downloader-gc
...
Add garbage collector to downloader
2013-04-29 07:38:27 -07:00
Pablo Hoffman
af5c13fa14
Add garbage collector to downloader
...
This fixes a couple of issues:
- reactor callLater leaks when using download delay (test was
re-enabled)
- downloader slot leaking on broad crawls (slots were created but never
removed)
2013-04-29 10:16:30 -03:00
Pablo Hoffman
36ee36000e
bind mockserver to 0.0.0.0, to listen on all 127.* range (useful for testing broad crawls)
2013-04-27 04:17:19 -03:00
Pablo Hoffman
9361c89573
remove scrapyd doc, as it was moved to its own repo
2013-04-27 04:15:42 -03:00
Daniel Graña
5ba2b60a4b
fix broken doctests
2013-04-25 11:56:56 -03:00
