BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-23 19:43:39 +00:00
Code Releases Wiki Activity
scrapy/tests/test_squeues.py

166 lines
4.6 KiB
Python
Raw Normal View History

Do not serialize unpickable objects (py3)
2018-01-16 16:14:35 -03:00
from sys import version_info
Fix queue serialization test on PyPy It is not affected by Twisted bug #7989 and is more permissive with pickling (especially with protocol=2).
2016-09-01 15:39:16 +03:00
import pickle
ported code to use queuelib
2013-04-23 17:48:09 -03:00
from queuelib.tests import test_queue as t
Rename scrapy/squeue.py to scrapy/squeues.py
2015-04-23 11:51:07 -03:00
from scrapy.squeues import MarshalFifoDiskQueue, MarshalLifoDiskQueue, PickleFifoDiskQueue, PickleLifoDiskQueue
persistent scheduler: use pickle (instead of marshal) as the default serialization format, to support serializing more objects out of the box. also removed __slots__ from Request/Response objects to make them serializable by default.
2011-09-01 14:27:29 -03:00
from scrapy.item import Item, Field
from scrapy.http import Request
Move scrapy/contrib/loader to scrapy/loader
2015-04-20 23:21:41 -03:00
from scrapy.loader import ItemLoader
Do not serialize unpickable objects (py3)
2018-01-16 16:14:35 -03:00
from scrapy.selector import Selector
added MarshalDiskQueue unittests
2011-08-05 20:32:22 -03:00
persistent scheduler: use pickle (instead of marshal) as the default serialization format, to support serializing more objects out of the box. also removed __slots__ from Request/Response objects to make them serializable by default.
2011-09-01 14:27:29 -03:00
class TestItem(Item):
name = Field()
Change function name so it does not mess up with pytest autodiscover
2014-04-03 05:58:15 -03:00
def _test_procesor(x):
persistent scheduler: use pickle (instead of marshal) as the default serialization format, to support serializing more objects out of the box. also removed __slots__ from Request/Response objects to make them serializable by default.
2011-09-01 14:27:29 -03:00
return x + x
class TestLoader(ItemLoader):
default_item_class = TestItem
Change function name so it does not mess up with pytest autodiscover
2014-04-03 05:58:15 -03:00
name_out = staticmethod(_test_procesor)
added MarshalDiskQueue unittests
2011-08-05 20:32:22 -03:00
Fix queue serialization test on PyPy It is not affected by Twisted bug #7989 and is more permissive with pickling (especially with protocol=2).
2016-09-01 15:39:16 +03:00
def nonserializable_object_test(self):
Do not serialize unpickable objects (py3)
2018-01-16 16:14:35 -03:00
q = self.queue()
Fix queue serialization test on PyPy It is not affected by Twisted bug #7989 and is more permissive with pickling (especially with protocol=2).
2016-09-01 15:39:16 +03:00
try:
pickle.dumps(lambda x: x)
except Exception:
# Trigger Twisted bug #7989
import twisted.persisted.styles # NOQA
self.assertRaises(ValueError, q.push, lambda x: x)
else:
# Use a different unpickleable object
class A(object): pass
a = A()
a.__reduce__ = a.__reduce_ex__ = None
self.assertRaises(ValueError, q.push, a)
Do not serialize unpickable objects (py3)
2018-01-16 16:14:35 -03:00
if version_info.major == 3 and version_info.minor >= 6:
# Selectors should fail (lxml.html.HtmlElement objects can't be pickled)
sel = Selector(text='<html><body><p>some text</p></body></html>')
self.assertRaises(ValueError, q.push, sel)
Fix queue serialization test on PyPy It is not affected by Twisted bug #7989 and is more permissive with pickling (especially with protocol=2).
2016-09-01 15:39:16 +03:00
Some changes to persistent scheduler after some initial usage feedback: * added LIFO queues, in addition to the original FIFO queues * use LIFO queues (instead of FIFO queues) by default, since they resemble DFO better which is a more convenient crawling order for most cases * do not adjust the priority based on depth by default (DEPTH_PRIORITY = 0) If someone does need to use strict BFO order, it can be by done by setting: DEPTH_PRIORITY = 1 SCHEDULER_DISK_QUEUE = 'scrapy.squeue.PickleFifoDiskQueue' SCHEDULER_MEMORY_QUEUE = 'scrapy.squeue.FifoMemoryQueue'
2011-09-23 13:03:07 -03:00
class MarshalFifoDiskQueueTest(t.FifoDiskQueueTest):
added MarshalDiskQueue unittests
2011-08-05 20:32:22 -03:00
chunksize = 100000
def queue(self):
fix scrapy squeue tests after recent changes to queuelib
2015-09-09 15:55:55 -03:00
return MarshalFifoDiskQueue(self.qpath, chunksize=self.chunksize)
added MarshalDiskQueue unittests
2011-08-05 20:32:22 -03:00
def test_serialize(self):
q = self.queue()
q.push('a')
q.push(123)
q.push({'a': 'dict'})
self.assertEqual(q.pop(), 'a')
self.assertEqual(q.pop(), 123)
self.assertEqual(q.pop(), {'a': 'dict'})
Fix queue serialization test on PyPy It is not affected by Twisted bug #7989 and is more permissive with pickling (especially with protocol=2).
2016-09-01 15:39:16 +03:00
test_nonserializable_object = nonserializable_object_test
add setting to enable logging when unserializable requests are found
2011-09-01 19:40:44 -03:00
Some changes to persistent scheduler after some initial usage feedback: * added LIFO queues, in addition to the original FIFO queues * use LIFO queues (instead of FIFO queues) by default, since they resemble DFO better which is a more convenient crawling order for most cases * do not adjust the priority based on depth by default (DEPTH_PRIORITY = 0) If someone does need to use strict BFO order, it can be by done by setting: DEPTH_PRIORITY = 1 SCHEDULER_DISK_QUEUE = 'scrapy.squeue.PickleFifoDiskQueue' SCHEDULER_MEMORY_QUEUE = 'scrapy.squeue.FifoMemoryQueue'
2011-09-23 13:03:07 -03:00
class ChunkSize1MarshalFifoDiskQueueTest(MarshalFifoDiskQueueTest):
added MarshalDiskQueue unittests
2011-08-05 20:32:22 -03:00
chunksize = 1
Some changes to persistent scheduler after some initial usage feedback: * added LIFO queues, in addition to the original FIFO queues * use LIFO queues (instead of FIFO queues) by default, since they resemble DFO better which is a more convenient crawling order for most cases * do not adjust the priority based on depth by default (DEPTH_PRIORITY = 0) If someone does need to use strict BFO order, it can be by done by setting: DEPTH_PRIORITY = 1 SCHEDULER_DISK_QUEUE = 'scrapy.squeue.PickleFifoDiskQueue' SCHEDULER_MEMORY_QUEUE = 'scrapy.squeue.FifoMemoryQueue'
2011-09-23 13:03:07 -03:00
class ChunkSize2MarshalFifoDiskQueueTest(MarshalFifoDiskQueueTest):
added MarshalDiskQueue unittests
2011-08-05 20:32:22 -03:00
chunksize = 2
Some changes to persistent scheduler after some initial usage feedback: * added LIFO queues, in addition to the original FIFO queues * use LIFO queues (instead of FIFO queues) by default, since they resemble DFO better which is a more convenient crawling order for most cases * do not adjust the priority based on depth by default (DEPTH_PRIORITY = 0) If someone does need to use strict BFO order, it can be by done by setting: DEPTH_PRIORITY = 1 SCHEDULER_DISK_QUEUE = 'scrapy.squeue.PickleFifoDiskQueue' SCHEDULER_MEMORY_QUEUE = 'scrapy.squeue.FifoMemoryQueue'
2011-09-23 13:03:07 -03:00
class ChunkSize3MarshalFifoDiskQueueTest(MarshalFifoDiskQueueTest):
added MarshalDiskQueue unittests
2011-08-05 20:32:22 -03:00
chunksize = 3
Some changes to persistent scheduler after some initial usage feedback: * added LIFO queues, in addition to the original FIFO queues * use LIFO queues (instead of FIFO queues) by default, since they resemble DFO better which is a more convenient crawling order for most cases * do not adjust the priority based on depth by default (DEPTH_PRIORITY = 0) If someone does need to use strict BFO order, it can be by done by setting: DEPTH_PRIORITY = 1 SCHEDULER_DISK_QUEUE = 'scrapy.squeue.PickleFifoDiskQueue' SCHEDULER_MEMORY_QUEUE = 'scrapy.squeue.FifoMemoryQueue'
2011-09-23 13:03:07 -03:00
class ChunkSize4MarshalFifoDiskQueueTest(MarshalFifoDiskQueueTest):
added MarshalDiskQueue unittests
2011-08-05 20:32:22 -03:00
chunksize = 4
persistent scheduler: use pickle (instead of marshal) as the default serialization format, to support serializing more objects out of the box. also removed __slots__ from Request/Response objects to make them serializable by default.
2011-09-01 14:27:29 -03:00
Some changes to persistent scheduler after some initial usage feedback: * added LIFO queues, in addition to the original FIFO queues * use LIFO queues (instead of FIFO queues) by default, since they resemble DFO better which is a more convenient crawling order for most cases * do not adjust the priority based on depth by default (DEPTH_PRIORITY = 0) If someone does need to use strict BFO order, it can be by done by setting: DEPTH_PRIORITY = 1 SCHEDULER_DISK_QUEUE = 'scrapy.squeue.PickleFifoDiskQueue' SCHEDULER_MEMORY_QUEUE = 'scrapy.squeue.FifoMemoryQueue'
2011-09-23 13:03:07 -03:00
class PickleFifoDiskQueueTest(MarshalFifoDiskQueueTest):
persistent scheduler: use pickle (instead of marshal) as the default serialization format, to support serializing more objects out of the box. also removed __slots__ from Request/Response objects to make them serializable by default.
2011-09-01 14:27:29 -03:00
chunksize = 100000
def queue(self):
fix scrapy squeue tests after recent changes to queuelib
2015-09-09 15:55:55 -03:00
return PickleFifoDiskQueue(self.qpath, chunksize=self.chunksize)
persistent scheduler: use pickle (instead of marshal) as the default serialization format, to support serializing more objects out of the box. also removed __slots__ from Request/Response objects to make them serializable by default.
2011-09-01 14:27:29 -03:00
def test_serialize_item(self):
q = self.queue()
i = TestItem(name='foo')
q.push(i)
i2 = q.pop()
assert isinstance(i2, TestItem)
self.assertEqual(i, i2)
def test_serialize_loader(self):
q = self.queue()
l = TestLoader()
q.push(l)
l2 = q.pop()
assert isinstance(l2, TestLoader)
assert l2.default_item_class is TestItem
self.assertEqual(l2.name_out('x'), 'xx')
def test_serialize_request_recursive(self):
q = self.queue()
r = Request('http://www.example.com')
r.meta['request'] = r
q.push(r)
r2 = q.pop()
assert isinstance(r2, Request)
self.assertEqual(r.url, r2.url)
assert r2.meta['request'] is r2
Some changes to persistent scheduler after some initial usage feedback: * added LIFO queues, in addition to the original FIFO queues * use LIFO queues (instead of FIFO queues) by default, since they resemble DFO better which is a more convenient crawling order for most cases * do not adjust the priority based on depth by default (DEPTH_PRIORITY = 0) If someone does need to use strict BFO order, it can be by done by setting: DEPTH_PRIORITY = 1 SCHEDULER_DISK_QUEUE = 'scrapy.squeue.PickleFifoDiskQueue' SCHEDULER_MEMORY_QUEUE = 'scrapy.squeue.FifoMemoryQueue'
2011-09-23 13:03:07 -03:00
class ChunkSize1PickleFifoDiskQueueTest(PickleFifoDiskQueueTest):
persistent scheduler: use pickle (instead of marshal) as the default serialization format, to support serializing more objects out of the box. also removed __slots__ from Request/Response objects to make them serializable by default.
2011-09-01 14:27:29 -03:00
chunksize = 1
Some changes to persistent scheduler after some initial usage feedback: * added LIFO queues, in addition to the original FIFO queues * use LIFO queues (instead of FIFO queues) by default, since they resemble DFO better which is a more convenient crawling order for most cases * do not adjust the priority based on depth by default (DEPTH_PRIORITY = 0) If someone does need to use strict BFO order, it can be by done by setting: DEPTH_PRIORITY = 1 SCHEDULER_DISK_QUEUE = 'scrapy.squeue.PickleFifoDiskQueue' SCHEDULER_MEMORY_QUEUE = 'scrapy.squeue.FifoMemoryQueue'
2011-09-23 13:03:07 -03:00
class ChunkSize2PickleFifoDiskQueueTest(PickleFifoDiskQueueTest):
persistent scheduler: use pickle (instead of marshal) as the default serialization format, to support serializing more objects out of the box. also removed __slots__ from Request/Response objects to make them serializable by default.
2011-09-01 14:27:29 -03:00
chunksize = 2
Some changes to persistent scheduler after some initial usage feedback: * added LIFO queues, in addition to the original FIFO queues * use LIFO queues (instead of FIFO queues) by default, since they resemble DFO better which is a more convenient crawling order for most cases * do not adjust the priority based on depth by default (DEPTH_PRIORITY = 0) If someone does need to use strict BFO order, it can be by done by setting: DEPTH_PRIORITY = 1 SCHEDULER_DISK_QUEUE = 'scrapy.squeue.PickleFifoDiskQueue' SCHEDULER_MEMORY_QUEUE = 'scrapy.squeue.FifoMemoryQueue'
2011-09-23 13:03:07 -03:00
class ChunkSize3PickleFifoDiskQueueTest(PickleFifoDiskQueueTest):
persistent scheduler: use pickle (instead of marshal) as the default serialization format, to support serializing more objects out of the box. also removed __slots__ from Request/Response objects to make them serializable by default.
2011-09-01 14:27:29 -03:00
chunksize = 3
Some changes to persistent scheduler after some initial usage feedback: * added LIFO queues, in addition to the original FIFO queues * use LIFO queues (instead of FIFO queues) by default, since they resemble DFO better which is a more convenient crawling order for most cases * do not adjust the priority based on depth by default (DEPTH_PRIORITY = 0) If someone does need to use strict BFO order, it can be by done by setting: DEPTH_PRIORITY = 1 SCHEDULER_DISK_QUEUE = 'scrapy.squeue.PickleFifoDiskQueue' SCHEDULER_MEMORY_QUEUE = 'scrapy.squeue.FifoMemoryQueue'
2011-09-23 13:03:07 -03:00
class ChunkSize4PickleFifoDiskQueueTest(PickleFifoDiskQueueTest):
persistent scheduler: use pickle (instead of marshal) as the default serialization format, to support serializing more objects out of the box. also removed __slots__ from Request/Response objects to make them serializable by default.
2011-09-01 14:27:29 -03:00
chunksize = 4
Some changes to persistent scheduler after some initial usage feedback: * added LIFO queues, in addition to the original FIFO queues * use LIFO queues (instead of FIFO queues) by default, since they resemble DFO better which is a more convenient crawling order for most cases * do not adjust the priority based on depth by default (DEPTH_PRIORITY = 0) If someone does need to use strict BFO order, it can be by done by setting: DEPTH_PRIORITY = 1 SCHEDULER_DISK_QUEUE = 'scrapy.squeue.PickleFifoDiskQueue' SCHEDULER_MEMORY_QUEUE = 'scrapy.squeue.FifoMemoryQueue'
2011-09-23 13:03:07 -03:00
class MarshalLifoDiskQueueTest(t.LifoDiskQueueTest):
def queue(self):
fix scrapy squeue tests after recent changes to queuelib
2015-09-09 15:55:55 -03:00
return MarshalLifoDiskQueue(self.qpath)
Some changes to persistent scheduler after some initial usage feedback: * added LIFO queues, in addition to the original FIFO queues * use LIFO queues (instead of FIFO queues) by default, since they resemble DFO better which is a more convenient crawling order for most cases * do not adjust the priority based on depth by default (DEPTH_PRIORITY = 0) If someone does need to use strict BFO order, it can be by done by setting: DEPTH_PRIORITY = 1 SCHEDULER_DISK_QUEUE = 'scrapy.squeue.PickleFifoDiskQueue' SCHEDULER_MEMORY_QUEUE = 'scrapy.squeue.FifoMemoryQueue'
2011-09-23 13:03:07 -03:00
def test_serialize(self):
q = self.queue()
q.push('a')
q.push(123)
q.push({'a': 'dict'})
self.assertEqual(q.pop(), {'a': 'dict'})
self.assertEqual(q.pop(), 123)
self.assertEqual(q.pop(), 'a')
Fix queue serialization test on PyPy It is not affected by Twisted bug #7989 and is more permissive with pickling (especially with protocol=2).
2016-09-01 15:39:16 +03:00
test_nonserializable_object = nonserializable_object_test
Some changes to persistent scheduler after some initial usage feedback: * added LIFO queues, in addition to the original FIFO queues * use LIFO queues (instead of FIFO queues) by default, since they resemble DFO better which is a more convenient crawling order for most cases * do not adjust the priority based on depth by default (DEPTH_PRIORITY = 0) If someone does need to use strict BFO order, it can be by done by setting: DEPTH_PRIORITY = 1 SCHEDULER_DISK_QUEUE = 'scrapy.squeue.PickleFifoDiskQueue' SCHEDULER_MEMORY_QUEUE = 'scrapy.squeue.FifoMemoryQueue'
2011-09-23 13:03:07 -03:00
class PickleLifoDiskQueueTest(MarshalLifoDiskQueueTest):
def queue(self):
fix scrapy squeue tests after recent changes to queuelib
2015-09-09 15:55:55 -03:00
return PickleLifoDiskQueue(self.qpath)
Some changes to persistent scheduler after some initial usage feedback: * added LIFO queues, in addition to the original FIFO queues * use LIFO queues (instead of FIFO queues) by default, since they resemble DFO better which is a more convenient crawling order for most cases * do not adjust the priority based on depth by default (DEPTH_PRIORITY = 0) If someone does need to use strict BFO order, it can be by done by setting: DEPTH_PRIORITY = 1 SCHEDULER_DISK_QUEUE = 'scrapy.squeue.PickleFifoDiskQueue' SCHEDULER_MEMORY_QUEUE = 'scrapy.squeue.FifoMemoryQueue'
2011-09-23 13:03:07 -03:00
def test_serialize_item(self):
q = self.queue()
i = TestItem(name='foo')
q.push(i)
i2 = q.pop()
assert isinstance(i2, TestItem)
self.assertEqual(i, i2)
def test_serialize_loader(self):
q = self.queue()
l = TestLoader()
q.push(l)
l2 = q.pop()
assert isinstance(l2, TestLoader)
assert l2.default_item_class is TestItem
self.assertEqual(l2.name_out('x'), 'xx')
def test_serialize_request_recursive(self):
q = self.queue()
r = Request('http://www.example.com')
r.meta['request'] = r
q.push(r)
r2 = q.pop()
assert isinstance(r2, Request)
self.assertEqual(r.url, r2.url)
assert r2.meta['request'] is r2
Copy Permalink