BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-03-01 19:47:36 +00:00
Code Releases Wiki Activity
1,252 Commits 39 Branches 119 Tags
Commit Graph

1252 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Andres Moreira
cab6387c87 profiling: add new priorityqueue implementation (PQ5c) with cache support. fixed bug in run.py with the use of the with statement. 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401002
 2009-03-19 13:11:27 +00:00
Daniel Grana
bd1be0ac67 profiling: fix priority distribution and add new ones 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401001
 2009-03-19 12:04:19 +00:00
Daniel Grana
2e63ef60f1 profiling: remove useless import 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401000
 2009-03-19 05:42:23 +00:00
Daniel Grana
fdfa5302a4 profiling: minor fixes and new testcase for IndexError exception 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40999
 2009-03-19 05:18:35 +00:00
Daniel Grana
9d1855a66e profiling: add alternative priorityqueue implementations and disable not useful ones from default runs 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40998
 2009-03-19 04:09:24 +00:00
Daniel Grana
1c659ce615 profiling: add list+deque+cache+islice just to compare with pure slice 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40997
 2009-03-19 03:33:06 +00:00
Daniel Grana
685807e3c4 profiling: improve list+deque+cache performance 
results:

 == With 1 priorities (/tmp/pq-13882-1-50000) ==

 pushpops = 50000, times = 30
 heapq implementation: 12.6956150532
 dict+deque implementation: 5.3080239296
 deque+heapq implementation: 3.11057305336
 deque+defaultdict+deque implementation: 3.06583619118
 list+deque implementation: 4.85028195381
 list+deque+cache implementation: 4.86092495918

 == With 3 priorities (/tmp/pq-13882-3-50000) ==

 pushpops = 50000, times = 30
 heapq implementation: 13.9048631191
 dict+deque implementation: 6.526501894
 deque+heapq implementation: 9.95749187469
 deque+defaultdict+deque implementation: 4.94318699837
 list+deque implementation: 5.48832702637
 list+deque+cache implementation: 4.77395009995

 == With 5 priorities (/tmp/pq-13882-5-50000) ==

 pushpops = 50000, times = 30
 heapq implementation: 14.1862449646
 dict+deque implementation: 7.45535206795
 deque+heapq implementation: 11.7175529003
 deque+defaultdict+deque implementation: 5.40972518921
 list+deque implementation: 5.87488412857
 list+deque+cache implementation: 4.73579287529

 == With 10 priorities (/tmp/pq-13882-10-50000) ==

 pushpops = 50000, times = 30
 heapq implementation: 14.2052979469
 dict+deque implementation: 9.94834208488
 deque+heapq implementation: 13.0460109711
 deque+defaultdict+deque implementation: 5.79300785065
 list+deque implementation: 6.9981739521
 list+deque+cache implementation: 4.81988596916

 == With 100 priorities (/tmp/pq-13882-100-50000) ==

 pushpops = 50000, times = 30
 heapq implementation: 14.9574189186
 dict+deque implementation: 55.6348400116
 deque+heapq implementation: 14.9515259266
 deque+defaultdict+deque implementation: 10.6776599884
 list+deque implementation: 25.9212520123
 list+deque+cache implementation: 4.77596998215

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40996
 2009-03-19 03:11:27 +00:00
Daniel Grana
44ba741277 profiling: automatically test include new PriorityQueue classes in testcases 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40995
 2009-03-19 02:47:59 +00:00
Daniel Grana
e1c1865c2a profiling: add options to priority queue profiler 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40994
 2009-03-19 02:25:20 +00:00
Daniel Grana
22c51638a8 profiling: add priorityqueue alternatives and profiling / test cases 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40993
 2009-03-18 19:44:36 +00:00
Andres Moreira
2951e8425f Improved PriorityQueue and PriorityStack that boost the performance again. Also updated the unittest for those datatypes. 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40992
 2009-03-18 17:09:26 +00:00
Pablo Hoffman
7b664a57b2 grammar adjustment 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40991
 2009-03-18 00:47:07 +00:00
Daniel Grana
fb3a2dab2f core: allow returning None from spiders to support pipeline/download throttling 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40990
 2009-03-17 21:30:33 +00:00
Ismael Carnales
c54a3fe7f6 removed item_to_dict 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40989
 2009-03-17 15:22:57 +00:00
Ismael Carnales
63150e2ed4 added item_to_dict util function 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40988
 2009-03-16 01:12:57 +00:00
Daniel Grana
f9e5915fd4 newitem: change itemadaptor tests to avoid false positives 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40987
 2009-03-13 20:05:37 +00:00
Andres Moreira
cbf6d76c5d Improved PriorityQueue and PriorityStack with a patch sent by Federico Feroldi. The new implementation is about two times faster than the old. 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40986
 2009-03-12 19:06:57 +00:00
Ismael Carnales
518a7c87dd make field_adaptors non-public 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40985
 2009-03-11 13:26:29 +00:00
Ismael Carnales
2ba017b14d add tests for adaptation of multivaluated fields 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40984
 2009-03-11 13:01:26 +00:00
Daniel Grana
0be771152b stats: fix stats key typo 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40983
 2009-03-11 03:01:30 +00:00
Pablo Hoffman
f8ce241484 fixed MailSender documentation example - closes #74 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40982
 2009-03-10 19:56:22 +00:00
Ismael Carnales
ec1a6af04a added MultiValuedField to newitem 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40981
 2009-03-10 19:35:45 +00:00
Ismael Carnales
2f75c2c93e removed unneeded import 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40980
 2009-03-10 19:17:53 +00:00
Ismael Carnales
04baa7aed7 removed declarative from newitem (now using 'plain' metaclasses) added basic test for newitem 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40979
 2009-03-10 18:40:26 +00:00
Ismael Carnales
774e7f623a fixed default adaptor inheritance, added test 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40978
 2009-03-10 17:59:53 +00:00
Daniel Grana
67ab30a5ba core: add application/xhtml+xml support to responsetypes 
more info at http://www.w3.org/TR/xhtml-media-types/#media-types

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40977
 2009-03-10 17:51:03 +00:00
Ismael Carnales
63e6ce6e47 made IDENTITY a function of adaptor module 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40976
 2009-03-10 16:57:27 +00:00
Ismael Carnales
698e6e73ea moved default_adaptor logic fully to the metaclass in ItemAdaptor 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40975
 2009-03-10 16:34:48 +00:00
Ismael Carnales
06e049e551 move meta class initialization to metaclass in ItemAdaptor 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40974
 2009-03-10 16:15:49 +00:00
Ismael Carnales
f105359666 use metaclass __getattr__ to return default_adaptor in ItemAdaptor 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40973
 2009-03-10 15:53:42 +00:00
Ismael Carnales
8bda81c01d add inheritance of item and adaptors to itemadaptor test 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40972
 2009-03-10 14:03:27 +00:00
Ismael Carnales
f9aa0a6a01 removed is_adaptor check in ItemAdaptor in favour of funcion adaptizer 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40971
 2009-03-10 13:38:32 +00:00
Pablo Hoffman
4093a4b83e added support to run scrapy tests with: python -m scrapy.tests 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40970
 2009-03-05 11:12:07 +00:00
Daniel Grana
e3c9f5951a contrib: fix a header bug in httpcompression middleware and add tests 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40969
 2009-03-04 20:11:12 +00:00
Ismael Carnales
8ad4211e26 small changes to make code 3.0 compatible :) 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40968
 2009-03-04 17:11:20 +00:00
Ismael Carnales
ec22373b5a use ItemAdaptor.__classinit__ to set field_adaptors, also make defaulted field adaptors static_methods 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40967
 2009-03-03 22:35:15 +00:00
Ismael Carnales
b5b6b0cefc removed unneeded decorator from Item xD 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40966
 2009-03-03 21:32:37 +00:00
Ismael Carnales
154dded1de using declarative in ItemAdaptor, more to come 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40965
 2009-03-03 21:29:00 +00:00
Ismael Carnales
e744e87517 code style changes for newitem, and only return fields or special attrs in Item 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40964
 2009-03-03 21:18:40 +00:00
Damian Canabal
d26b968f51 fixing typo 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40963
 2009-03-03 21:14:50 +00:00
Ismael Carnales
60de639434 added declarative module for declarative style classses and make Item use it 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40962
 2009-03-03 21:12:37 +00:00
Daniel Grana
0d47fdf6e8 utils: revert canonicalize_url implementation, it wasnt intended to be changed 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40961
 2009-03-03 15:18:17 +00:00
Daniel Grana
8724ee1674 utils: url canonicalization keeps blanks query parameters 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40960
 2009-03-03 15:05:50 +00:00
Ismael Carnales
2e1a170563 added LxmlParserLinkExtractor a new try on link extracting 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40959
 2009-03-03 07:42:54 +00:00
Daniel Grana
309bf9277e xmlrpc: send request as POST by default and use dont_filter 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40958
 2009-03-03 05:37:41 +00:00
Daniel Grana
ff7c6d3782 http: add XmlRpcRequest based on xmlrpclib 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40957
 2009-03-03 00:37:06 +00:00
Daniel Grana
ab3fdbdb8e core: prevent download timeout from been accidentally disabled 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40956
 2009-03-02 23:22:36 +00:00
Ismael Carnales
875e8e57d3 added lxml based link extractor for supporting sites with broken htm 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40955
 2009-03-02 19:49:42 +00:00
Ismael Carnales
127e5ae029 convert process_attr to a parameter in contructor so extending the class is not needed 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40954
 2009-03-02 18:03:48 +00:00
Daniel Grana
1f38460ba4 allow returning None and non-iterables from spiders 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40953
 2009-03-02 17:29:48 +00:00