BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-26 06:43:44 +00:00
Code Releases Wiki Activity
1,120 Commits 39 Branches 119 Tags
Commit Graph

1120 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Pablo Hoffman
4093a4b83e added support to run scrapy tests with: python -m scrapy.tests 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40970
 2009-03-05 11:12:07 +00:00
Daniel Grana
e3c9f5951a contrib: fix a header bug in httpcompression middleware and add tests 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40969
 2009-03-04 20:11:12 +00:00
Ismael Carnales
8ad4211e26 small changes to make code 3.0 compatible :) 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40968
 2009-03-04 17:11:20 +00:00
Ismael Carnales
ec22373b5a use ItemAdaptor.__classinit__ to set field_adaptors, also make defaulted field adaptors static_methods 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40967
 2009-03-03 22:35:15 +00:00
Ismael Carnales
b5b6b0cefc removed unneeded decorator from Item xD 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40966
 2009-03-03 21:32:37 +00:00
Ismael Carnales
154dded1de using declarative in ItemAdaptor, more to come 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40965
 2009-03-03 21:29:00 +00:00
Ismael Carnales
e744e87517 code style changes for newitem, and only return fields or special attrs in Item 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40964
 2009-03-03 21:18:40 +00:00
Damian Canabal
d26b968f51 fixing typo 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40963
 2009-03-03 21:14:50 +00:00
Ismael Carnales
60de639434 added declarative module for declarative style classses and make Item use it 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40962
 2009-03-03 21:12:37 +00:00
Daniel Grana
0d47fdf6e8 utils: revert canonicalize_url implementation, it wasnt intended to be changed 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40961
 2009-03-03 15:18:17 +00:00
Daniel Grana
8724ee1674 utils: url canonicalization keeps blanks query parameters 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40960
 2009-03-03 15:05:50 +00:00
Ismael Carnales
2e1a170563 added LxmlParserLinkExtractor a new try on link extracting 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40959
 2009-03-03 07:42:54 +00:00
Daniel Grana
309bf9277e xmlrpc: send request as POST by default and use dont_filter 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40958
 2009-03-03 05:37:41 +00:00
Daniel Grana
ff7c6d3782 http: add XmlRpcRequest based on xmlrpclib 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40957
 2009-03-03 00:37:06 +00:00
Daniel Grana
ab3fdbdb8e core: prevent download timeout from been accidentally disabled 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40956
 2009-03-02 23:22:36 +00:00
Ismael Carnales
875e8e57d3 added lxml based link extractor for supporting sites with broken htm 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40955
 2009-03-02 19:49:42 +00:00
Ismael Carnales
127e5ae029 convert process_attr to a parameter in contructor so extending the class is not needed 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40954
 2009-03-02 18:03:48 +00:00
Daniel Grana
1f38460ba4 allow returning None and non-iterables from spiders 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40953
 2009-03-02 17:29:48 +00:00
Daniel Grana
2acdd9b409 utils: add any value to iterable function 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40952
 2009-03-02 17:29:18 +00:00
Ismael Carnales
315934bf6b added new LinkExtractor based on HTMLParser and with a hook for preprocessing the attribute 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40951
 2009-03-02 17:27:29 +00:00
Daniel Grana
57c7c5e9f6 images: log a short message when images request is ignored to dupefilter 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40950
 2009-03-02 15:37:57 +00:00
Pablo Hoffman
4b0d69cc6d removed some unused imports 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40949
 2009-03-02 14:52:34 +00:00
Ismael Carnales
d8f31f5678 pass the item_instance of ItemAdaptor as an adaptor_arg 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40948
 2009-03-02 02:58:10 +00:00
Ismael Carnales
16c20f5c4c added default_adaptor to ItemAdaptor 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40947
 2009-03-01 14:26:11 +00:00
Ismael Carnales
f145e850ad discover adaptors in the python way 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40946
 2009-03-01 14:16:05 +00:00
Ismael Carnales
41ef6c262e added default_adaptor to the newitem documentation 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40945
 2009-03-01 13:09:54 +00:00
Ismael Carnales
6724819423 updated newitem proposed docs 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40944
 2009-03-01 12:59:28 +00:00
Ismael Carnales
ff950e7aa9 added intro/tutorial for new Item and ItemAdaptor 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40943
 2009-03-01 12:28:55 +00:00
Daniel Grana
43b87acd81 docs: rename setting doc accordly 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40942
 2009-03-01 11:14:30 +00:00
Daniel Grana
e998f92297 docs: document SCHEDULER_MIDDLEWARE setting 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40941
 2009-03-01 11:04:09 +00:00
Pablo Hoffman
ee80b1bcc0 added doc about 'dont_merge_cookies' recently added by Damian 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40940
 2009-02-28 03:04:31 +00:00
Pablo Hoffman
ca22e332d1 renamed to_list function to arg_to_list, added docstring and one more test 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40939
 2009-02-28 02:51:18 +00:00
Pablo Hoffman
10be9b695d moved Ismael to Scrapy contributors 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40938
 2009-02-28 02:42:46 +00:00
Pablo Hoffman
55fb3c0dd9 removed UsageError from docs 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40937
 2009-02-28 02:42:32 +00:00
Pablo Hoffman
d16169e9ab replaced UsageError exception by more standard (and explicit) ones such as TypeError and ValueError 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40936
 2009-02-28 02:39:28 +00:00
Daniel Grana
bfe5a554c2 newitem: clenaup ItemAdaptor inheritance search 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40935
 2009-02-27 18:48:09 +00:00
Pablo Hoffman
e590764dc4 added Andres to AUTHORS 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40934
 2009-02-27 18:47:48 +00:00
Daniel Grana
a88119aa8b newitem: redefine adaptor fields as staticmethods 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40933
 2009-02-27 18:44:40 +00:00
samus_
7215ffa1eb a bit of convention 
--HG--
rename : scrapy/trunk/scrapy/contrib/downloadermiddleware/http_compression.py => scrapy/trunk/scrapy/contrib/downloadermiddleware/httpcompression.py
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40932
 2009-02-25 09:53:44 +00:00
samus_
bf960fb102 renamed scrapy.contrib.downloadermiddleware.compression.CompressionMiddleware to scrapy.contrib.downloadermiddleware.http_compression.HTTPCompressionMiddleware 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40931
 2009-02-25 09:38:14 +00:00
Daniel Grana
bfcde7e061 newitem: more tests on itemadaptor inheritance 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40930
 2009-02-25 09:31:39 +00:00
Daniel Grana
fdaa2ec3ff newitem: support multiple depth inheritance of ItemAdaptor 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40929
 2009-02-25 09:15:28 +00:00
Daniel Grana
03db659040 newitem: make adaptor inheritance python2.5 compatible 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40928
 2009-02-25 08:44:58 +00:00
Ismael Carnales
0a598ff0b1 allow setting not adapted item values in ItemAdaptor 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40927
 2009-02-25 08:28:24 +00:00
Ismael Carnales
6d2a3e093b added to_date adaptor to format a date suitable for DateField 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40926
 2009-02-25 07:42:35 +00:00
Daniel Grana
3e8945a477 newitem: another fix to ItemAdaptor inheritance and tests included 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40925
 2009-02-25 07:39:14 +00:00
Daniel Grana
cf3afb9490 newitem: fix itemadaptor inheritance 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40924
 2009-02-25 06:52:06 +00:00
samus_
1fa5c7bade back to the basics 8) 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40923
 2009-02-25 06:48:41 +00:00
samus_
5af4f3a279 fix except syntax 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40922
 2009-02-25 06:37:17 +00:00
samus_
2f60224b91 fix except syntax 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40921
 2009-02-25 06:33:08 +00:00