BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-03-01 19:27:27 +00:00
Code Releases Wiki Activity
5,352 Commits 39 Branches 119 Tags
Commit Graph

5352 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Daniel Grana
2acdd9b409 utils: add any value to iterable function 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40952
 2009-03-02 17:29:18 +00:00
Ismael Carnales
315934bf6b added new LinkExtractor based on HTMLParser and with a hook for preprocessing the attribute 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40951
 2009-03-02 17:27:29 +00:00
Daniel Grana
57c7c5e9f6 images: log a short message when images request is ignored to dupefilter 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40950
 2009-03-02 15:37:57 +00:00
Pablo Hoffman
4b0d69cc6d removed some unused imports 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40949
 2009-03-02 14:52:34 +00:00
Ismael Carnales
d8f31f5678 pass the item_instance of ItemAdaptor as an adaptor_arg 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40948
 2009-03-02 02:58:10 +00:00
Ismael Carnales
16c20f5c4c added default_adaptor to ItemAdaptor 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40947
 2009-03-01 14:26:11 +00:00
Ismael Carnales
f145e850ad discover adaptors in the python way 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40946
 2009-03-01 14:16:05 +00:00
Ismael Carnales
41ef6c262e added default_adaptor to the newitem documentation 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40945
 2009-03-01 13:09:54 +00:00
Ismael Carnales
6724819423 updated newitem proposed docs 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40944
 2009-03-01 12:59:28 +00:00
Ismael Carnales
ff950e7aa9 added intro/tutorial for new Item and ItemAdaptor 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40943
 2009-03-01 12:28:55 +00:00
Daniel Grana
43b87acd81 docs: rename setting doc accordly 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40942
 2009-03-01 11:14:30 +00:00
Daniel Grana
e998f92297 docs: document SCHEDULER_MIDDLEWARE setting 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40941
 2009-03-01 11:04:09 +00:00
Pablo Hoffman
ee80b1bcc0 added doc about 'dont_merge_cookies' recently added by Damian 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40940
 2009-02-28 03:04:31 +00:00
Pablo Hoffman
ca22e332d1 renamed to_list function to arg_to_list, added docstring and one more test 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40939
 2009-02-28 02:51:18 +00:00
Pablo Hoffman
10be9b695d moved Ismael to Scrapy contributors 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40938
 2009-02-28 02:42:46 +00:00
Pablo Hoffman
55fb3c0dd9 removed UsageError from docs 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40937
 2009-02-28 02:42:32 +00:00
Pablo Hoffman
d16169e9ab replaced UsageError exception by more standard (and explicit) ones such as TypeError and ValueError 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40936
 2009-02-28 02:39:28 +00:00
Daniel Grana
bfe5a554c2 newitem: clenaup ItemAdaptor inheritance search 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40935
 2009-02-27 18:48:09 +00:00
Pablo Hoffman
e590764dc4 added Andres to AUTHORS 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40934
 2009-02-27 18:47:48 +00:00
Daniel Grana
a88119aa8b newitem: redefine adaptor fields as staticmethods 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40933
 2009-02-27 18:44:40 +00:00
samus_
7215ffa1eb a bit of convention 
--HG--
rename : scrapy/trunk/scrapy/contrib/downloadermiddleware/http_compression.py => scrapy/trunk/scrapy/contrib/downloadermiddleware/httpcompression.py
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40932
 2009-02-25 09:53:44 +00:00
samus_
bf960fb102 renamed scrapy.contrib.downloadermiddleware.compression.CompressionMiddleware to scrapy.contrib.downloadermiddleware.http_compression.HTTPCompressionMiddleware 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40931
 2009-02-25 09:38:14 +00:00
Daniel Grana
bfcde7e061 newitem: more tests on itemadaptor inheritance 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40930
 2009-02-25 09:31:39 +00:00
Daniel Grana
fdaa2ec3ff newitem: support multiple depth inheritance of ItemAdaptor 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40929
 2009-02-25 09:15:28 +00:00
Daniel Grana
03db659040 newitem: make adaptor inheritance python2.5 compatible 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40928
 2009-02-25 08:44:58 +00:00
Ismael Carnales
0a598ff0b1 allow setting not adapted item values in ItemAdaptor 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40927
 2009-02-25 08:28:24 +00:00
Ismael Carnales
6d2a3e093b added to_date adaptor to format a date suitable for DateField 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40926
 2009-02-25 07:42:35 +00:00
Daniel Grana
3e8945a477 newitem: another fix to ItemAdaptor inheritance and tests included 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40925
 2009-02-25 07:39:14 +00:00
Daniel Grana
cf3afb9490 newitem: fix itemadaptor inheritance 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40924
 2009-02-25 06:52:06 +00:00
samus_
1fa5c7bade back to the basics 8) 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40923
 2009-02-25 06:48:41 +00:00
samus_
5af4f3a279 fix except syntax 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40922
 2009-02-25 06:37:17 +00:00
samus_
2f60224b91 fix except syntax 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40921
 2009-02-25 06:33:08 +00:00
Ismael Carnales
4675404fc1 ooops, removed extra print statement from adaptors.py 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40920
 2009-02-25 06:30:31 +00:00
Ismael Carnales
9fb0f5094e renamed extractors to adaptors 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40919
 2009-02-25 06:24:55 +00:00
samus_
c6e76dadc6 (missed parameter) 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40918
 2009-02-25 05:49:23 +00:00
samus_
de1b9e5ec9 added gzip support for logs 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40917
 2009-02-25 05:48:01 +00:00
Daniel Grana
82dc57e59b headers: complete new Headers behaviour migration. closes #47 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40916
 2009-02-25 00:29:10 +00:00
Daniel Grana
5de0816d9f url: add custom port based testcase 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40915
 2009-02-25 00:27:34 +00:00
Daniel Grana
5d3489ac10 headers: add docstring and title prior encoding keys. refs #47 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40914
 2009-02-25 00:27:02 +00:00
Pablo Hoffman
987ceb540e fixed import error (thanks Shane) and added some basic tests for ScrapedItem 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40913
 2009-02-24 14:17:16 +00:00
Daniel Grana
934535740b headers: cleanup Headers class and add normvalue method to CaselessDict 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40912
 2009-02-24 06:58:12 +00:00
Daniel Grana
f4b55bfd20 datatypes: add CaselessDict test and cleanup 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40911
 2009-02-24 06:57:28 +00:00
Daniel Grana
d504a2b428 bugfix newitem adaptor imports 
--HG--
rename : scrapy/trunk/scrapy/tests/test_itemextractor.py => scrapy/trunk/scrapy/tests/test_itemadaptor.py
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40910
 2009-02-24 05:13:31 +00:00
Daniel Grana
119706ae23 headers: remove unused methods 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40909
 2009-02-24 05:12:58 +00:00
Ismael Carnales
5afa621d6a moved extractors.py to adaptors.py 
--HG--
rename : scrapy/trunk/scrapy/contrib_exp/newitem/extractors.py => scrapy/trunk/scrapy/contrib_exp/newitem/adaptors.py
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40908
 2009-02-24 04:04:25 +00:00
Ismael Carnales
8398212404 renamed ItemExtractor to ItemAdaptor 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40907
 2009-02-24 04:02:34 +00:00
Ismael Carnales
b26102745a minor code style changes to extractors.py 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40906
 2009-02-24 03:51:40 +00:00
Daniel Grana
53983e6e23 newitem: remove ExtractorField and port googledir example 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40905
 2009-02-24 03:42:21 +00:00
Daniel Grana
15909ef62f newitem: rename treeadapt as adaptor, and bugfix a mutable dictionary bug 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40904
 2009-02-24 03:41:44 +00:00
Damian Canabal
ebe106b7a2 check for dont_merge_cookies flag in request.meta 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40903
 2009-02-24 00:48:50 +00:00