BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-26 06:23:41 +00:00
Code Releases Wiki Activity
1,091 Commits 39 Branches 119 Tags
Commit Graph

1091 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Daniel Grana
e998f92297 docs: document SCHEDULER_MIDDLEWARE setting 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40941
 2009-03-01 11:04:09 +00:00
Pablo Hoffman
ee80b1bcc0 added doc about 'dont_merge_cookies' recently added by Damian 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40940
 2009-02-28 03:04:31 +00:00
Pablo Hoffman
ca22e332d1 renamed to_list function to arg_to_list, added docstring and one more test 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40939
 2009-02-28 02:51:18 +00:00
Pablo Hoffman
10be9b695d moved Ismael to Scrapy contributors 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40938
 2009-02-28 02:42:46 +00:00
Pablo Hoffman
55fb3c0dd9 removed UsageError from docs 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40937
 2009-02-28 02:42:32 +00:00
Pablo Hoffman
d16169e9ab replaced UsageError exception by more standard (and explicit) ones such as TypeError and ValueError 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40936
 2009-02-28 02:39:28 +00:00
Daniel Grana
bfe5a554c2 newitem: clenaup ItemAdaptor inheritance search 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40935
 2009-02-27 18:48:09 +00:00
Pablo Hoffman
e590764dc4 added Andres to AUTHORS 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40934
 2009-02-27 18:47:48 +00:00
Daniel Grana
a88119aa8b newitem: redefine adaptor fields as staticmethods 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40933
 2009-02-27 18:44:40 +00:00
samus_
7215ffa1eb a bit of convention 
--HG--
rename : scrapy/trunk/scrapy/contrib/downloadermiddleware/http_compression.py => scrapy/trunk/scrapy/contrib/downloadermiddleware/httpcompression.py
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40932
 2009-02-25 09:53:44 +00:00
samus_
bf960fb102 renamed scrapy.contrib.downloadermiddleware.compression.CompressionMiddleware to scrapy.contrib.downloadermiddleware.http_compression.HTTPCompressionMiddleware 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40931
 2009-02-25 09:38:14 +00:00
Daniel Grana
bfcde7e061 newitem: more tests on itemadaptor inheritance 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40930
 2009-02-25 09:31:39 +00:00
Daniel Grana
fdaa2ec3ff newitem: support multiple depth inheritance of ItemAdaptor 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40929
 2009-02-25 09:15:28 +00:00
Daniel Grana
03db659040 newitem: make adaptor inheritance python2.5 compatible 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40928
 2009-02-25 08:44:58 +00:00
Ismael Carnales
0a598ff0b1 allow setting not adapted item values in ItemAdaptor 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40927
 2009-02-25 08:28:24 +00:00
Ismael Carnales
6d2a3e093b added to_date adaptor to format a date suitable for DateField 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40926
 2009-02-25 07:42:35 +00:00
Daniel Grana
3e8945a477 newitem: another fix to ItemAdaptor inheritance and tests included 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40925
 2009-02-25 07:39:14 +00:00
Daniel Grana
cf3afb9490 newitem: fix itemadaptor inheritance 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40924
 2009-02-25 06:52:06 +00:00
samus_
1fa5c7bade back to the basics 8) 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40923
 2009-02-25 06:48:41 +00:00
samus_
5af4f3a279 fix except syntax 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40922
 2009-02-25 06:37:17 +00:00
samus_
2f60224b91 fix except syntax 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40921
 2009-02-25 06:33:08 +00:00
Ismael Carnales
4675404fc1 ooops, removed extra print statement from adaptors.py 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40920
 2009-02-25 06:30:31 +00:00
Ismael Carnales
9fb0f5094e renamed extractors to adaptors 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40919
 2009-02-25 06:24:55 +00:00
samus_
c6e76dadc6 (missed parameter) 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40918
 2009-02-25 05:49:23 +00:00
samus_
de1b9e5ec9 added gzip support for logs 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40917
 2009-02-25 05:48:01 +00:00
Daniel Grana
82dc57e59b headers: complete new Headers behaviour migration. closes #47 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40916
 2009-02-25 00:29:10 +00:00
Daniel Grana
5de0816d9f url: add custom port based testcase 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40915
 2009-02-25 00:27:34 +00:00
Daniel Grana
5d3489ac10 headers: add docstring and title prior encoding keys. refs #47 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40914
 2009-02-25 00:27:02 +00:00
Pablo Hoffman
987ceb540e fixed import error (thanks Shane) and added some basic tests for ScrapedItem 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40913
 2009-02-24 14:17:16 +00:00
Daniel Grana
934535740b headers: cleanup Headers class and add normvalue method to CaselessDict 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40912
 2009-02-24 06:58:12 +00:00
Daniel Grana
f4b55bfd20 datatypes: add CaselessDict test and cleanup 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40911
 2009-02-24 06:57:28 +00:00
Daniel Grana
d504a2b428 bugfix newitem adaptor imports 
--HG--
rename : scrapy/trunk/scrapy/tests/test_itemextractor.py => scrapy/trunk/scrapy/tests/test_itemadaptor.py
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40910
 2009-02-24 05:13:31 +00:00
Daniel Grana
119706ae23 headers: remove unused methods 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40909
 2009-02-24 05:12:58 +00:00
Ismael Carnales
5afa621d6a moved extractors.py to adaptors.py 
--HG--
rename : scrapy/trunk/scrapy/contrib_exp/newitem/extractors.py => scrapy/trunk/scrapy/contrib_exp/newitem/adaptors.py
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40908
 2009-02-24 04:04:25 +00:00
Ismael Carnales
8398212404 renamed ItemExtractor to ItemAdaptor 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40907
 2009-02-24 04:02:34 +00:00
Ismael Carnales
b26102745a minor code style changes to extractors.py 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40906
 2009-02-24 03:51:40 +00:00
Daniel Grana
53983e6e23 newitem: remove ExtractorField and port googledir example 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40905
 2009-02-24 03:42:21 +00:00
Daniel Grana
15909ef62f newitem: rename treeadapt as adaptor, and bugfix a mutable dictionary bug 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40904
 2009-02-24 03:41:44 +00:00
Damian Canabal
ebe106b7a2 check for dont_merge_cookies flag in request.meta 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40903
 2009-02-24 00:48:50 +00:00
Daniel Grana
b0624e23ed proper rename dupefilter test 
--HG--
rename : scrapy/trunk/scrapy/tests/test_filters.py => scrapy/trunk/scrapy/tests/test_dupefilter.py
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40902
 2009-02-23 20:27:02 +00:00
Daniel Grana
d1265a4199 move dupe filter code outside of core. refs #49 
--HG--
rename : scrapy/trunk/scrapy/core/filters.py => scrapy/trunk/scrapy/dupefilter/__init__.py
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40901
 2009-02-23 20:21:11 +00:00
Daniel Grana
c6326a0425 core: add a custom HTTPClientFactory to reuse url parsing, and remove malformed headers monkeypatches. closes #52 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40900
 2009-02-23 19:49:37 +00:00
Pablo Hoffman
86d0256815 added FAQ about HTTP proxies 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40899
 2009-02-21 21:26:10 +00:00
samus_
9d5a1d77b6 adding to_list() util and test 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40898
 2009-02-20 18:17:55 +00:00
Daniel Grana
8d8a5c5caa change ExtractorField callable signature to honor adaptor_args 
and add alternative treeadapt implementation.

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40897
 2009-02-20 17:59:58 +00:00
Pablo Hoffman
0c2e019f66 added test for load_object 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40896
 2009-02-20 17:35:25 +00:00
Ismael Carnales
059bf613fa added example project that uses newitem and ItemExtractor 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40895
 2009-02-20 12:02:56 +00:00
Ismael Carnales
5a5aef6629 made newitem.Item inherit from ScrapedItem for compatibility 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40894
 2009-02-20 11:42:36 +00:00
Daniel Grana
8d5222f1db core: add missing imports 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40893
 2009-02-20 11:31:09 +00:00
Daniel Grana
4bf196fc99 core: add scheduler middleware and move duplicate fitler there 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40892
 2009-02-20 11:28:37 +00:00