BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-26 20:44:04 +00:00
Code Releases Wiki Activity
931 Commits 39 Branches 119 Tags
Commit Graph

931 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
samus_
bf960fb102 renamed scrapy.contrib.downloadermiddleware.compression.CompressionMiddleware to scrapy.contrib.downloadermiddleware.http_compression.HTTPCompressionMiddleware 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40931
 2009-02-25 09:38:14 +00:00
Daniel Grana
bfcde7e061 newitem: more tests on itemadaptor inheritance 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40930
 2009-02-25 09:31:39 +00:00
Daniel Grana
fdaa2ec3ff newitem: support multiple depth inheritance of ItemAdaptor 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40929
 2009-02-25 09:15:28 +00:00
Daniel Grana
03db659040 newitem: make adaptor inheritance python2.5 compatible 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40928
 2009-02-25 08:44:58 +00:00
Ismael Carnales
0a598ff0b1 allow setting not adapted item values in ItemAdaptor 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40927
 2009-02-25 08:28:24 +00:00
Ismael Carnales
6d2a3e093b added to_date adaptor to format a date suitable for DateField 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40926
 2009-02-25 07:42:35 +00:00
Daniel Grana
3e8945a477 newitem: another fix to ItemAdaptor inheritance and tests included 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40925
 2009-02-25 07:39:14 +00:00
Daniel Grana
cf3afb9490 newitem: fix itemadaptor inheritance 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40924
 2009-02-25 06:52:06 +00:00
samus_
1fa5c7bade back to the basics 8) 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40923
 2009-02-25 06:48:41 +00:00
samus_
5af4f3a279 fix except syntax 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40922
 2009-02-25 06:37:17 +00:00
samus_
2f60224b91 fix except syntax 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40921
 2009-02-25 06:33:08 +00:00
Ismael Carnales
4675404fc1 ooops, removed extra print statement from adaptors.py 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40920
 2009-02-25 06:30:31 +00:00
Ismael Carnales
9fb0f5094e renamed extractors to adaptors 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40919
 2009-02-25 06:24:55 +00:00
samus_
c6e76dadc6 (missed parameter) 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40918
 2009-02-25 05:49:23 +00:00
samus_
de1b9e5ec9 added gzip support for logs 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40917
 2009-02-25 05:48:01 +00:00
Daniel Grana
82dc57e59b headers: complete new Headers behaviour migration. closes #47 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40916
 2009-02-25 00:29:10 +00:00
Daniel Grana
5de0816d9f url: add custom port based testcase 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40915
 2009-02-25 00:27:34 +00:00
Daniel Grana
5d3489ac10 headers: add docstring and title prior encoding keys. refs #47 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40914
 2009-02-25 00:27:02 +00:00
Pablo Hoffman
987ceb540e fixed import error (thanks Shane) and added some basic tests for ScrapedItem 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40913
 2009-02-24 14:17:16 +00:00
Daniel Grana
934535740b headers: cleanup Headers class and add normvalue method to CaselessDict 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40912
 2009-02-24 06:58:12 +00:00
Daniel Grana
f4b55bfd20 datatypes: add CaselessDict test and cleanup 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40911
 2009-02-24 06:57:28 +00:00
Daniel Grana
d504a2b428 bugfix newitem adaptor imports 
--HG--
rename : scrapy/trunk/scrapy/tests/test_itemextractor.py => scrapy/trunk/scrapy/tests/test_itemadaptor.py
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40910
 2009-02-24 05:13:31 +00:00
Daniel Grana
119706ae23 headers: remove unused methods 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40909
 2009-02-24 05:12:58 +00:00
Ismael Carnales
5afa621d6a moved extractors.py to adaptors.py 
--HG--
rename : scrapy/trunk/scrapy/contrib_exp/newitem/extractors.py => scrapy/trunk/scrapy/contrib_exp/newitem/adaptors.py
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40908
 2009-02-24 04:04:25 +00:00
Ismael Carnales
8398212404 renamed ItemExtractor to ItemAdaptor 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40907
 2009-02-24 04:02:34 +00:00
Ismael Carnales
b26102745a minor code style changes to extractors.py 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40906
 2009-02-24 03:51:40 +00:00
Daniel Grana
53983e6e23 newitem: remove ExtractorField and port googledir example 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40905
 2009-02-24 03:42:21 +00:00
Daniel Grana
15909ef62f newitem: rename treeadapt as adaptor, and bugfix a mutable dictionary bug 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40904
 2009-02-24 03:41:44 +00:00
Damian Canabal
ebe106b7a2 check for dont_merge_cookies flag in request.meta 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40903
 2009-02-24 00:48:50 +00:00
Daniel Grana
b0624e23ed proper rename dupefilter test 
--HG--
rename : scrapy/trunk/scrapy/tests/test_filters.py => scrapy/trunk/scrapy/tests/test_dupefilter.py
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40902
 2009-02-23 20:27:02 +00:00
Daniel Grana
d1265a4199 move dupe filter code outside of core. refs #49 
--HG--
rename : scrapy/trunk/scrapy/core/filters.py => scrapy/trunk/scrapy/dupefilter/__init__.py
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40901
 2009-02-23 20:21:11 +00:00
Daniel Grana
c6326a0425 core: add a custom HTTPClientFactory to reuse url parsing, and remove malformed headers monkeypatches. closes #52 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40900
 2009-02-23 19:49:37 +00:00
Pablo Hoffman
86d0256815 added FAQ about HTTP proxies 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40899
 2009-02-21 21:26:10 +00:00
samus_
9d5a1d77b6 adding to_list() util and test 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40898
 2009-02-20 18:17:55 +00:00
Daniel Grana
8d8a5c5caa change ExtractorField callable signature to honor adaptor_args 
and add alternative treeadapt implementation.

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40897
 2009-02-20 17:59:58 +00:00
Pablo Hoffman
0c2e019f66 added test for load_object 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40896
 2009-02-20 17:35:25 +00:00
Ismael Carnales
059bf613fa added example project that uses newitem and ItemExtractor 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40895
 2009-02-20 12:02:56 +00:00
Ismael Carnales
5a5aef6629 made newitem.Item inherit from ScrapedItem for compatibility 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40894
 2009-02-20 11:42:36 +00:00
Daniel Grana
8d5222f1db core: add missing imports 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40893
 2009-02-20 11:31:09 +00:00
Daniel Grana
4bf196fc99 core: add scheduler middleware and move duplicate fitler there 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40892
 2009-02-20 11:28:37 +00:00
Daniel Grana
549c30d8d5 redirect: remove stupid set 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40891
 2009-02-20 11:27:46 +00:00
Daniel Grana
071aa71b68 add filters module 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40890
 2009-02-19 19:53:30 +00:00
Daniel Grana
0e98526daa duplicatesfilter: add a singeton duplicates filter and adapt current middleware and redirection middleware 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40889
 2009-02-19 19:43:24 +00:00
Daniel Grana
ac735a185e cleanup redirection middleware 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40888
 2009-02-19 19:42:42 +00:00
Ismael Carnales
65e8be46bc add typecheck of funcs in ExtractorField.__init__ 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40887
 2009-02-19 19:05:45 +00:00
Ismael Carnales
edf5dfb264 added newitem.extractors based on old adaptors 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40886
 2009-02-19 18:56:59 +00:00
Daniel Grana
756cda3873 duplicatefilter: dont raise ignorerequest, but add request to filter on spider input 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40885
 2009-02-19 16:49:03 +00:00
Daniel Grana
fe7935005c duplicatefilter: filter request prior to reach spider 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40884
 2009-02-19 16:37:33 +00:00
Daniel Grana
5338a48db8 remove missing import 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40883
 2009-02-19 16:17:08 +00:00
Ismael Carnales
26bcc826e3 renamed ItemField to Field for upcoming ItemExtractor and FieldExtractor 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40882
 2009-02-19 14:46:23 +00:00