BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-26 12:43:48 +00:00
Code Releases Wiki Activity
1,016 Commits 39 Branches 119 Tags
Commit Graph

1016 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Daniel Grana
82dc57e59b headers: complete new Headers behaviour migration. closes #47 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40916
 2009-02-25 00:29:10 +00:00
Daniel Grana
5de0816d9f url: add custom port based testcase 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40915
 2009-02-25 00:27:34 +00:00
Daniel Grana
5d3489ac10 headers: add docstring and title prior encoding keys. refs #47 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40914
 2009-02-25 00:27:02 +00:00
Pablo Hoffman
987ceb540e fixed import error (thanks Shane) and added some basic tests for ScrapedItem 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40913
 2009-02-24 14:17:16 +00:00
Daniel Grana
934535740b headers: cleanup Headers class and add normvalue method to CaselessDict 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40912
 2009-02-24 06:58:12 +00:00
Daniel Grana
f4b55bfd20 datatypes: add CaselessDict test and cleanup 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40911
 2009-02-24 06:57:28 +00:00
Daniel Grana
d504a2b428 bugfix newitem adaptor imports 
--HG--
rename : scrapy/trunk/scrapy/tests/test_itemextractor.py => scrapy/trunk/scrapy/tests/test_itemadaptor.py
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40910
 2009-02-24 05:13:31 +00:00
Daniel Grana
119706ae23 headers: remove unused methods 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40909
 2009-02-24 05:12:58 +00:00
Ismael Carnales
5afa621d6a moved extractors.py to adaptors.py 
--HG--
rename : scrapy/trunk/scrapy/contrib_exp/newitem/extractors.py => scrapy/trunk/scrapy/contrib_exp/newitem/adaptors.py
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40908
 2009-02-24 04:04:25 +00:00
Ismael Carnales
8398212404 renamed ItemExtractor to ItemAdaptor 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40907
 2009-02-24 04:02:34 +00:00
Ismael Carnales
b26102745a minor code style changes to extractors.py 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40906
 2009-02-24 03:51:40 +00:00
Daniel Grana
53983e6e23 newitem: remove ExtractorField and port googledir example 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40905
 2009-02-24 03:42:21 +00:00
Daniel Grana
15909ef62f newitem: rename treeadapt as adaptor, and bugfix a mutable dictionary bug 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40904
 2009-02-24 03:41:44 +00:00
Damian Canabal
ebe106b7a2 check for dont_merge_cookies flag in request.meta 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40903
 2009-02-24 00:48:50 +00:00
Daniel Grana
b0624e23ed proper rename dupefilter test 
--HG--
rename : scrapy/trunk/scrapy/tests/test_filters.py => scrapy/trunk/scrapy/tests/test_dupefilter.py
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40902
 2009-02-23 20:27:02 +00:00
Daniel Grana
d1265a4199 move dupe filter code outside of core. refs #49 
--HG--
rename : scrapy/trunk/scrapy/core/filters.py => scrapy/trunk/scrapy/dupefilter/__init__.py
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40901
 2009-02-23 20:21:11 +00:00
Daniel Grana
c6326a0425 core: add a custom HTTPClientFactory to reuse url parsing, and remove malformed headers monkeypatches. closes #52 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40900
 2009-02-23 19:49:37 +00:00
Pablo Hoffman
86d0256815 added FAQ about HTTP proxies 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40899
 2009-02-21 21:26:10 +00:00
samus_
9d5a1d77b6 adding to_list() util and test 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40898
 2009-02-20 18:17:55 +00:00
Daniel Grana
8d8a5c5caa change ExtractorField callable signature to honor adaptor_args 
and add alternative treeadapt implementation.

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40897
 2009-02-20 17:59:58 +00:00
Pablo Hoffman
0c2e019f66 added test for load_object 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40896
 2009-02-20 17:35:25 +00:00
Ismael Carnales
059bf613fa added example project that uses newitem and ItemExtractor 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40895
 2009-02-20 12:02:56 +00:00
Ismael Carnales
5a5aef6629 made newitem.Item inherit from ScrapedItem for compatibility 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40894
 2009-02-20 11:42:36 +00:00
Daniel Grana
8d5222f1db core: add missing imports 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40893
 2009-02-20 11:31:09 +00:00
Daniel Grana
4bf196fc99 core: add scheduler middleware and move duplicate fitler there 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40892
 2009-02-20 11:28:37 +00:00
Daniel Grana
549c30d8d5 redirect: remove stupid set 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40891
 2009-02-20 11:27:46 +00:00
Daniel Grana
071aa71b68 add filters module 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40890
 2009-02-19 19:53:30 +00:00
Daniel Grana
0e98526daa duplicatesfilter: add a singeton duplicates filter and adapt current middleware and redirection middleware 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40889
 2009-02-19 19:43:24 +00:00
Daniel Grana
ac735a185e cleanup redirection middleware 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40888
 2009-02-19 19:42:42 +00:00
Ismael Carnales
65e8be46bc add typecheck of funcs in ExtractorField.__init__ 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40887
 2009-02-19 19:05:45 +00:00
Ismael Carnales
edf5dfb264 added newitem.extractors based on old adaptors 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40886
 2009-02-19 18:56:59 +00:00
Daniel Grana
756cda3873 duplicatefilter: dont raise ignorerequest, but add request to filter on spider input 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40885
 2009-02-19 16:49:03 +00:00
Daniel Grana
fe7935005c duplicatefilter: filter request prior to reach spider 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40884
 2009-02-19 16:37:33 +00:00
Daniel Grana
5338a48db8 remove missing import 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40883
 2009-02-19 16:17:08 +00:00
Ismael Carnales
26bcc826e3 renamed ItemField to Field for upcoming ItemExtractor and FieldExtractor 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40882
 2009-02-19 14:46:23 +00:00
Daniel Grana
7fc8d590f3 guid: cleanup guid attribute references including removal of replays and deprecated commands 
--HG--
rename : scrapy/trunk/scrapy/contrib/history/__init__.py => scrapy/trunk/scrapy/contrib_exp/history/__init__.py
rename : scrapy/trunk/scrapy/contrib/history/history.py => scrapy/trunk/scrapy/contrib_exp/history/history.py
rename : scrapy/trunk/scrapy/contrib/history/middleware.py => scrapy/trunk/scrapy/contrib_exp/history/middleware.py
rename : scrapy/trunk/scrapy/contrib/history/scheduler.py => scrapy/trunk/scrapy/contrib_exp/history/scheduler.py
rename : scrapy/trunk/scrapy/contrib/history/store.py => scrapy/trunk/scrapy/contrib_exp/history/store.py
rename : scrapy/trunk/scrapy/contrib/pipeline/shoveitem.py => scrapy/trunk/scrapy/contrib_exp/pipeline/shoveitem.py
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40881
 2009-02-19 14:22:23 +00:00
Ismael Carnales
81ce9bd458 calculate ItemField.default on __init__ 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40880
 2009-02-19 13:57:40 +00:00
Ismael Carnales
db4ccaf78f add __all__ to fields.py in newitem 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40879
 2009-02-19 13:34:02 +00:00
Ismael Carnales
5c45ecb7ff turned default_value() of ItemField into a property 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40878
 2009-02-19 12:45:54 +00:00
Ismael Carnales
76c84bda38 always deiter on ItemField value assignation 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40877
 2009-02-19 12:39:58 +00:00
Daniel Grana
22549161a5 duplicatefilter: fix unittest to filter a request with same url as start_requests. rel r868. ref #49 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40876
 2009-02-19 11:55:09 +00:00
Ismael Carnales
1017f12163 don't allow setting attributes that aren't fields, and return field default values on newitem 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40875
 2009-02-19 11:40:21 +00:00
Ismael Carnales
d95878c0f7 corrected import paths in newitem 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40874
 2009-02-19 11:26:01 +00:00
Pablo Hoffman
c11e5ad3dd fixed grammar error. thanks Fabio 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40873
 2009-02-19 11:07:41 +00:00
samus_
c546585af0 conflict solved, reverted r869 and applied changes for r868 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40872
 2009-02-19 05:12:55 +00:00
samus_
fefccfaa31 conflict among tests, reverting 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40871
 2009-02-19 04:59:12 +00:00
samus_
14900a7452 fix to default parameter (must be tuple not string) 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40870
 2009-02-19 04:21:49 +00:00
samus_
de13bbe7f5 fix to filter start_urls too 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40869
 2009-02-19 04:19:05 +00:00
Ismael Carnales
9739f9320e move newitem from contrib to contrib.exp 
--HG--
rename : scrapy/trunk/scrapy/contrib/newitem/__init__.py => scrapy/trunk/scrapy/contrib_exp/newitem/__init__.py
rename : scrapy/trunk/scrapy/contrib/newitem/fields.py => scrapy/trunk/scrapy/contrib_exp/newitem/fields.py
rename : scrapy/trunk/scrapy/contrib/newitem/models.py => scrapy/trunk/scrapy/contrib_exp/newitem/models.py
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40868
 2009-02-18 18:57:26 +00:00
Ismael Carnales
3039453148 added newitem with new item model and fields 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40867
 2009-02-18 18:38:26 +00:00