BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-26 17:04:07 +00:00
Code Releases Wiki Activity
931 Commits 39 Branches 119 Tags
Commit Graph

931 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Daniel Grana
7fc8d590f3 guid: cleanup guid attribute references including removal of replays and deprecated commands 
--HG--
rename : scrapy/trunk/scrapy/contrib/history/__init__.py => scrapy/trunk/scrapy/contrib_exp/history/__init__.py
rename : scrapy/trunk/scrapy/contrib/history/history.py => scrapy/trunk/scrapy/contrib_exp/history/history.py
rename : scrapy/trunk/scrapy/contrib/history/middleware.py => scrapy/trunk/scrapy/contrib_exp/history/middleware.py
rename : scrapy/trunk/scrapy/contrib/history/scheduler.py => scrapy/trunk/scrapy/contrib_exp/history/scheduler.py
rename : scrapy/trunk/scrapy/contrib/history/store.py => scrapy/trunk/scrapy/contrib_exp/history/store.py
rename : scrapy/trunk/scrapy/contrib/pipeline/shoveitem.py => scrapy/trunk/scrapy/contrib_exp/pipeline/shoveitem.py
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40881
 2009-02-19 14:22:23 +00:00
Ismael Carnales
81ce9bd458 calculate ItemField.default on __init__ 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40880
 2009-02-19 13:57:40 +00:00
Ismael Carnales
db4ccaf78f add __all__ to fields.py in newitem 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40879
 2009-02-19 13:34:02 +00:00
Ismael Carnales
5c45ecb7ff turned default_value() of ItemField into a property 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40878
 2009-02-19 12:45:54 +00:00
Ismael Carnales
76c84bda38 always deiter on ItemField value assignation 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40877
 2009-02-19 12:39:58 +00:00
Daniel Grana
22549161a5 duplicatefilter: fix unittest to filter a request with same url as start_requests. rel r868. ref #49 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40876
 2009-02-19 11:55:09 +00:00
Ismael Carnales
1017f12163 don't allow setting attributes that aren't fields, and return field default values on newitem 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40875
 2009-02-19 11:40:21 +00:00
Ismael Carnales
d95878c0f7 corrected import paths in newitem 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40874
 2009-02-19 11:26:01 +00:00
Pablo Hoffman
c11e5ad3dd fixed grammar error. thanks Fabio 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40873
 2009-02-19 11:07:41 +00:00
samus_
c546585af0 conflict solved, reverted r869 and applied changes for r868 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40872
 2009-02-19 05:12:55 +00:00
samus_
fefccfaa31 conflict among tests, reverting 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40871
 2009-02-19 04:59:12 +00:00
samus_
14900a7452 fix to default parameter (must be tuple not string) 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40870
 2009-02-19 04:21:49 +00:00
samus_
de13bbe7f5 fix to filter start_urls too 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40869
 2009-02-19 04:19:05 +00:00
Ismael Carnales
9739f9320e move newitem from contrib to contrib.exp 
--HG--
rename : scrapy/trunk/scrapy/contrib/newitem/__init__.py => scrapy/trunk/scrapy/contrib_exp/newitem/__init__.py
rename : scrapy/trunk/scrapy/contrib/newitem/fields.py => scrapy/trunk/scrapy/contrib_exp/newitem/fields.py
rename : scrapy/trunk/scrapy/contrib/newitem/models.py => scrapy/trunk/scrapy/contrib_exp/newitem/models.py
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40868
 2009-02-18 18:57:26 +00:00
Ismael Carnales
3039453148 added newitem with new item model and fields 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40867
 2009-02-18 18:38:26 +00:00
samus_
918908db8e removed empty dir 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40866
 2009-02-18 15:22:15 +00:00
Ismael Carnales
f8affd0c3d added fields.py with ItemField class and subclasses based on Django's for new items 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40865
 2009-02-18 13:36:47 +00:00
Ismael Carnales
b20ac057b3 added rule shorthand, for creating CrawlSpider rules 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40864
 2009-02-18 11:18:11 +00:00
Daniel Grana
84f63b146b cluster: merge spider settings prior to run spider, not when scheduling 
also reformat some long lines.

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40863
 2009-02-17 14:20:19 +00:00
Pablo Hoffman
714c3e20e4 minor typo fix 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40862
 2009-02-16 23:45:14 +00:00
Pablo Hoffman
489e86bd16 moved copytree function to scrapy.utils.python (more appropiate location) and fixed minor bug ('Error' not defined). also removed unused var 'project_module_path' 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40861
 2009-02-16 23:43:43 +00:00
Ismael Carnales
e6458d057b completed first version of basic tutorial 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40860
 2009-02-16 16:42:35 +00:00
Ismael Carnales
730575a53f define WindowsError (also in shutils from 2.6) 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40859
 2009-02-16 13:30:20 +00:00
Ismael Carnales
5c1dd25284 removed spiders cache from example project, thxs Michael 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40858
 2009-02-16 13:15:55 +00:00
Ismael Carnales
3e8bcafd8b added copytree from python 2.6 to utils.misc and make startproject use it to ignore .svn and .pyc files 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40857
 2009-02-16 12:50:00 +00:00
Ismael Carnales
522fe78c2b remove old project example 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40856
 2009-02-16 11:52:49 +00:00
Ismael Carnales
bb9a732edf updated tutorial with new googledir project from r853 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40855
 2009-02-16 11:51:07 +00:00
Ismael Carnales
eb1e62a28c add new googledir example project with new structure 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40854
 2009-02-16 11:49:44 +00:00
Daniel Grana
755235b568 utils: renamed load_class function as load_object 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40853
 2009-02-13 17:21:50 +00:00
Daniel Grana
3876a1827f cluster: move branched cluster as experimental contrib 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40852
 2009-02-13 17:20:10 +00:00
Pablo Hoffman
47cf45c916 removed redudant part of Scrapy introduction to make it simpler. thanks Ismael for pointing that out 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40851
 2009-02-12 20:58:42 +00:00
Daniel Grana
8557f6bc57 duplicatefilter: lower log level of skipped requests message 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40850
 2009-02-12 15:55:11 +00:00
Daniel Grana
2b14251510 cache: read metadata only when when looking for cached items. refs #61 
thanks Patrick Mezard for patch.

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40849
 2009-02-12 08:00:07 +00:00
Daniel Grana
d68646422d storedb: gracefully fail test if mysql is not installed. 
thanks Patrick Mezard for patch.

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40848
 2009-02-12 07:59:40 +00:00
Daniel Grana
3e4bc6141a core: remove obsolte groupfilter code 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40847
 2009-02-12 07:39:57 +00:00
Daniel Grana
c9f2865c83 core: Get rid of duplicate filtering as a scheduler builtin feature. closes #49. 
Implements a DuplicatesFilterMiddleware as spidermiddleware, a wraper
using a minimal defined API of a filtering class configurable by
settings.

Enabling this middleware doesn't gives us same functionality compared to
scheduler duplicate filter builtin, but it filter the most important
source for duplicate requests, the spiders.

What requests aren't filtered by new middleware? The ones originated
from any part of scrapy outside of spiders, like S3 images requests or
any other request manually schedule using ``scrapyengine.schedule()``
method.

Previously, we usually added dont_filter=True to requests created
outside of spiders to avoid collisions downloading same pages than
spider. Now, this is not required anymore because new middleware filters
just the spider generated requests.

There is a caveat, as usual downloadmiddlewares can returns a Request
object at any point of the chain, and that request is scheduled and
downloaded as usual too. One of the downloadmiddlewares using this
feature is RedirectMiddleware that counts on scheduler filtering builtin
to avoid redirection loops. I think we can implement a request time to
live decreasing counter and add it to request's ``meta`` attribute with
a default value if not present, and decrement each time the request is
redirected.

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40846
 2009-02-12 07:07:13 +00:00
Daniel Grana
0aba276a64 tutorial: indent class docstring 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40845
 2009-02-12 04:39:49 +00:00
Daniel Grana
73d8177ecc tutorial: lot of line wrapping and changes to double backticks instead of emphatized words 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40844
 2009-02-12 04:38:13 +00:00
Daniel Grana
7c056c620e tutorial: fix outofdate and broken tutorial after adaptors were moved to experimental 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40843
 2009-02-12 03:58:54 +00:00
Daniel Grana
f43c58da1f adds a reference about empty request body and Content-Length header fix, closes #60. 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40842
 2009-02-10 18:27:59 +00:00
Daniel Grana
8fdf06ed08 Avoid sending Content-Length header when body is an empty string. 
Some sites can't handle "Content-Length: 0" header, but twisted
HTTPClientFactory adds Content-Length header unless body is None.

scrapy enforces request.body usage as string, using None is not
possible.

thanks Matt for report, See:
http://groups.google.com/group/scrapy-users/browse_thread/thread/380ffa111879989e?hl=en

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40841
 2009-02-10 13:23:38 +00:00
Pablo Hoffman
f4224be411 - added get_meta_refresh to scrapy.utils.response 
- added tests for get_meta_refresh_url
- added RedirectMiddleware tests

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40840
 2009-02-10 06:20:43 +00:00
Pablo Hoffman
d376b20996 minor fix to doctest 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40839
 2009-02-10 04:46:07 +00:00
Pablo Hoffman
90c1597cef fixed bug with items_to_csv and Excel. thanks Mat for the patch. See: 
http://groups.google.com/group/scrapy-developers/browse_thread/thread/9e2b80d226333011

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40838
 2009-02-10 02:43:19 +00:00
Daniel Grana
1b2de80124 remove scrapy/dos external referenced from scrapy.org site 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40837
 2009-02-09 17:20:41 +00:00
Ismael Carnales
844b59b5b3 reduced introduction text in proposed doc 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40836
 2009-02-09 15:06:22 +00:00
Ismael Carnales
d98f35af94 added items section (ScrapedItems) to proposed doc 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40835
 2009-02-09 14:29:48 +00:00
Ismael Carnales
3a3723ba2e removed reference to guid in ScrapedItem 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40834
 2009-02-09 13:59:15 +00:00
Pablo Hoffman
38c62ca357 settings must be documented in alphabetical order 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40833
 2009-02-08 21:00:26 +00:00
Ismael Carnales
45340d3d87 better index for proposed documentation 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40832
 2009-02-06 20:19:17 +00:00