BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-22 23:03:52 +00:00
Code Releases Wiki Activity
137 Commits 39 Branches 119 Tags
Commit Graph

137 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
elpolilla
c76200561d bugfix in scraping report 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40137
 2008-07-30 11:20:22 +00:00
Matias Aguirre
7b0877c50e Changes: 
* Simplify article app, it isn't necessary to save them
      in db, instead this tool should render static templates
      directly based in the url.
      Example: if the url is "/article/today" it will look for
      the template "articles/today.html" in articles templates
      directory. This app is configured to handle any url, so
      it will render an url like "/about" (if there isn't other
      url defined to handle "about" before article definition),
      and in this case will try to render the template
      "article.html" in articles templates dir

    * Removed models, not necessary now

    * Removed templatetags, not necessary now

    * Removed flatpages middleware ??

    * Added url to articles app, this will used as a last case
      to handle undefined urls.

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40136
 2008-07-29 13:54:20 +00:00
Daniel Grana
28bb53fa22 process result's items using generators to give pipeline a chance to consume while parsing new items 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40135
 2008-07-29 12:27:55 +00:00
elpolilla
640e8b9131 scrapy report util improved 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40134
 2008-07-29 01:14:01 +00:00
olveyra
02b87f7d49 in settings template, set DEFAULT_ITEM_CLASS to 
scrapy.item.ScrapedItem to enable console

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40133
 2008-07-28 22:29:55 +00:00
olveyra
1cbfe46161 disabled webconsole by default 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40132
 2008-07-28 22:11:19 +00:00
elpolilla
ea9571aaa0 --report option modified (i had forgotten to report the variants) 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40131
 2008-07-28 13:01:34 +00:00
elpolilla
f86324fac3 --report option added to crawl command 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40130
 2008-07-28 12:42:00 +00:00
Pablo Hoffman
4b2e20abfd added scrapy.utils.markup module 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40129
 2008-07-28 04:15:51 +00:00
olveyra
c9c624dd66 minor adjustments 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40128
 2008-07-27 22:00:08 +00:00
olveyra
86c7f37d6a moved options nocache and nopipeline from decobot to scrapy 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40127
 2008-07-27 16:11:23 +00:00
olveyra
b59f62dc91 Added function convert_entity from decobot.utils.text_extraction 
(to complete and fix revision 124)

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40126
 2008-07-27 14:54:11 +00:00
olveyra
1e2ddb47a3 remove wrong import from decobot and added unquote_html 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40125
 2008-07-27 02:49:20 +00:00
olveyra
137ec64318 minor adjustments and some fixes, readded scrapy-admin.py with 
execution permission

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40124
 2008-07-27 02:23:14 +00:00
olveyra
b851ec5f10 deleted scrapy-admin to commit again with execution access 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40123
 2008-07-27 01:38:53 +00:00
samus_
fbb5860f49 re-enabling replays with the new mechanism 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40122
 2008-07-26 15:51:39 +00:00
olveyra
bc00f8cce2 re-reverted commit 119 back again to 118. 
The code removed is confusing.

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40121
 2008-07-25 22:42:57 +00:00
Pablo Hoffman
67ee6bff2e restored code removed in r118. there's nothing wrong with that code 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40120
 2008-07-25 21:28:49 +00:00
olveyra
c97c495d16 removed code that generates confusing and mistaken import error 
message when the import error raises inside scrapy_settings.

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40119
 2008-07-25 19:40:43 +00:00
samus_
0bb68f34f6 forgot to import sys module :P 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40118
 2008-07-25 19:26:46 +00:00
olveyra
8e18ecd5ce first version of scrapy-admin 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40117
 2008-07-25 19:11:01 +00:00
samus_
bd7c80ed0a adding new replay method (beta) 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40116
 2008-07-25 18:50:45 +00:00
Matias Aguirre
22e3d3a02e Add save_on_top to admin sections 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40115
 2008-07-25 17:36:34 +00:00
Matias Aguirre
e421206573 Add publish field 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40114
 2008-07-25 16:59:27 +00:00
Matias Aguirre
9f8ce8f116 Add bool icon for public field in admin 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40113
 2008-07-25 16:59:00 +00:00
Matias Aguirre
baee62cf44 Change positions urls in download and article applications, this are like admin urls now and the views are decorated to force a staff user check before 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40112
 2008-07-25 15:38:11 +00:00
Matias Aguirre
fd4aab5f33 Force all textareas to be 30 rows height 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40111
 2008-07-25 15:35:11 +00:00
Matias Aguirre
80a99dde83 Adding missing template sections 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40110
 2008-07-25 15:23:59 +00:00
Matias Aguirre
4c193f47ed Adding if clause that will display blog entries box only if there are any entry to list 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40109
 2008-07-25 15:19:16 +00:00
Matias Aguirre
56e110af78 Override admin base template with a custom one that has some JS to resize textaeras, useful when editing an article 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40108
 2008-07-25 15:17:43 +00:00
Matias Aguirre
ee8d9a594a Adding nav and footer links statically instead of use the removed links application 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40107
 2008-07-25 15:16:37 +00:00
Matias Aguirre
202e08a9d3 Removing flatpages from applications list, this isn't used 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40106
 2008-07-25 15:11:56 +00:00
Matias Aguirre
d4337610d9 Removing link application, was too much overkill to the functionality needed 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40105
 2008-07-25 15:10:41 +00:00
Pablo Hoffman
9dd144f487 removed parent attribute from Response class 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40104
 2008-07-25 13:42:40 +00:00
olveyra
f3f34d0816 - added basic statistics (needs to be improved) 
- node remotion when connection lost event

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40103
 2008-07-24 15:18:29 +00:00
samus_
e86ee64727 small improvements to the cache 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40102
 2008-07-24 01:52:28 +00:00
olveyra
8b8e3e7532 fixed a bug in which pending list remains stalled because not proper 
clean out of loading list

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40101
 2008-07-23 19:23:47 +00:00
olveyra
0dffb68ecf not update nodes after scheduling, to avoid enter in a loop 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40100
 2008-07-23 19:13:39 +00:00
olveyra
b5b79042ab - added worker to master notifications. 
- deleted statistics code. will change approach

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4099
 2008-07-23 17:15:16 +00:00
olveyra
980369ba60 changed default webservice port to connect on to 8060 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4098
 2008-07-23 12:32:38 +00:00
olveyra
3a06868c21 deleted date folder from log path 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4097
 2008-07-23 12:21:05 +00:00
Pablo Hoffman
7e32fe10f2 do not hide svn errors 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4096
 2008-07-23 03:13:26 +00:00
Pablo Hoffman
8a0fdaa89b logfiles are now appendable instead of truncable 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4095
 2008-07-22 23:13:08 +00:00
samus_
a7301cc9a2 implemented __getslice__ for the XPathSelectorList, still broken when using step-slices 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4094
 2008-07-22 18:38:02 +00:00
olveyra
a27b085c04 fix 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4093
 2008-07-22 17:56:04 +00:00
samus_
5e2db86cde fixed pickle 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4092
 2008-07-22 17:53:37 +00:00
olveyra
7f62faea5f Added cluster statistics report 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4091
 2008-07-22 17:21:55 +00:00
samus_
85a4514603 reverted wrong XPathSelector commit 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4090
 2008-07-22 16:48:10 +00:00
samus_
e8b5a07a15 cPickle as pickle 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4089
 2008-07-22 16:44:30 +00:00
samus_
30202c54a4 ís_cached fix2 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4088
 2008-07-22 16:27:52 +00:00