BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-26 23:44:01 +00:00
Code Releases Wiki Activity
2,659 Commits 39 Branches 119 Tags
Commit Graph

2659 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Matias Aguirre
4c193f47ed Adding if clause that will display blog entries box only if there are any entry to list 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40109
 2008-07-25 15:19:16 +00:00
Matias Aguirre
56e110af78 Override admin base template with a custom one that has some JS to resize textaeras, useful when editing an article 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40108
 2008-07-25 15:17:43 +00:00
Matias Aguirre
ee8d9a594a Adding nav and footer links statically instead of use the removed links application 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40107
 2008-07-25 15:16:37 +00:00
Matias Aguirre
202e08a9d3 Removing flatpages from applications list, this isn't used 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40106
 2008-07-25 15:11:56 +00:00
Matias Aguirre
d4337610d9 Removing link application, was too much overkill to the functionality needed 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40105
 2008-07-25 15:10:41 +00:00
Pablo Hoffman
9dd144f487 removed parent attribute from Response class 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40104
 2008-07-25 13:42:40 +00:00
olveyra
f3f34d0816 - added basic statistics (needs to be improved) 
- node remotion when connection lost event

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40103
 2008-07-24 15:18:29 +00:00
samus_
e86ee64727 small improvements to the cache 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40102
 2008-07-24 01:52:28 +00:00
olveyra
8b8e3e7532 fixed a bug in which pending list remains stalled because not proper 
clean out of loading list

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40101
 2008-07-23 19:23:47 +00:00
olveyra
0dffb68ecf not update nodes after scheduling, to avoid enter in a loop 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40100
 2008-07-23 19:13:39 +00:00
olveyra
b5b79042ab - added worker to master notifications. 
- deleted statistics code. will change approach

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4099
 2008-07-23 17:15:16 +00:00
olveyra
980369ba60 changed default webservice port to connect on to 8060 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4098
 2008-07-23 12:32:38 +00:00
olveyra
3a06868c21 deleted date folder from log path 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4097
 2008-07-23 12:21:05 +00:00
Pablo Hoffman
7e32fe10f2 do not hide svn errors 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4096
 2008-07-23 03:13:26 +00:00
Pablo Hoffman
8a0fdaa89b logfiles are now appendable instead of truncable 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4095
 2008-07-22 23:13:08 +00:00
samus_
a7301cc9a2 implemented __getslice__ for the XPathSelectorList, still broken when using step-slices 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4094
 2008-07-22 18:38:02 +00:00
olveyra
a27b085c04 fix 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4093
 2008-07-22 17:56:04 +00:00
samus_
5e2db86cde fixed pickle 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4092
 2008-07-22 17:53:37 +00:00
olveyra
7f62faea5f Added cluster statistics report 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4091
 2008-07-22 17:21:55 +00:00
samus_
85a4514603 reverted wrong XPathSelector commit 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4090
 2008-07-22 16:48:10 +00:00
samus_
e8b5a07a15 cPickle as pickle 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4089
 2008-07-22 16:44:30 +00:00
samus_
30202c54a4 ís_cached fix2 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4088
 2008-07-22 16:27:52 +00:00
samus_
e7fbaa50d8 reverted wrong XPathSelector commit 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4087
 2008-07-22 16:12:42 +00:00
samus_
c9851d8ec9 ís_cached fix 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4086
 2008-07-22 16:11:57 +00:00
samus_
5778ecf15d small fix for string representation of XPathSelector when the node is not an xml object 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4085
 2008-07-22 14:19:41 +00:00
samus_
97b2e7df55 added pickled_meta to avoid using eval 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4084
 2008-07-22 11:51:53 +00:00
samus_
76a9d2da10 migrated all sha to hashlib 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4083
 2008-07-22 11:33:05 +00:00
olveyra
ce905ce78f - code fixes 
- fix when rescheduling with a new priority
- added freeslots to status_as_dict
- remove a domain from loading list only when reported as running in
some node.

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4082
 2008-07-21 19:14:37 +00:00
samus_
9c345016b8 added INFO message 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4081
 2008-07-21 14:07:57 +00:00
samus_
d2121141a3 implemented CACHE2_EXPIRATION_SECS and migrated sha to hashlib 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4080
 2008-07-21 13:23:23 +00:00
Pablo Hoffman
b45d87d0fe removed duplicated code 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4079
 2008-07-18 01:51:49 +00:00
olveyra
cdb3b02510 fix 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4078
 2008-07-17 19:20:10 +00:00
olveyra
7c1ebfd16a option help fix 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4077
 2008-07-17 19:06:03 +00:00
olveyra
d765b133f0 - --default-spider option gives the default spider (old --spider option) 
- --spider option now forces to use the given spider domain when
arguments are urls

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4076
 2008-07-17 19:00:05 +00:00
olveyra
abde0e5f51 readed 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4075
 2008-07-17 18:15:46 +00:00
olveyra
fb43b935c6 deleted 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4074
 2008-07-17 18:14:52 +00:00
olveyra
6f028cc38a moved cluster-ctl script to scrapy branch 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4073
 2008-07-17 18:09:32 +00:00
olveyra
f515942f54 - added verbosity levels 
- now log paths includes a folder named by the date, so it is easier
to mantain logs in server

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4072
 2008-07-17 15:41:15 +00:00
olveyra
5403fd3d9f - added disable_node and enable_node functions 
- removed unused imports
- autoreload of lost nodes
- some code improvements and fixes

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4071
 2008-07-16 19:17:46 +00:00
Ezequiel Rivero
8150fd6a16 size fix for menu in trac 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4070
 2008-07-16 17:50:58 +00:00
olveyra
e8eab24dc0 don't update status in run callback, so to avoid lots of bouncing 
domains. The domains will be loaded softly on each node update

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4069
 2008-07-16 12:09:42 +00:00
olveyra
338a485bc1 - load only one domain per node (and load the following when the run 
callback is executed). This way, we avoid to load lot of domains that
will bounce. Also, we mix up better the domains between available nodes.

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4068
 2008-07-16 11:20:53 +00:00
olveyra
ace6e3c430 - Reschedule a domain that is already running or loading in some node 
- No schedule a domain that is already scheduled

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4067
 2008-07-15 18:53:37 +00:00
olveyra
69d1c57692 small fix and pepocho fix 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4066
 2008-07-15 18:06:14 +00:00
olveyra
e71aa06f97 Check the node will not run a domain that is already running 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4065
 2008-07-15 12:11:32 +00:00
olveyra
2b6189a3f8 renamed setting SCRAPY_PICKLED_SETTINGS to 
SCRAPY_PICKLED_SETTINGS_TO_OVERRIDE

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4064
 2008-07-14 14:10:26 +00:00
olveyra
11bc122a54 pickled settings must go in overrides settings not default, because of 
precedence

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4063
 2008-07-14 13:39:58 +00:00
Pablo Hoffman
49175d1990 added new test case for xpathselector_iternodes 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4062
 2008-07-14 13:31:00 +00:00
Pablo Hoffman
28b5bb3240 added utf-16 (and other encodings) support to xpathselector_iternodes 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4061
 2008-07-14 13:22:05 +00:00
Pablo Hoffman
90e93a7635 imports should be at module's top 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4060
 2008-07-14 13:18:58 +00:00