BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-21 05:53:15 +00:00
Code Releases Wiki Activity
7,850 Commits 39 Branches 119 Tags
Commit Graph

7850 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
olveyra
0dffb68ecf not update nodes after scheduling, to avoid enter in a loop 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40100
 2008-07-23 19:13:39 +00:00
olveyra
b5b79042ab - added worker to master notifications. 
- deleted statistics code. will change approach

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4099
 2008-07-23 17:15:16 +00:00
olveyra
980369ba60 changed default webservice port to connect on to 8060 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4098
 2008-07-23 12:32:38 +00:00
olveyra
3a06868c21 deleted date folder from log path 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4097
 2008-07-23 12:21:05 +00:00
Pablo Hoffman
7e32fe10f2 do not hide svn errors 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4096
 2008-07-23 03:13:26 +00:00
Pablo Hoffman
8a0fdaa89b logfiles are now appendable instead of truncable 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4095
 2008-07-22 23:13:08 +00:00
samus_
a7301cc9a2 implemented __getslice__ for the XPathSelectorList, still broken when using step-slices 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4094
 2008-07-22 18:38:02 +00:00
olveyra
a27b085c04 fix 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4093
 2008-07-22 17:56:04 +00:00
samus_
5e2db86cde fixed pickle 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4092
 2008-07-22 17:53:37 +00:00
olveyra
7f62faea5f Added cluster statistics report 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4091
 2008-07-22 17:21:55 +00:00
samus_
85a4514603 reverted wrong XPathSelector commit 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4090
 2008-07-22 16:48:10 +00:00
samus_
e8b5a07a15 cPickle as pickle 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4089
 2008-07-22 16:44:30 +00:00
samus_
30202c54a4 ís_cached fix2 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4088
 2008-07-22 16:27:52 +00:00
samus_
e7fbaa50d8 reverted wrong XPathSelector commit 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4087
 2008-07-22 16:12:42 +00:00
samus_
c9851d8ec9 ís_cached fix 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4086
 2008-07-22 16:11:57 +00:00
samus_
5778ecf15d small fix for string representation of XPathSelector when the node is not an xml object 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4085
 2008-07-22 14:19:41 +00:00
samus_
97b2e7df55 added pickled_meta to avoid using eval 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4084
 2008-07-22 11:51:53 +00:00
samus_
76a9d2da10 migrated all sha to hashlib 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4083
 2008-07-22 11:33:05 +00:00
olveyra
ce905ce78f - code fixes 
- fix when rescheduling with a new priority
- added freeslots to status_as_dict
- remove a domain from loading list only when reported as running in
some node.

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4082
 2008-07-21 19:14:37 +00:00
samus_
9c345016b8 added INFO message 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4081
 2008-07-21 14:07:57 +00:00
samus_
d2121141a3 implemented CACHE2_EXPIRATION_SECS and migrated sha to hashlib 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4080
 2008-07-21 13:23:23 +00:00
Pablo Hoffman
b45d87d0fe removed duplicated code 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4079
 2008-07-18 01:51:49 +00:00
olveyra
cdb3b02510 fix 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4078
 2008-07-17 19:20:10 +00:00
olveyra
7c1ebfd16a option help fix 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4077
 2008-07-17 19:06:03 +00:00
olveyra
d765b133f0 - --default-spider option gives the default spider (old --spider option) 
- --spider option now forces to use the given spider domain when
arguments are urls

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4076
 2008-07-17 19:00:05 +00:00
olveyra
abde0e5f51 readed 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4075
 2008-07-17 18:15:46 +00:00
olveyra
fb43b935c6 deleted 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4074
 2008-07-17 18:14:52 +00:00
olveyra
6f028cc38a moved cluster-ctl script to scrapy branch 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4073
 2008-07-17 18:09:32 +00:00
olveyra
f515942f54 - added verbosity levels 
- now log paths includes a folder named by the date, so it is easier
to mantain logs in server

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4072
 2008-07-17 15:41:15 +00:00
olveyra
5403fd3d9f - added disable_node and enable_node functions 
- removed unused imports
- autoreload of lost nodes
- some code improvements and fixes

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4071
 2008-07-16 19:17:46 +00:00
Ezequiel Rivero
8150fd6a16 size fix for menu in trac 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4070
 2008-07-16 17:50:58 +00:00
olveyra
e8eab24dc0 don't update status in run callback, so to avoid lots of bouncing 
domains. The domains will be loaded softly on each node update

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4069
 2008-07-16 12:09:42 +00:00
olveyra
338a485bc1 - load only one domain per node (and load the following when the run 
callback is executed). This way, we avoid to load lot of domains that
will bounce. Also, we mix up better the domains between available nodes.

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4068
 2008-07-16 11:20:53 +00:00
olveyra
ace6e3c430 - Reschedule a domain that is already running or loading in some node 
- No schedule a domain that is already scheduled

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4067
 2008-07-15 18:53:37 +00:00
olveyra
69d1c57692 small fix and pepocho fix 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4066
 2008-07-15 18:06:14 +00:00
olveyra
e71aa06f97 Check the node will not run a domain that is already running 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4065
 2008-07-15 12:11:32 +00:00
olveyra
2b6189a3f8 renamed setting SCRAPY_PICKLED_SETTINGS to 
SCRAPY_PICKLED_SETTINGS_TO_OVERRIDE

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4064
 2008-07-14 14:10:26 +00:00
olveyra
11bc122a54 pickled settings must go in overrides settings not default, because of 
precedence

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4063
 2008-07-14 13:39:58 +00:00
Pablo Hoffman
49175d1990 added new test case for xpathselector_iternodes 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4062
 2008-07-14 13:31:00 +00:00
Pablo Hoffman
28b5bb3240 added utf-16 (and other encodings) support to xpathselector_iternodes 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4061
 2008-07-14 13:22:05 +00:00
Pablo Hoffman
90e93a7635 imports should be at module's top 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4060
 2008-07-14 13:18:58 +00:00
olveyra
9b3c3c14e5 logs extensions NotConfigured exception message 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4059
 2008-07-14 12:42:58 +00:00
olveyra
377e17b7db More data in log message 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4058
 2008-07-14 12:35:46 +00:00
olveyra
a4cfea54e8 fixed env name 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4057
 2008-07-11 19:34:39 +00:00
olveyra
1eb583a357 added log msg 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4056
 2008-07-11 19:31:08 +00:00
olveyra
ba0f87ed2c changed CLUSTER_WORKER_LOGDIR to CLUSTER_LOGDIR 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4055
 2008-07-11 18:27:19 +00:00
olveyra
585f35fb3a Cluster Master improvements: 
- rescheduling now goes with original priority decreased by one
- Added GLOBAL_CLUSTER_SETTINGS
- Added PB remote method load_node so the worker also can initiate a
connection

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4054
 2008-07-11 16:21:23 +00:00
olveyra
0c35b7ed2a Now worker pass settings to process via SCRAPY_PICKLED_SETTINGS 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4053
 2008-07-11 16:08:28 +00:00
olveyra
8ea2dc94d7 renamed pickled data 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4052
 2008-07-11 16:05:20 +00:00
olveyra
94993b769a added SCRAPY_PICKLED_DEFAULT_SETTINGS, a string to pass by environment 
setting an arbitrary set of settings via "pickle" python module.

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4051
 2008-07-11 15:59:08 +00:00