BackTrackZ/scrapy
BackTrackZ/scrapy
1
0
Fork 0
mirror of https://github.com/scrapy/scrapy.git synced 2025-02-22 18:03:51 +00:00
Code Releases Wiki Activity
71 Commits 39 Branches 119 Tags
Commit Graph

71 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
olveyra
5403fd3d9f - added disable_node and enable_node functions 
- removed unused imports
- autoreload of lost nodes
- some code improvements and fixes

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4071
 2008-07-16 19:17:46 +00:00
Ezequiel Rivero
8150fd6a16 size fix for menu in trac 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4070
 2008-07-16 17:50:58 +00:00
olveyra
e8eab24dc0 don't update status in run callback, so to avoid lots of bouncing 
domains. The domains will be loaded softly on each node update

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4069
 2008-07-16 12:09:42 +00:00
olveyra
338a485bc1 - load only one domain per node (and load the following when the run 
callback is executed). This way, we avoid to load lot of domains that
will bounce. Also, we mix up better the domains between available nodes.

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4068
 2008-07-16 11:20:53 +00:00
olveyra
ace6e3c430 - Reschedule a domain that is already running or loading in some node 
- No schedule a domain that is already scheduled

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4067
 2008-07-15 18:53:37 +00:00
olveyra
69d1c57692 small fix and pepocho fix 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4066
 2008-07-15 18:06:14 +00:00
olveyra
e71aa06f97 Check the node will not run a domain that is already running 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4065
 2008-07-15 12:11:32 +00:00
olveyra
2b6189a3f8 renamed setting SCRAPY_PICKLED_SETTINGS to 
SCRAPY_PICKLED_SETTINGS_TO_OVERRIDE

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4064
 2008-07-14 14:10:26 +00:00
olveyra
11bc122a54 pickled settings must go in overrides settings not default, because of 
precedence

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4063
 2008-07-14 13:39:58 +00:00
Pablo Hoffman
49175d1990 added new test case for xpathselector_iternodes 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4062
 2008-07-14 13:31:00 +00:00
Pablo Hoffman
28b5bb3240 added utf-16 (and other encodings) support to xpathselector_iternodes 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4061
 2008-07-14 13:22:05 +00:00
Pablo Hoffman
90e93a7635 imports should be at module's top 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4060
 2008-07-14 13:18:58 +00:00
olveyra
9b3c3c14e5 logs extensions NotConfigured exception message 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4059
 2008-07-14 12:42:58 +00:00
olveyra
377e17b7db More data in log message 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4058
 2008-07-14 12:35:46 +00:00
olveyra
a4cfea54e8 fixed env name 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4057
 2008-07-11 19:34:39 +00:00
olveyra
1eb583a357 added log msg 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4056
 2008-07-11 19:31:08 +00:00
olveyra
ba0f87ed2c changed CLUSTER_WORKER_LOGDIR to CLUSTER_LOGDIR 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4055
 2008-07-11 18:27:19 +00:00
olveyra
585f35fb3a Cluster Master improvements: 
- rescheduling now goes with original priority decreased by one
- Added GLOBAL_CLUSTER_SETTINGS
- Added PB remote method load_node so the worker also can initiate a
connection

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4054
 2008-07-11 16:21:23 +00:00
olveyra
0c35b7ed2a Now worker pass settings to process via SCRAPY_PICKLED_SETTINGS 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4053
 2008-07-11 16:08:28 +00:00
olveyra
8ea2dc94d7 renamed pickled data 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4052
 2008-07-11 16:05:20 +00:00
olveyra
94993b769a added SCRAPY_PICKLED_DEFAULT_SETTINGS, a string to pass by environment 
setting an arbitrary set of settings via "pickle" python module.

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4051
 2008-07-11 15:59:08 +00:00
Daniel Grana
c97e7d0daf show if crawled pages are cached or live versions 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4050
 2008-07-08 19:48:12 +00:00
olveyra
805eeec844 deleted unneeded log 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4049
 2008-07-08 18:08:20 +00:00
olveyra
fc4e0eaa2a fix a bug with rescheduling a domain when no free slot are in worker 
(a domain was passed instead of a list, and that raises the one-letter
domains bug), also reschedule with priority=0)

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4048
 2008-07-08 18:03:20 +00:00
olveyra
863d94adec added CLUSTER_MASTER_CACHEFILE setting, changed SVN_WORKDIR to 
CLUSTER_WORKER_SVNWORKDIR

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4047
 2008-07-07 18:00:35 +00:00
olveyra
e78f7a93bd better way to stop engine with process signals 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4046
 2008-07-07 17:51:09 +00:00
Pablo Hoffman
648b312bca engine.stop() is now always executed when reactor stops 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4045
 2008-07-07 17:37:55 +00:00
olveyra
6bdbcd8173 pep8 pal pablo 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4044
 2008-07-04 19:02:19 +00:00
olveyra
1d876db80b pass to crawl process the correct python path 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4043
 2008-07-04 18:59:36 +00:00
olveyra
6bbbb0c1f6 master logs response from worker. 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4042
 2008-07-04 13:23:21 +00:00
olveyra
e11d824ed0 added --pidfile option 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4041
 2008-07-04 13:22:27 +00:00
olveyra
23b3408403 - added svn update support 
- removed passing of env variables
- added automatic group settings load

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4040
 2008-07-02 19:42:36 +00:00
olveyra
8344819a92 added persistence of pending list in master 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4039
 2008-07-02 15:07:29 +00:00
olveyra
9c713529e4 fix and webservice api doc update 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4038
 2008-07-01 15:28:07 +00:00
olveyra
74ef3ab5b0 added environment options 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4037
 2008-07-01 15:17:00 +00:00
olveyra
d2684dc17b Ok, first functional version of pbcluster 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4036
 2008-07-01 13:17:23 +00:00
Pablo Hoffman
d993f493b5 some improvments to XPathSelectors: 
- x() method now returns the same XPathSelector type of its parent
- added tests to check this
- added tests to verify that XML and HTML XPathSelector behave differently when
  parsing some non trivial markup

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4035
 2008-07-01 02:21:12 +00:00
Pablo Hoffman
1f7f0d0994 replaced XMLNodeIterator with xpathselector_iternodes 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4034
 2008-06-30 17:20:56 +00:00
Ezequiel Rivero
3ee14952aa css styling changes. 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4033
 2008-06-30 15:07:27 +00:00
olveyra
8a81fe82c7 deleted prints 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4032
 2008-06-30 14:24:49 +00:00
olveyra
53838d6a7d pbcluster commit 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4031
 2008-06-30 14:20:27 +00:00
Pablo Hoffman
4caadf6b67 added (yet another) xml node iterator based entirely in regex 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4030
 2008-06-29 06:08:48 +00:00
Pablo Hoffman
f9fc8a1b65 exported XmlXPathSelector and HtmlXPathSelector in scrapy.xpath, changed scrape command to instantiate those 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4029
 2008-06-29 05:34:28 +00:00
Pablo Hoffman
6cc91df9ed some improvements to XPathSelector and friends 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4028
 2008-06-29 05:20:31 +00:00
Pablo Hoffman
9cf83faf1c moved outside scrapy 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4027
 2008-06-29 01:10:44 +00:00
Pablo Hoffman
1638ef2b7a removed link_extraction and text_extraction modules test 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4026
 2008-06-29 01:08:23 +00:00
Pablo Hoffman
83dcf8aff9 commited initial scrapy code, taken from the old repo at r31560 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4025
 2008-06-28 23:37:28 +00:00
Pablo Hoffman
37ca8701eb removed unused dir dev.scrapy.org 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4024
 2008-06-28 22:57:47 +00:00
Pablo Hoffman
729499910e moved dev.scrapy.org/htdocs to static.scrapy.org 
--HG--
rename : sites/dev.scrapy.org/htdocs/asc.png => sites/static.scrapy.org/asc.png
rename : sites/dev.scrapy.org/htdocs/attachment.png => sites/static.scrapy.org/attachment.png
rename : sites/dev.scrapy.org/htdocs/changeset.png => sites/static.scrapy.org/changeset.png
rename : sites/dev.scrapy.org/htdocs/closedticket.png => sites/static.scrapy.org/closedticket.png
rename : sites/dev.scrapy.org/htdocs/collapsed.png => sites/static.scrapy.org/collapsed.png
rename : sites/dev.scrapy.org/htdocs/css/about.css => sites/static.scrapy.org/css/about.css
rename : sites/dev.scrapy.org/htdocs/css/admin.css => sites/static.scrapy.org/css/admin.css
rename : sites/dev.scrapy.org/htdocs/css/browser.css => sites/static.scrapy.org/css/browser.css
rename : sites/dev.scrapy.org/htdocs/css/changeset.css => sites/static.scrapy.org/css/changeset.css
rename : sites/dev.scrapy.org/htdocs/css/code.css => sites/static.scrapy.org/css/code.css
rename : sites/dev.scrapy.org/htdocs/css/diff.css => sites/static.scrapy.org/css/diff.css
rename : sites/dev.scrapy.org/htdocs/css/prefs.css => sites/static.scrapy.org/css/prefs.css
rename : sites/dev.scrapy.org/htdocs/css/report.css => sites/static.scrapy.org/css/report.css
rename : sites/dev.scrapy.org/htdocs/css/roadmap.css => sites/static.scrapy.org/css/roadmap.css
rename : sites/dev.scrapy.org/htdocs/css/search.css => sites/static.scrapy.org/css/search.css
rename : sites/dev.scrapy.org/htdocs/css/ticket.css => sites/static.scrapy.org/css/ticket.css
rename : sites/dev.scrapy.org/htdocs/css/timeline.css => sites/static.scrapy.org/css/timeline.css
rename : sites/dev.scrapy.org/htdocs/css/trac-0.11rc2.css.diff => sites/static.scrapy.org/css/trac-0.11rc2.css.diff
rename : sites/dev.scrapy.org/htdocs/css/trac.css => sites/static.scrapy.org/css/trac.css
rename : sites/dev.scrapy.org/htdocs/css/wiki.css => sites/static.scrapy.org/css/wiki.css
rename : sites/dev.scrapy.org/htdocs/desc.png => sites/static.scrapy.org/desc.png
rename : sites/dev.scrapy.org/htdocs/dots.gif => sites/static.scrapy.org/dots.gif
rename : sites/dev.scrapy.org/htdocs/draft.png => sites/static.scrapy.org/draft.png
rename : sites/dev.scrapy.org/htdocs/edit_toolbar.png => sites/static.scrapy.org/edit_toolbar.png
rename : sites/dev.scrapy.org/htdocs/editedticket.png => sites/static.scrapy.org/editedticket.png
rename : sites/dev.scrapy.org/htdocs/envelope.png => sites/static.scrapy.org/envelope.png
rename : sites/dev.scrapy.org/htdocs/expanded.png => sites/static.scrapy.org/expanded.png
rename : sites/dev.scrapy.org/htdocs/expander_normal.png => sites/static.scrapy.org/expander_normal.png
rename : sites/dev.scrapy.org/htdocs/expander_normal_hover.png => sites/static.scrapy.org/expander_normal_hover.png
rename : sites/dev.scrapy.org/htdocs/expander_open.png => sites/static.scrapy.org/expander_open.png
rename : sites/dev.scrapy.org/htdocs/expander_open_hover.png => sites/static.scrapy.org/expander_open_hover.png
rename : sites/dev.scrapy.org/htdocs/extlink.gif => sites/static.scrapy.org/extlink.gif
rename : sites/dev.scrapy.org/htdocs/feed.png => sites/static.scrapy.org/feed.png
rename : sites/dev.scrapy.org/htdocs/file.png => sites/static.scrapy.org/file.png
rename : sites/dev.scrapy.org/htdocs/folder.png => sites/static.scrapy.org/folder.png
rename : sites/dev.scrapy.org/htdocs/guide/basic-workflow.png => sites/static.scrapy.org/guide/basic-workflow.png
rename : sites/dev.scrapy.org/htdocs/guide/original-workflow.png => sites/static.scrapy.org/guide/original-workflow.png
rename : sites/dev.scrapy.org/htdocs/ics.png => sites/static.scrapy.org/ics.png
rename : sites/dev.scrapy.org/htdocs/imggrid.png => sites/static.scrapy.org/imggrid.png
rename : sites/dev.scrapy.org/htdocs/js/blame.js => sites/static.scrapy.org/js/blame.js
rename : sites/dev.scrapy.org/htdocs/js/diff.js => sites/static.scrapy.org/js/diff.js
rename : sites/dev.scrapy.org/htdocs/js/expand_dir.js => sites/static.scrapy.org/js/expand_dir.js
rename : sites/dev.scrapy.org/htdocs/js/folding.js => sites/static.scrapy.org/js/folding.js
rename : sites/dev.scrapy.org/htdocs/js/ie_pre7_hacks.js => sites/static.scrapy.org/js/ie_pre7_hacks.js
rename : sites/dev.scrapy.org/htdocs/js/jquery.js => sites/static.scrapy.org/js/jquery.js
rename : sites/dev.scrapy.org/htdocs/js/keyboard_nav.js => sites/static.scrapy.org/js/keyboard_nav.js
rename : sites/dev.scrapy.org/htdocs/js/noconflict.js => sites/static.scrapy.org/js/noconflict.js
rename : sites/dev.scrapy.org/htdocs/js/query.js => sites/static.scrapy.org/js/query.js
rename : sites/dev.scrapy.org/htdocs/js/search.js => sites/static.scrapy.org/js/search.js
rename : sites/dev.scrapy.org/htdocs/js/suggest.js => sites/static.scrapy.org/js/suggest.js
rename : sites/dev.scrapy.org/htdocs/js/trac.js => sites/static.scrapy.org/js/trac.js
rename : sites/dev.scrapy.org/htdocs/js/wikitoolbar.js => sites/static.scrapy.org/js/wikitoolbar.js
rename : sites/dev.scrapy.org/htdocs/loading.gif => sites/static.scrapy.org/loading.gif
rename : sites/dev.scrapy.org/htdocs/lock-locked.png => sites/static.scrapy.org/lock-locked.png
rename : sites/dev.scrapy.org/htdocs/logo.jpg => sites/static.scrapy.org/logo.jpg
rename : sites/dev.scrapy.org/htdocs/main-bg.jpg => sites/static.scrapy.org/main-bg.jpg
rename : sites/dev.scrapy.org/htdocs/milestone.png => sites/static.scrapy.org/milestone.png
rename : sites/dev.scrapy.org/htdocs/newticket.png => sites/static.scrapy.org/newticket.png
rename : sites/dev.scrapy.org/htdocs/parent.png => sites/static.scrapy.org/parent.png
rename : sites/dev.scrapy.org/htdocs/python.png => sites/static.scrapy.org/python.png
rename : sites/dev.scrapy.org/htdocs/topbar_gradient.png => sites/static.scrapy.org/topbar_gradient.png
rename : sites/dev.scrapy.org/htdocs/topbar_gradient2.png => sites/static.scrapy.org/topbar_gradient2.png
rename : sites/dev.scrapy.org/htdocs/trac.ico => sites/static.scrapy.org/trac.ico
rename : sites/dev.scrapy.org/htdocs/trac_banner.png => sites/static.scrapy.org/trac_banner.png
rename : sites/dev.scrapy.org/htdocs/trac_logo_mini.png => sites/static.scrapy.org/trac_logo_mini.png
rename : sites/dev.scrapy.org/htdocs/vgradient.png => sites/static.scrapy.org/vgradient.png
rename : sites/dev.scrapy.org/htdocs/wiki.png => sites/static.scrapy.org/wiki.png
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4023
 2008-06-28 22:56:32 +00:00
Pablo Hoffman
ebb4b9787f removed executable bit from logo.jpg 
--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%4022
 2008-06-28 22:49:48 +00:00