<Shameless Plug\> I'm moderating a webinar called: Cool Crunching: Understanding Green HPC on Wednesday (Dec 3) at 11AM EST. More info and registration: http://linux-mag.com/id/7172 -- Doug -- -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.
maybe Joe means this: http://www.basement-supercomputing.com/content/view/19/45/ which is being updated for SGE 6.* I basically hacked it together because I could not easily understand qstat. Source is available, be advised code is ugly as it started out as hack while I was debugging SGE parallel environments and test suites. I should have a new version "real soon now" there are also some web based tools that do this for SGE as well (http://xml-qstat.org/) -- Doug > Thomas Vixel wrote: >> I've been googling for a top-like cli tool to use on our cluster, but >> the closest thing that comes up is Rocks' "cluster top" script. That >> could be tweaked to work via the cli, but due to factors beyond my >> control (management) all functionality has to come from a pre-fab >> program rather than a software stack with local, custom modifications. >> >> I'm sure this has come up more than once in the HPC sector as well -- >> could anyone point me to any top-like apps for our cluster? > > We have a ctop we have written a while ago. Depends upon pdsh, though > with a little effort, even that could be removed (albeit being a(Continue reading)
> Robert G. Brown wrote: >> On Sun, 30 Nov 2008, Greg Lindahl wrote: >> >>> >>> On Sun, Nov 30, 2008 at 11:45:44AM -0500, Robert G. Brown wrote: >>> >>>> That's fine, but I'm curious. How do you expect to run a cluster >>>> information tool over a network without a socket at both ends? >>> >>> There's always "qstat". The OP didn't really say what sorts of >>> information he was looking for... >> >>(Continue reading)Hey, didn't think of that -- an enormous Quake cluster? >> >> Although I didn't realize that qstat worked by electronic telepathy
> > to bad we can't use EPR pairs for this ... Well maybe not in this universe ... -- Doug > > -- > Joseph Landman, Ph.D > Founder and CEO > Scalable Informatics LLC, > email: landman <at> scalableinformatics.com
On Mon, 1 Dec 2008, Thomas Vixel wrote: > The main requirements were that 1) "it must look like top", 2) it must > be cli-based, 3) it should not introduce another piece of (server) > software (and thus point of failure) into the system, 4) and it should > not require any local hacks on our part. > > Since this is a web cluster, I suppose the most logical transport > would be HTTP, but everything I've seen so far would require me to > violate (4) to satisfy 1-3. SSH & SNMP are also available, I could see > SNMP being problematic for a project like this. SSH *might* work and > *might* be scalable for a project like this since the main expense is > in the construction the connections (and they can be re-used across > the course of the execution), but I have yet to find a top-like > program that leverages it. > > xmlsysd & wulfstat would appear to be the least expensive solution to > this problem, but as I said, management already nixed it. Um, I don't know what you call a "local hack", but it IS called xmlsysd for a reason. You don't have to run it as an actual public daemon. It produces xml, and in inetd mode it reads from stdin and writes to stdout. Try this. Install xmlsysd on something. Then in the command line, enter: ./xmlsysd -i 7880 (the port number isn't important -- you're just telling it to use inetd mode). It will run and nothing will happen. Then WITHOUT BREAKING OUT just enter(Continue reading)
I'm getting this error when I run ibchecknet on my cluster: #warn: counter VL15Dropped = 476 (threshold 100) lid 1 port 1 Error check on lid 1 (aurora HCA-1) port 1: FAILED I've googled around this morning, but haven't found anything helpful. Most of the hits turn up code with the phrase "VL15Dropped", but nothing explaining what this error means, what causes it, or how to fix it. After clearing the counters with 'perfquery -r', the VL15Dropped count starts increasing from zero almost immediately. Any ideas what this error represents or how to fix? Could it be a bad cable? -- Prentice
I can try to help you here, and would need to understand your setup and on which port the drop is occurring on. Bad cable causing this seems very unlikely. Gilad. -----Original Message----- From: beowulf-bounces <at> beowulf.org [mailto:beowulf-bounces <at> beowulf.org] On Behalf Of Prentice Bisbal Sent: Tuesday, December 02, 2008 7:24 AM To: Beowulf Mailing List Subject: [Beowulf] InfiniBand VL15 error I'm getting this error when I run ibchecknet on my cluster: #warn: counter VL15Dropped = 476 (threshold 100) lid 1 port 1 Error check on lid 1 (aurora HCA-1) port 1: FAILED I've googled around this morning, but haven't found anything helpful. Most of the hits turn up code with the phrase "VL15Dropped", but nothing explaining what this error means, what causes it, or how to fix it. After clearing the counters with 'perfquery -r', the VL15Dropped count starts increasing from zero almost immediately. Any ideas what this error represents or how to fix? Could it be a bad cable? -- Prentice(Continue reading)
The main requirements were that 1) "it must look like top", 2) it must be cli-based, 3) it should not introduce another piece of (server) software (and thus point of failure) into the system, 4) and it should not require any local hacks on our part. Since this is a web cluster, I suppose the most logical transport would be HTTP, but everything I've seen so far would require me to violate (4) to satisfy 1-3. SSH & SNMP are also available, I could see SNMP being problematic for a project like this. SSH *might* work and *might* be scalable for a project like this since the main expense is in the construction the connections (and they can be re-used across the course of the execution), but I have yet to find a top-like program that leverages it. xmlsysd & wulfstat would appear to be the least expensive solution to this problem, but as I said, management already nixed it. Honestly, if it weren't for (4), I'd probably just grab the top source and graft it onto a SSH library. It might not be the most efficient solution, but for a HA cluster it doesn't necessarily HAVE to be. On 11/30/08, Robert G. Brown <rgb <at> phy.duke.edu> wrote: > On Sat, 29 Nov 2008, Greg Kurtzer wrote: > >> Warewulf has a real time top like command for the cluster nodes and >> has been known to scale up to the thousands of nodes: >> >> http://www.runlevelzero.net/images/wwtop-screenshot.png >> >> We are just kicking off Warewulf development again now that Perceus(Continue reading)
Hi Guys ,
I am still thinking about my cluster and most probably will only begin development next year june/july .
Question :
If i develop my system on 10 computers (nodes) which are all normal desktops and then would like to place this in data hosting facility which has access to real time information . I am going to need to buy new servers (thin 1 u servers ). Would this be the best choice as desktops take up more space and therefore will be more expensive . how do you guys get around this problem ? or dont you ?
Regards
Malcolm
<div><p>Hi Guys , <br><br>I am still thinking about my cluster and most probably will only begin development next year june/july .<br><br> Question :<br><br>If i develop my system on 10 computers (nodes) which are all normal desktops and then would like to place this in data hosting facility which has access to real time information . I am going to need to buy new servers (thin 1 u servers ). Would this be the best choice as desktops take up more space and therefore will be more expensive . how do you guys get around this problem ? or dont you ?<br><br>Regards <br><br>Malcolm<br><br><br><br><br><br><br><br><br><br></p></div>
That does sound interesting, but more for some of my personal projects.
It wouldn't work for the situation at hand because:
1) It sounds like it introduces a SPF (the head node).
2) Giving our developers cluster-wide 'killall' & 'kill' functionality
makes me cringe.
Most of them only know just enough about Linux to be dangerous.
3) It would require completely reworking our current cluster solution;
a daunting task to say the least.
4) There isn't much love for commercial & non-OSS software at our company.
On 11/30/08, Donald Becker <becker <at> scyld.com> wrote:
> On Wed, 26 Nov 2008, Thomas Vixel wrote:
>
>> I've been googling for a top-like cli tool to use on our cluster, but
>> the closest thing that comes up is Rocks' "cluster top" script. That
>> could be tweaked to work via the cli, but due to factors beyond my
>> control (management) all functionality has to come from a pre-fab
>> program rather than a software stack with local, custom modifications.
>>
>> I'm sure this has come up more than once in the HPC sector as well --
>> could anyone point me to any top-like apps for our cluster?
>
> Most remote job mechanisms only think about starting remote processes, not
> about the full create-monitor-control-report functionality.
>
> The Scyld system (currently branded "Clusterware") defaults to using a
> built-in unified process space. That presents all of the processes
> running over the cluster in a process space on the master machine, with
> fully POSIX semantics. It neatly solves your need with... the standard
> 'top' program.
>
> Most scheduling systems also have a way to monitor processes that they
> start, but I haven't found one that takes advantage of all information
> available and reports it quickly/efficiently.
>
> There are many advantages of the Scyld implementation
> -- no new or modified process management tools need to be written.
> Standard utilities such as 'top' and 'ps' work unmodified,
> as well as tools we didn't specifically plan for e.g. GUI versions of
> 'pstree'.
> -- The 'killall' program works over the cluster, efficiently.
> -- All signals work as expected, including 'kill -9'. (Most remote
> process starting mechanisms will just kill off the local endpoint,
> leaving the remote process running-but-confused.)
> -- Process groups and controlling-TTY groups works properly for job
> control and signals
> -- Running jobs report their status and statistics accurately -- an
> updated 'rusage' structure is sent once per second, and a final
> rusage structure and exit status is sent when the process terminates.
>
> The "downside" is that we explicitly use Linux features and details,
> relying on kernel-version-specific features. That's an issue if it's a
> one-off hack, but we've been using this approach continuously for
> a decade, since the Linux 2.2 kernel and over multiple
> architectures. We've been producing supported commercial releases
> since 2000, longer than anyone else in the business.
>
> --
> Donald Becker becker <at> scyld.com
> Penguin Computing / Scyld Software
> www.penguincomputing.com www.scyld.com
> Annapolis MD and San Francisco CA
>
>
On Tue, Dec 02, 2008 at 10:24:15AM -0500, Prentice Bisbal wrote: > #warn: counter VL15Dropped = 476 (threshold 100) lid 1 port 1 > Error check on lid 1 (aurora HCA-1) port 1: FAILED IB is blissfully fading from my brain, but I think this refers to control packets being dropped due to resource limits on the recipient. That takes talent if you're using a Mellanox HCA, as pretty much all of the VL15 packets are interpreted by the processor in the HCA. -- greg
RSS Feed178 | |
|---|---|
303 | |
196 | |
113 | |
115 | |
79 | |
356 | |
136 | |
389 | |
102 | |
2 | |
95 | |
7 | |
29 | |
67 | |
98 | |
184 | |
52 | |
69 | |
88 | |
31 | |
69 | |
56 | |
45 | |
35 | |
82 | |
35 | |
145 | |
37 | |
18 | |
33 | |
158 | |
69 | |
47 | |
150 | |
94 | |
67 | |
131 | |
127 | |
220 | |
152 | |
201 | |
139 | |
291 | |
270 | |
204 | |
89 | |
187 | |
234 | |
631 | |
163 | |
288 | |
230 | |
381 | |
140 | |
341 | |
452 | |
408 | |
591 | |
462 | |
291 | |
236 | |
157 | |
245 | |
120 | |
239 | |
409 | |
370 | |
225 | |
213 | |
293 | |
161 | |
210 | |
443 | |
398 | |
179 | |
190 | |
201 | |
166 | |
129 | |
318 | |
232 | |
100 | |
288 | |
287 | |
100 | |
267 | |
220 | |
14700 |
