--mx-kill

Greetings,

     As SysAdmins, we're thinking of making --mx-kill 60 the
default on our mpirun commands to force clean up of orphans
left behind when user's codes error out.  Is there any
potential problems that we may be setting ourselves up
for by doing this?  Any guidance would be appreciated.

george wm turner
uits/rats  <at>  indiana university
812 855 5156

Isend max.message size?

Hello,

I'm here for first time and I'm interested about myrinet technology.
Now, I'm also student of Czech Technical University in Praque, chair of Computer
Science.
I am implementing Segmented version of paralel quicksort alghoritm which I
tested on university's myrinet cluster called star(star.felk.cvut.cz).

The problem is:
When I used MPI_Isend function for exchanging data between processors (sending
for ex. one huge message containing thousands of MPI_DOUBLE members!) and test
the end of the Isend by MPI_Test function...the repeat loop will never end
(means that data won't be send).

When the count (number of members are less) then everything is O.K. and MPI_Test
 will finish both loops.

I try to write simple code for better undertanding:

.....
MPI_Isend(databuf,count,MPI_DOUBLE,dest,tag,MPI_COMM_WORLD,&request1);
MPI_Isend(indexbuf,count, MPI_UNSIGNED,dest,tag,MPI_COMM_WORLD,&request2);
do {
      MPI_Test(&request1, &flg, &status);
    } while (!flg);
do {
      MPI_Test(&request2, &flg, &status);
    } while (!flg);

/* dealocation of buffers */

Re: --mx-kill

Hi George,

george wm turner wrote:
>     As SysAdmins, we're thinking of making --mx-kill 60 the
> default on our mpirun commands to force clean up of orphans
> left behind when user's codes error out.  Is there any
> potential problems that we may be setting ourselves up
> for by doing this?  Any guidance would be appreciated.

With the --mx-kill 60 flag, mpirun.ch_mx will try to kill all remaining 
processes 60 seconds after the first process exits. In most situation, 
all processes part of one MPI job will exit roughly at the same time. 
Unfortunately, it is possible to have an application that do extra work 
in one specific process after the call to MPI_Finalize(). In this case, 
this process will be wrongly killed if the extra work is longer than 60 
seconds.

I have never seen such an application on the field though. In my humble 
opinion, the best way to handle this dilemma would be to use a 
comfortable timeout by default, and be aware that some users may need to 
use a different script to either extend the timeout or disable it.

It's a trade-off between convenience for the majority and compatibility 
for the exception 

Patricl
--

-- 
Patrick Geoffray
Myricom, Inc.
http://www.myri.com

Re: Isend max.message size?

Hi Jiri,

tomanj2 <at> fel.cvut.cz wrote:
> The problem is:
> When I used MPI_Isend function for exchanging data between processors (sending
> for ex. one huge message containing thousands of MPI_DOUBLE members!) and test
> the end of the Isend by MPI_Test function...the repeat loop will never end
> (means that data won't be send).
> 
> When the count (number of members are less) then everything is O.K. and MPI_Test
>  will finish both loops.

> Is it a problem with count parameter (number of items) resp.  message is too
> huge? What is the maximal message size that I can send?

The maximum message size is 4GB (length coded on 32 bits), 2GB if signed 
types are used somewhere. However, I suspect that you are far from these 
limits. In most MPI implementations, there is a different behavior if 
the message size is below or above a specific threshold (~32 KB). If the 
message is small enough, it will be sent wherever the matching receive 
as been posted on the other side. In this case, there is the possibility 
that the message will be unexpected (receive not yet posted), so it 
would need to be saved in temporary buffer. If the message is too big, 
it is not reasonable to risk it to be unexpected, so the data is sent 
only if the receive has been posted, there is a rendez-vous step 
(synchronization).

I would suspect that you do not post a valid matching MPI receive and, 
depending on the size of the message, the send completes eagerly or not.

Re: --mx-kill

On Thursday 14 September 2006 09:34, george wm turner wrote:

>      As SysAdmins, we're thinking of making --mx-kill 60 the
> default on our mpirun commands to force clean up of orphans
> left behind when user's codes error out.

If you're using Torque then have you considered using Pete Wyckoff's excellent 
mpiexec instead ?

That will talk directly to the pbs_mom's to get its information and to start 
the processes using the TM interface. Much neater all round, PBS gets to 
monitor the processes CPU times more accurately and it is directly aware of 
processes dieing.

cheers,
Chris
--

-- 
 Christopher Samuel - (03)9925 4751 - VPAC Deputy Systems Manager
 Victorian Partnership for Advanced Computing http://www.vpac.org/
 Bldg 91, 110 Victoria Street, Carlton South, VIC 3053, Australia

_______________________________________________
Myrinet mailing list
Myrinet <at> osc.edu
http://email.osc.edu/mailman/listinfo/myrinet

Re: --mx-kill

On Tuesday 19 September 2006 18:14, Chris Samuel wrote:

> If you're using Torque then have you considered using Pete Wyckoff's
> excellent mpiexec instead ?

...and here's the link..

http://www.osc.edu/~pw/mpiexec/index.php

Sorry about that, it's been a long day/week/month..

--

-- 
 Christopher Samuel - (03)9925 4751 - VPAC Deputy Systems Manager
 Victorian Partnership for Advanced Computing http://www.vpac.org/
 Bldg 91, 110 Victoria Street, Carlton South, VIC 3053, Australia

_______________________________________________
Myrinet mailing list
Myrinet <at> osc.edu
http://email.osc.edu/mailman/listinfo/myrinet

Re: --mx-kill

Chris, et al,

The Peter Wykoff's mpiexec at the OSC site is an excellent
solution for Torque; however, it does not work with PBSpro
nor Loadleveler; the latter being the resource manager
for this current installation.  For cluster using Torque
I would recommend it myself for the reasons you gave.

(Sometimes a "me too" is just an appropriate response :)

george wm turner
uits/rats  <at>  indiana university
812 855 5156

On Tue, 19 Sep 2006, Chris Samuel wrote:

> On Thursday 14 September 2006 09:34, george wm turner wrote:
>
>>      As SysAdmins, we're thinking of making --mx-kill 60 the
>> default on our mpirun commands to force clean up of orphans
>> left behind when user's codes error out.
>
> If you're using Torque then have you considered using Pete Wyckoff's excellent
> mpiexec instead ?
>
> That will talk directly to the pbs_mom's to get its information and to start
> the processes using the TM interface. Much neater all round, PBS gets to
> monitor the processes CPU times more accurately and it is directly aware of
> processes dieing.
>> [appended from 2nd message]

Using new Myrinet NICs with old M3-E32/M3-SW18-8F switch

Hi,

I have an old cluster: each node with M3F-PCI64B-2
Myricom NIC connected via M3-E32/M3-SW18-8F switch. I
have unused slots in the switch that I would like to
use to integrate new nodes to the existing Myrinet
network. 

Can I use newest Myricom cards such as M3F-PCIXD-2,
M3F-PCIXF-2, M3F-PCIXE-2 in the old Myrinet
environment, that is mixing old and new card in the
network managed by M3-E32/M3-SW18-8F switch?

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com