Vangelis Koukis | 16 May 18:13 2007

Access to Lanai SRAM using DMA over the PCI bus

Hello all,

for a research project, I'm trying to read and write directly from/to the
SRAM of my M3F-PCIXD-2 cards, accessing it through the PCI memory-mapped
address space.

When programming another device on the same PCI segment to do DMA
writes to the physical address range where the SRAM of the Lanai
is mapped, I can get very satisfactory throughputs, in the range of
~320MB/s. It seems I'm only limited by the PCI bus itself (64bit/66MHz,
gm_debug reports 318MB/s bus_read, 378MB/s bus_write).

However, when programming the same device to do *reads* from this address
space using DMA, i.e. to fetch data from SRAM, it can't do more than
~20MB/s... When dealing with main memory, the device reads and writes
at ~300MB/s (using PC133 SDRAM).

I'm using a Supermicro P3TDE6 motherboard (Serverworks ServerSet III
HC-SL chipset, Broadcom CIOB20 PCI bridge).

Any pointers, comments, help would be greatly appreciated.


Vangelis Koukis, PhD candidate
Computing Systems Laboratory,
National Technical University of Athens.
Institute of Communication and Computer Systems (ICCS)
vkoukis <at>
de Almeida, Valmor F. | 29 Mar 05:53 2007

mtrr: type mismatch for f8000000


Following the instructions at

I took the following steps to patch an installed version of mx

Downloaded patch mx-1.1.6-Linux- from myrinet web
Moved into mx-1.1.6/
cd mx-1.1.6/
patch -p0 < mx-1.1.6-Linux-
./configure --prefix=/opt/mx-1.1.6
make install

do in every node including master
/opt/mx/sin/mx_start_stop start
remove /etc/udev/rules.d/10-mx.rules

Still get the mismatch notice: see below. Also at the very bottom of
this file /etc/init.d/net.myri0 is being sought and it does not exist.
Output from mx_bug_report attached. Thanks for your help.


(Continue reading)

de Almeida, Valmor F. | 28 Mar 03:35 2007

RE: [Myricom help #50446] invalid rule '/etc/udev/rules.d/10-mx.rules:3'

> -----Original Message-----
> From: Myricom Technical Support [mailto:help <at>]
> Sent: Tuesday, March 27, 2007 8:48 PM
> To: de Almeida, Valmor F.
> Cc: myrinet <at>; help <at>
> Subject: RE: [Myricom help #50446] invalid rule '/etc/udev/rules.d/10-
> mx.rules:3'
> to be a big portability issue. Eventually we fell back to
> a more portable way of creating the /dev/mx* devices.

I've noticed that.

> The recommendation in your case is to delete the file
> /etc/udev/rules.d/10-mx.rules .

Did that.

> We have made a change in our next release which avoids
> creating that file in the first place. (Let us know if
> you would like a patch).

Patch will not be needed.

Thank you for your prompt response.


(Continue reading)

de Almeida, Valmor F. | 28 Mar 02:25 2007

invalid rule '/etc/udev/rules.d/10-mx.rules:3'


 I just migrated from GM to MX-1.1.6 and installed with the linux kernel All went well including
the module loading. The mx_info gives me the listing below. However on the log files I find:

Mar 27 20:00:50 x2 sshd(pam_unix)[4902]: session opened for user root by root(uid=0)
Mar 27 20:03:25 x2 udevd[929]: add_to_rules: invalid KERNEL operation
Mar 27 20:03:25 x2 udevd[929]: add_to_rules: invalid rule '/etc/udev/rules.d/10-mx.rules:3'
Mar 27 20:03:25 x2 udevd[929]: add_to_rules: invalid KERNEL operation
Mar 27 20:03:25 x2 udevd[929]: add_to_rules: invalid rule '/etc/udev/rules.d/10-mx.rules:4'
Mar 27 20:03:35 x2 mx_mcp: module license 'unspecified' taints kernel.
Mar 27 20:03:35 x2 mx INFO: On i686, kernel version: #1 SMP PREEMPT Tue Mar 27 18:50:33 EDT 2007
Mar 27 20:03:35 x2 mx INFO: MX module compiled with kernel headers of #1 SMP PREEMPT Sun Mar 18
22:25:56 EDT 2007

Could you please comment on the lines with "invalid rule..." and "taints kernel." Is this a problem?


Valmor de Almeida

MX Version: 1.1.6
MX Build: root <at> x1:/root/third-party_downloads/mx-1.1.6 Mon Mar 26 05:06:39 EDT 2007
1 Myrinet board installed.
The MX driver is configured to support up to 4 instances and 1024 nodes.
Instance #0:  224.9 MHz LANai, 133.2 MHz PCI bus, 2 MB SRAM
        Status:         Running, P0: Link up
(Continue reading)

de Almeida, Valmor F. | 8 Jan 17:53 2007

one-sided comm with OpenMPI-MX


Is one-sided communication supported with the combination OpenMPI 1.1.2
and MX-2G 1.1.6 on PCIXD NIC's?

Just double checking: is the Linux kernel 2.6.16 supported?


Valmor de Almeida
Florian Brulhart | 23 Oct 18:20 2006

Simple question about the GM alarm


I'm still working for my project and we try to reduce our thread number...

My question is :
If I don't use thread, can I use the gm_set_alarm to call a polling method or the gm_set_alarm block the "caller" thread ?

Thanks for any help,


Myrinet mailing list
Myrinet <at>
tomanj2 | 22 Oct 21:39 2006

how to improve synch.delay (Isend)


I am still working on implementation of Parallel Segmented Quicksort Alg.,using
mpi Isend and Irecv functions for exchanging KBs of data between CPU's.
My problem is in performence of this exchange.
Implementation of data exchange is simple:
1]All CPUs send all data which need to be exchange using Isend(...) to another
/* called several times */
MPI_Send(...)  sends message envelope(for Iprobe testing) with size of Isend
MPI_Isend(...) use non-blocking send for data (data for send)
MPI_Isend(...) use non-blocking sends for indexes (data for send)
2]When all data for exchange were sent by Isend,all CPUs then use Iprobe to
receive message size and then call data by Irecv several times
/* called for several times */
while("counter") {
  do {
   Iprobe(...) get message size
  Recv(size,...) get data message size

  Irecv(..,size,.) receive data by non-blocking send
  Irecv(..,size,.) receive indexes (also data) by non-blocking send
3]before finally deallocaion of send and receive buffers I call MPI_Waitall for
test of completion Isend.

THE PROBLEM IS: when I run my quicksort alg. with more CPU, total elapsed time
is worse then for less CPU! I think the reason is in synchronization delays
between CPUs which grows with their number.
Do I use Isend correctly?Is there any other way how to rewrite,improve this 
procedure or shall I use another type of send?

I am still beginner around MPI lib.
Thanks a lot for hepl Jiri Tomanek
Florian Brulhart | 17 Oct 19:21 2006

Passing a struct with GM

Hello guys,

We are two student who work on a Sent/Receive on a Myrinet Network.
We try to send a "struct" with GM between two nodes. The send work and we can got the message on the receiver part, but we cannot read the information of our message. I will try to explain that with our source code.

This struct is :

struct gm_msg{
   int idConn;
   char * data;

somewhere in our code, we create another struct gm_connect_msg { int nodeID int portID;} and we send them with this method :
gm_connect_msg msg_connect;
msg_connect.nodeID = 1;
msg_connect.portID = 2;

int paroc_combox_gm::Send(const char *m, int len){
printf("[DEBUG]Init of Send(char*, int)\n");
  gm_recv_event_t *event;
  void *sendBuffer=NULL;
  int expected_callbacks=0;
  gm_status_t status;
  gm_msg msg;
  int length;
  printf("[DEBUG] variables sets, begin memcpy\n");
  gm_printf("[DEBUG] m param :%s, his lenght : %i \n",m,len); = (char *)malloc(len);
  printf("[DEBUG] end of memcpy \n");
  msg.idConn = idConn;
  length = sizeof(msg);
  /* Buffers are preallocated to CACTUS_MSG_SIZE other buffers need to be handled */
  nNextSendBuffer = (nNextSendBuffer + 1) % nSendTokens;

  printf("[DEBUG] End of init of send \n");
  gm_printf("[DEBUG] find a free buffer and send Token\n");

  if(gm_send_buffer[nNextSendBuffer].state != BUFFER_FREE){
    while(nNextSendBuffer < nSendTokens){
      nNextSendBuffer=(nNextSendBuffer+1) % nSendTokens;
      if(gm_send_buffer[nNextSendBuffer].state == BUFFER_FREE)
      if(nNextSendBuffer==nSendTokens && gm_send_buffer[nNextSendBuffer].state==BUFFER_IN_USE){
        fprintf(stdout,"Not enough Tokens for send \n");
        return -1;
  printf("[DEBUG] free buffer ok!\n");

  printf("Len: %i\n", strlen(;

  gm_printf("[DEBUG] registration of the message in buffer \n");

  gm_printf("DESTINATION : Node=%i ;Port=%i", destNodeID, destPortID);


    fprintf(stdout,"Send-Msg Sent Message successfully\n");
    while (1){
      event = gm_receive (gm_port);
      switch (GM_RECV_EVENT_TYPE(event))
        case GM_RECV_EVENT:
        case GM_PEER_RECV_EVENT:
          gm_printf ("[send] Receive Event (UNEXPECTED)\n");
          status = GM_FAILURE; /* Unexpected incoming message */
          return -1;

        case GM_NO_RECV_EVENT:

         gm_unknown (gm_port, event);    /* gm_unknown calls the callback */
         return 0;
  return (GM_SUCCESS);

And our Receive function is :
int paroc_combox_gm::Recv(char * msg, int msgLength){
  printf("Recv() in Combox_gm\n");
  gm_recv_event_t *event;
  void *buffer;
  unsigned int size;
  int messages_expected = 1;
  unsigned int len;
  void * message;
  gm_msg m;

  gm_connect_msg tmpmsg;
  gm_connect_msg * tmpptr;

  printf("\t Init passed\n");

  while(messages_expected > 0){
    event = gm_receive (gm_port);
    //      fprintf(stdout,"Received an Event \n");
    switch (GM_RECV_EVENT_TYPE(event)){
    case GM_RECV_EVENT:
      printf("\tyou have a new message\n");
      len=gm_ntoh_u32(event-> recv.length);

      memcpy(&m, gm_ntohp(event->recv.message), len);

      printf("\tm init\n");
      printf("idConn = %i\n",m.idConn );

      printf("\t tmp init\n");
      tmpptr = (gm_connect_msg *);
      printf("\tmsg copy done\n");
      printf("\ : %s\n",;
      printf("\tm.idConn : %d\n",m.idConn); //this is right
      printf("\ttmpptr.node : %d",tmpptr->nodeID);/ /this is wrong

      /* Return the buffer for reuse */
      buffer = gm_ntohp (event->recv.buffer);
      size = (unsigned int)gm_ntoh_u8 (event->recv.size);
      gm_provide_receive_buffer (gm_port, buffer, gm_log2_roundup(MSG_SIZE),
      printf("\trecv buffer re-provided\n");

    case GM_NO_RECV_EVENT:

      gm_unknown (gm_port, event);      /* gm_unknown calls the callback */

but unfortunately, when the message arrive, we have some magic number for the nodeID and the portID like "1406914151".

Have you any idea ?

Thanks for all..

Florian Brulhart & Stephane Droz

Myrinet mailing list
Myrinet <at>
John | 28 Sep 23:35 2006

Using new Myrinet NICs with old M3-E32/M3-SW18-8F switch


I have an old cluster: each node with M3F-PCI64B-2
Myricom NIC connected via M3-E32/M3-SW18-8F switch. I
have unused slots in the switch that I would like to
use to integrate new nodes to the existing Myrinet

Can I use newest Myricom cards such as M3F-PCIXD-2,
M3F-PCIXF-2, M3F-PCIXE-2 in the old Myrinet
environment, that is mixing old and new card in the
network managed by M3-E32/M3-SW18-8F switch?

Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
tomanj2 | 18 Sep 21:47 2006

Isend max.message size?


I'm here for first time and I'm interested about myrinet technology.
Now, I'm also student of Czech Technical University in Praque, chair of Computer
I am implementing Segmented version of paralel quicksort alghoritm which I
tested on university's myrinet cluster called star(

The problem is:
When I used MPI_Isend function for exchanging data between processors (sending
for ex. one huge message containing thousands of MPI_DOUBLE members!) and test
the end of the Isend by MPI_Test function...the repeat loop will never end
(means that data won't be send).

When the count (number of members are less) then everything is O.K. and MPI_Test
 will finish both loops.

I try to write simple code for better undertanding:

MPI_Isend(indexbuf,count, MPI_UNSIGNED,dest,tag,MPI_COMM_WORLD,&request2);
do {
      MPI_Test(&request1, &flg, &status);
    } while (!flg);
do {
      MPI_Test(&request2, &flg, &status);
    } while (!flg);

/* dealocation of buffers */

Is it a problem with count parameter (number of items) resp.  message is too
huge? What is the maximal message size that I can send?

Thanks a lot. Jiri Tomanek
george wm turner | 14 Sep 01:34 2006



     As SysAdmins, we're thinking of making --mx-kill 60 the
default on our mpirun commands to force clean up of orphans
left behind when user's codes error out.  Is there any
potential problems that we may be setting ourselves up
for by doing this?  Any guidance would be appreciated.

george wm turner
uits/rats  <at>  indiana university
812 855 5156