headers
Elaine Quarles | 25 May 2013 00:36
Favicon

OrangeFS 2.9 Beta Version Available

A beta version of OrangeFS is available for download from the orangefs.org web site. It has two new features:

 

-  OrangeFS 2.9 supports two new security modes in addition to standard security mode. The first mode, key-based security, utilizes cryptographic key pairs for servers and clients. The second mode, certificate-based security, utilizes certificates uniquely associated with each OrangeFS user, as well as an LDAP directory for identity mapping.

 

-  Directory entries may now be distributed among multiple metadata servers to help minimize bottlenecks.

 

More information on each of these features is available in the ChangeLog file within the distribution. If you need additional assistance, please send questions to ofs-support <at> omnibond.com.

 

Thanks,

Elaine

 

_______________________________________________
Pvfs2-users mailing list
Pvfs2-users <at> beowulf-underground.org
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
Permalink | Reply | 0 comments |
headers
Michael Robbert | 25 May 2013 00:06
Picon
Favicon
Gravatar

Performance drop

We recently noticed a performance problem with our OrangeFS server.

Here are the server stats:
3 servers, built identically with identical hardware

[root <at> orangefs02 ~]# /usr/sbin/pvfs2-server --version
2.8.7-orangefs (mode: aio-threaded)

[root <at> orangefs02 ~]# uname -r
2.6.18-308.16.1.el5.584g0000

4 core E5603 1.60GHz
12GB of RAM

OrangeFS is being served to clients using bmi_tcp over DDR Infiniband.
Backend storage is PanFS with 2x10Gig connections on the servers. 
Performance to the backend looks fine using bonnie++. >100MB/sec write 
and ~250MB/s read to each stack. ~300 creates/sec.

On the OrangeFS clients are running kernel version 2.6.18-238.19.1.el5.

The biggest problem I have right now is that delete are taking a long 
time. Almost 1 sec per file.

[root <at> fatcompute-11-32 L_10_V0.2_eta0.3_wRes_truncerr1e-11]# find N2/|wc -l
137
[root <at> fatcompute-11-32 L_10_V0.2_eta0.3_wRes_truncerr1e-11]# time rm -rf N2

real    1m31.096s
user    0m0.000s
sys     0m0.015s

Similar results for file creates:

[root <at> fatcompute-11-32 ]# date;for i in `seq 1 50`;do touch 
file${i};done;date
Fri May 24 16:04:17 MDT 2013
Fri May 24 16:05:05 MDT 2013

What else do you need to know? Which debug flags? What should we be 
looking at?
I don't see any load on the servers and I've restarted server and 
rebooted server nodes.

Thanks for any pointers,
Mike Robbert
Colorado School of Mines
Attachment (smime.p7s): application/pkcs7-signature, 4003 bytes
_______________________________________________
Pvfs2-users mailing list
Pvfs2-users <at> beowulf-underground.org
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
Permalink | Reply | 3 comments |
headers
Scott Roberts | 14 May 2013 18:30
Favicon

Executing the Java binary on OrangeFS

Good afternoon,

Has anyone encountered issues executing applications directly from an OrangeFS mount?  Most applications like Python, Perl, &c. work fine.  I untarred the 64-bit version of Java 7 to the location /export/ofs/apps/src with no errors.   However, executing Java throws this error:

./java -version
mmap failed for CEN and END part of zip file
Error occurred during initialization of VM
java/lang/ClassNotFoundException: error in opening JAR file /export/ofs/apps/src/jdk1.7.0_21/jre/lib/rt.jar

Any ideas?

Cheers,
--Scott
_______________________________________________
Pvfs2-users mailing list
Pvfs2-users <at> beowulf-underground.org
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
Permalink | Reply | 5 comments |
headers
Boyd Wilson | 13 May 2013 14:07
Favicon

Re: Fast way to copy files to OrangeFS?

Glad that sped things up!

-b


On Sun, May 12, 2013 at 10:00 PM, Scott Roberts <scotty <at> jhu.edu> wrote:
Thank you Boyd.  Transfer time is down to 45 seconds.

Cheers,
--Scott

From: Boyd Wilson <boydw <at> omnibond.com>
Date: Sunday, May 12, 2013 8:46 PM
To: Scott Roberts <scotty <at> jhu.edu>
Cc: PVFS2-Users <pvfs2-users <at> beowulf-underground.org>
Subject: Re: [Pvfs2-users] Fast way to copy files to OrangeFS?

Scott
You can pre-load the user lib and then use either cp -r or rsync and it should help. 

-Boyd. 

Sent from my iPhone

On Sunday, May 12, 2013, Scott Roberts wrote:
All,

I'm migrating data from our old storage array to our new OrangeFS array.  Using rsync to transfer 14GB of data where the average file size is 70MB takes about 5 minutes with an average transfer speed of 60MB/s.  If I copy the files individually using pvfs2-cp they transfer at  750MB/s.

Does anyone have a script handy that will recursively copy directories using pvfs2-cp?

Cheers,
--Scott
 

_______________________________________________
Pvfs2-users mailing list
Pvfs2-users <at> beowulf-underground.org
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
Permalink | Reply | 0 comments |
headers
Scott Roberts | 12 May 2013 19:37
Favicon

Fast way to copy files to OrangeFS?

All,

I'm migrating data from our old storage array to our new OrangeFS array.  Using rsync to transfer 14GB of data where the average file size is 70MB takes about 5 minutes with an average transfer speed of 60MB/s.  If I copy the files individually using pvfs2-cp they transfer at  750MB/s.

Does anyone have a script handy that will recursively copy directories using pvfs2-cp?

Cheers,
--Scott
_______________________________________________
Pvfs2-users mailing list
Pvfs2-users <at> beowulf-underground.org
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
Permalink | Reply | 4 comments |
headers
xihuang sun | 11 May 2013 09:28
Picon

need some advices on load monitoring

Hello all,
    I need to perform some tests on my small cluster running orangeFS2.8.7, now I want to monitor my cluster's cpu,memory and network with each node's details while running my test . I want to know how you developers did when orangeFS was developed and is there any good tools for monitoring? Thanks a lot..
_______________________________________________
Pvfs2-users mailing list
Pvfs2-users <at> beowulf-underground.org
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
Permalink | Reply | 0 comments |
headers
Vishwanath Venkatesan | 7 May 2013 22:28
Picon

Reg extracting stripe size

Hi,

I was looking at programmatically extract the stripe size set for the pvfs2 file system using pvfs-2.8.2. I see that the file system provides functions like pvfs2-dist-set-param, but there is no pvfs2-dist-get-param available in pvfs-2.8.2. Am I missing something? or is there a different  function that needs to be used.

Please let me know,

Thanks
Vish 
_______________________________________________
Pvfs2-users mailing list
Pvfs2-users <at> beowulf-underground.org
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
Permalink | Reply | 4 comments |
headers
Andrew Savchenko | 18 Apr 2013 19:15
Picon

Problems with 2.8.6-r9510 and kernel 3.2.29

Hello,

It looks like I have yet another issue with pvfs2-client kernel
module. Our current setup: orangefs-2.8.6 with patch to stable svn
revision 9510 and 3.2.29 vanilla kernel.

Users experience hangs of various commands while using pvfs2
directories, e.g. ls hangs indefinitely until killed with SIGKILL.
After such error happens I can't remove pvfs2 kernel module: though
all fuser or lsof detected processes are removed, usage count is
still 1.

I see a lot of kernel BUG messages in the kernel.log, for all of them
IP is kmem_cache_alloc. See full text in the attached file.

I will update to 2.8.7 + latest stable branch and for 3.2.43 kernel
and will report back after some testing.

Another question: any progress on kernels >= 3.4 support? 3.2 became
quite old these days. Maybe in the upcoming OrangeFS-3.0?

Best regards,
Andrew Savchenko
Attachment (kernel.log.xz): application/octet-stream, 6072 bytes
_______________________________________________
Pvfs2-users mailing list
Pvfs2-users <at> beowulf-underground.org
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
Permalink | Reply | 4 comments |
headers
克利 | 24 Mar 2013 03:51
Favicon

does pvfs2 support multi meta-data server?

Hi all:
In the doc of pvfs2, it says:
"Initial PVFS2 releases will only support one metadata server per file system, but this restriction will be released in the future."

Does pvfs2 support multiply meta-data server in the latest release(2.8.2)?


_______________________________________________
Pvfs2-users mailing list
Pvfs2-users <at> beowulf-underground.org
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
Permalink | Reply | 1 comment |
headers
Matthieu Dorier | 23 Mar 2013 15:31
Picon
Favicon

Strange performance behavior with IOR

Hi,

I've installed PVFS (orangeFS 2.8.7) on a small cluster (2 PVFS nodes, 28 compute nodes of 24 cores each, everything connected through infiniband but using an IP stack on top of it, so the protocol for PVFS is TCP), and I witness some strange performance behaviors with IOR (using ROMIO compiled against PVFS, no kernel support):

IOR is started on 336 processes (14 nodes), writing 4MB/process in a single shared file using MPI-I/O (4MB transfer size also). It completes 100 iterations.

First every time I start an instance of IOR, the first I/O operation is extremely slow. I'm guessing this is because ROMIO has to initialize everything, get the list of PVFS servers, etc. Is there a way to speed this up?

Then, I set some delay between each iteration, to better reflect the behavior of an actual scientific application. When I set this delay to 90 sec, I get decreasing performance from one iteration to another: the first write shows 2 GB/s, and after 100 iterations, I get about 300 MB/s. Both the "open", "write" and "close" time are increasing. More precisely the completion time increases linearly with the iteration number.

Finally if the delay is short (5 sec for example, or no delay at all), I get 2 GB/s for one operation, then barely 100 MB/s for the next, then 2 GB/s again, and so on.

I tried different values for the delay: 90 seconds, 60, 30, 20, 10, 5 and 0. For 90, 60 and 30, I observe the first situation (throughput decreasing). For 20 and 10 the throughput is very variable, but we can notice some periodicity. With 5 seconds the periodicity is obvious: one slow, one fast, one slow, one fast, etc.
When there is no delay at all, the periodicity is larger: 4 or 5 fast writes at around 2 GB/s, then 1 slow at 75 MB/s, and so on.

Any idea how to explain this behavior?
Thanks

Matthieu Dorier
PhD student at ENS Cachan Brittany and IRISA
http://people.irisa.fr/Matthieu.Dorier
_______________________________________________
Pvfs2-users mailing list
Pvfs2-users <at> beowulf-underground.org
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
Permalink | Reply | 16 comments |
headers
xihuang sun | 13 Mar 2013 19:47
Picon

orangefs io-test

Can anyone show me a simple io-test result?
I'm running orangefs on 4 nodes and i'm doing some io-tests.
I made this test by myself so i'm not confident with my results.
or can anyone give a test benchmark which can be performed on orangefs?
Another question is how can i make non-contiguous io test?

Thanks a lot
_______________________________________________
Pvfs2-users mailing list
Pvfs2-users <at> beowulf-underground.org
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
Permalink | Reply | 4 comments |

Gmane