headers
Jim Maas | 19 May 16:11
Picon
Picon
Favicon

fork() warning from mpi

I get the following message when running a job using R 2.15.0, Rmpi, 
doMPI, and a cluster system that uses a LSF operating system and 
infiniband connections.  I'm just a user and know little about the 
workings of such networks but the warning unsettles me a little.  I've 
asked people who write and maintain open-mpi and they are suggesting 
that something has changed in the latest version of R that causes this.

I'm wondering if there has been a change or if we have done something 
incorrectly when installing Rmpi on the system, not sure here but this 
might have to be built/compiled against open-mpi libraries.  The 
open-mpi people have suggested that R must be doing some king of fork 
after the call MPI-Init.

Could anyone tell me if this is a new problem, or have we done something 
incorrectly?  I have a toy model file that creates it and am happy to 
supply this.

Thanks a bunch,

J

===============================
An MPI process has executed an operation involving a call to the
"fork()" system call to create a child process.  Open MPI is currently
operating in a condition that could result in memory corruption or
other system errors; your MPI job may hang, crash, or produce silent
data corruption.  The use of fork() (or system() or other calls that
create child processes) is strongly discouraged.

The process that invoked fork was:
(Continue reading)

Permalink | Reply | 0 comments |
headers
Barnet Wagman | 14 May 18:07

New version of cloudRmpi

cloudRmpi v 1.2 is now available on CRAN.

cloudRmpi is a means for doing parallel processing in R, using MPI on a  
cloud-based network.  It currently
supports the use of Amazon's EC2 cloud computer service.

Changes in v 1.2:

Support for RStudio. RStudio Server is available on new AMIs.  cloudRmpi 
now has a function for securely connecting to an RStudio session running 
on the master node of an EC2-MPI network, via ssh port forwarding.  
(RStudio Server is a browser based, IDE-like interface to R).

The network specification dialog shows more information about AMIs.

The network manager has a command to repeat Open MPI network network 
configuration.

Regards,

Barnet Wagman
Permalink | Reply | 0 comments |
headers
Stefan Janse van Rensburg | 14 May 13:08
Picon

Condor, grid-appliance or pool-of-virtual machine

Dear List,

I am hoping somebody would be kind enough to share their recent 
experiences with R and Condor or R and BOINC. I am aware of the November 
2005 RNews article, but I would think that a lot has happened in this 
area since then. E.g. The gridR package.

I am writing a preliminary report on the viability of implementing some 
form of a grid computing setup for cycle-scavanging at Rhodes 
University, South Africa. Most of our lab machines are Windows XP and 7 
desktops. My preliminary readings suggest that running a VM with Linux 
guest OS would be the best option. Is see that there are several 
pre-prepared bundles for this sort of thing, such as Grid Appliance and 
Pool-of-Virtual Machines. Has any one used any of these with R?

Any help, whatsoever, would be much appreciated.

Kind regards,

Stefan Janse van Rensburg 
<http://www.ru.ac.za/statistics/staff/mrstefanjansevanrensburg/>

<http://www.ru.ac.za/statistics/staff/mrstefanjansevanrensburg/>

Lecturer, Dept. of Statistics, Rhodes University

Tel: +27-046-603-8682

--
(Continue reading)

Permalink | Reply | 0 comments |
headers
Jonathan Greenberg | 3 May 00:23
Favicon

Quickest way to make a large "empty" file on disk?

R-helpers:

What would be the absolute fastest way to make a large "empty" file (e.g.
filled with all zeroes) on disk, given a byte size and a given number
number of empty values.  I know I can use writeBin, but the "object" in
this case may be far too large to store in main memory.  I'm asking because
I'm going to use this file in conjunction with mmap to do parallel writes
to this file.  Say, I want to create a blank file of 10,000 floating point
numbers.

Thanks!

--j

--

-- 
Jonathan A. Greenberg, PhD
Assistant Professor
Department of Geography and Geographic Information Science
University of Illinois at Urbana-Champaign
607 South Mathews Avenue, MC 150
Urbana, IL 61801
Phone: 415-763-5476
AIM: jgrn307, MSN: jgrn307@..., Gchat: jgrn307, Skype: jgrn3007
http://www.geog.illinois.edu/people/JonathanGreenberg.html

	[[alternative HTML version deleted]]
Permalink | Reply | 13 comments |
headers
Lisa Solomon | 2 May 21:08
Favicon

Analytics & Data Mining Conference, San Diego, California

Subject: Analytics & Data Mining Conference, San Diego, California

Don't miss it, it's this month!

The 2012 Salford Analytics & Data Mining Conference aims to bring together
researchers, practitioners, and data enthusiasts to exchange ideas and
experiences.

Attendees will have the chance to have one-on-one meetings with the creators of the CART and RandomForests algorithms
(Dr. Adele Cutler, Dr. Jerome Friedman and Dr. Richard Olshen).

In addition, the following companies will give presentations:
Johnson & Johnson, Genentech, Visa, Fidelity, Qualcomm, Union Bank and more!

Conference website: http://www.salforddatamining.com/
Agenda: http://www.salforddatamining.com/agenda.html
Registration: http://www.salforddatamining.com/register.html

For additional information, contact
info@...<mailto:info@...>
or phone 619-543-8880.

Best regards,
Lisa Solomon
lisas@...<mailto:lisas@...>
Salford Systems

	[[alternative HTML version deleted]]
Permalink | Reply | 0 comments |
headers
Libo Sun | 30 Apr 23:14

Re: 'Rmpi' issue

Thanks for your good advice.

I changed to R 2.14.1 and Rmpi works fine. :)

However, there is another problem when I try to use snow package.

This is error message I got:

MPI: bl1.psc.teragrid.org: 0x54e500004f90ba1f: /bin/sh: line 0: exec:
RMPISNOW: not found

I am sure RMPISNOW is in the working directory.

Thanks,
Libo

On Mon, Apr 30, 2012 at 7:10 AM, Stephen Weston
<stephen.b.weston@...>wrote:

> Hi Libo,
>
> The traceback that you include shows that you're not able to
> load the Rmpi package.  But it's not because Rmpi isn't
> installed, because it's getting a segmentation violation when
> trying to load the Rmpi shared library, which sounds like a bad
> installation of Rmpi.  But it's also possible that you're
> loading inconsistent module files in your batch script, or
> you're doing something in your .Rprofile that is breaking it.
>
> Since it looks like you're using system provided modules files,
(Continue reading)

Permalink | Reply | 0 comments |
headers
Libo Sun | 30 Apr 06:07

'Rmpi' issue

Hi,

I am trying to test 'Rmpi' on Blacklight,
http://www.psc.edu/machines/sgi/uv/blacklight.php, in PSC.

The R code I am trying to run is very simple:

mpi.remote.exec(paste("I am",mpi.comm.rank(),"of",mpi.comm.size()))
mpi.remote.exec(paste("I am",Sys.info()1,"of",mpi.comm.size()))

The batch file:

#!/bin/csh
#PBS -l walltime=5:00
#PBS -l ncpus=16
#PBS -o test.out
#PBS -j oe
#PBS -q debug

# define module command
source /usr/share/modules/init/csh

set echo
date

# load R module
module load R/2.11.1
module load Rmpi
module swap mpt/2.04 mpt/2.01
(Continue reading)

Permalink | Reply | 1 comment |
headers
Martin Renner | 23 Apr 21:53
Picon

parallel and openblas

Parallel and openblas don't seem to mix well on my machine. If I link openblas, a job executed through
parallel (using either the multicore or snow (local socket cluster) setup), each of my 8 cores only
operates at 1/8 of 100% (taking a little longer than serial execution). Linking to the reference blas or to
single-threaded atlas does not cause this handicap when running snow or multicore. 

Is this a known problem (My google attempts were fruitless)? If yes, is there a fix for it? Do MKL or
multi-threaded atlas have the same issues? 

Thank you for your time. 

Martin

Martin Renner
Post-doctoral Fellow				phone: 907-226 4672
University of Washington			   or: 907-235 0728
School of Aquatic and Fishery Sciences		Seattle, USA

debian squeeze on 8-core Xeon
> sessionInfo()
R version 2.15.0 (2012-03-30)
Platform: x86_64-unknown-linux-gnu (64-bit)

locale:
[1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
[3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
[5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
[7] LC_PAPER=C                 LC_NAME=C                 
[9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
(Continue reading)

Permalink | Reply | 19 comments |
headers
ivo welch | 21 Apr 00:16
Picon

What to Experiment With?

Dear R HPC experts:

I have about $5,000 to spend on building fast computer hardware to run
our problems.  if it works well, I may be able to scrounge up another
$10k/year to scale it up.  I do not have the resources to program very
complex algorithms, administer a full cluster, etc.  (the effective
programmer's rate here is about $50/hour and up, and I have severe
restrictions against hiring outsiders.)  the programs basically have
to work with minimum special tweaking.

There are no real-time needs.  Typically, I operate on historical CRSP
and Compustat data, which are about 1-5GB (depending on subset).  most
of what I am doing involves linear regressions.  I often need to
calculate Newey-West/Hansen-Hodrick/White adjusted standard errors,
and I often do need to sort and rank, calculate means and covariances.
 these are not highly sophisticated stats, but it entails lots of it.
most of what I do is embarrassingly parallel.

Now, I think in the $5k price range, I have a couple of options.
Roughly, the landscape seems to be:

* 1 dual-socket xeon i7 computers.
* 5 (desktop) i7 computers, networked (socket snow?).
* 1 i7 computer, with 1 nvidia Tesla card
* 1 i7 computers with 2-3 commodity graphics cards
     --- apparently, nvidia cripples the DP performance of its gamer
cards, so AMD should be a *lot* faster
     at the same price, but I only see the lm() routine in
nvidia-specific gputools.  then again, for Newey-West,
     I may have to resort to my own calculations, anyway.  is there
(Continue reading)

Permalink | Reply | 6 comments |
headers
Jens Oehlschlägel | 15 Apr 12:54

How to shorten an atomic vector without copying?

Hi Performance gurus,

Does any of you know how to shorten an atomic vector without copying the 
contents to a new vector? The official SET_LENGTH and lengthgets seem to 
create a new atomic vector rather than realloc the old one.

Jens Oehlschlägel

_______________________________________________
R-sig-hpc mailing list
R-sig-hpc <at> r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
Permalink | Reply | 1 comment |
headers
Marius Hofert | 15 Apr 02:37
Picon
Favicon

Re: What happened to the vignette of 'parallel'?

Lieber Dirk,

das ging ja schnell :-) 

Vielen Dank und herzliche Gruesse,

Marius

PS: Habe gerade etwas auf Deiner eindruecklichen Homepage verweilt, sehr spannend!

Dirk Eddelbuettel <edd <at> debian.org> writes:

> Marius,
>
> Wie versprochen, parallel.pdf anbei.
>
>
>
> Gruesse,  Dirk

--

-- 
ETH Zurich
Dr. Marius Hofert
RiskLab, Department of Mathematics
HG E 65.2
Rämistrasse 101
8092 Zurich
Switzerland

Phone +41 44 632 2423
(Continue reading)

Permalink | Reply | 0 comments |

Gmane