headers
Paul Johnson | 7 Sep 2010 04:27
Picon

Re: Cluster R "environment" trouble. Using Rmpi

Dear Professor:

Thanks very much for the feedback.  That has helped me to cut errors
and get one of the user programs working.  One program is still
causing trouble, and I have a couple of questions below. I am sorry
that these are so elementary, but if I can understand this, then I can
write up some clear working examples for everybody.

On Wed, Aug 18, 2010 at 1:05 PM, Hao Yu <hyu@...> wrote:
> Hi Paul,
>
> Just got back from two conferences.
>
> First of all, when R slaves are spawned, they are "naked", meaning they
> are started with basic R functions/lib even that they are in the same dir
> with master. You have to tell slaves to get all necessary objects or to
> load libraries specifically. There are a few ways to do so.
>
> Use mpi.bcast.Robj2slave(an Robj) to send "an Robj" from master to all
> slaves. If a function to be executed on slaves depends on many
> functions/data, those functions/data must be sent to slaves first.
>
> Use mpi.bcast.cmd (cmd()) to tell salves to run cmd() like
> source("SimJob.R") (make sure to remove any execution commands in
> SimJob.R).

Can you please explain what "execution commands" means here?

I am *guessing* that anything that is forbidden in a "sourced" file is
also forbidden in a function that is passed to a node. Right?
(Continue reading)

Permalink | Reply | 1 comment |
headers
Hao Yu | 7 Sep 2010 16:41
Picon
Favicon

Re: Cluster R "environment" trouble. Using Rmpi

> mpi.bcast.Robj2slave( myThing )

will send the R object myThing to all slaves.

> mpi.bcast.cmd( myThing )

will tell all slaves to run myThing. But this just prints the function
myThing itself on slave. You need

mpi.bcast.cmd( myThing() )

to execute the function myThing.

Race condition: when source("someGreatCode.R") is executed on slaves, they
will try to load "someGreatCode.R" at the same time. But a file can only
be accessed once a time. If "someGreatCode.R" contains execution codes, it
is possible that the file access is locked to a particular slave until it
is done.  This prevents other slaves to access "someGreatCode.R" until the
lock is released.

If "someGreatCode.R" load from slaves are slow, it might be better to load
it on master and use mpi.bcastRobj2slave to move all necessary objects to
slaves. Make sure "someGreatCode.R" doesn't contain any lengthy execution
commands.

Hao

Paul Johnson wrote:
> Dear Professor:
>
(Continue reading)

Permalink | Reply | 0 comments |
headers
clement | 8 Sep 2010 09:11
Picon

Rmpi and ncdf4

Dear List Members,

I am using the ncdf4 package to work on General Circulation Model (GCM) 
data (NetCDF file format) and I would like to parallelize some 
calculations using Rmpi. Does anyone have an experience or advices in 
using Rmpi and ncdf4 packages?

--

-- 
Clement Tisseuil (Ph.D)
UMR BOREA 7208  « Biologie des Organismes et Ecosystèmes Aquatiques »
(CNRS-MNHN-UPMC-IRD), Equipe « Biodiversité et Macroécologie »,
43 rue Cuvier, 75005 Paris, France
Tel: 00 33 1 40 79 37 45
Fax: 00 33 1 40 79 37 71
webpage: http://www.clement-tisseuil.eu
Permalink | Reply | 4 comments |
headers
riccardo visca | 8 Sep 2010 17:03
Picon
Favicon

Cluster question

Hello all,

is there any way to run a parallel process on a cloud like amazon EC2 using 
GridR or Biocep?
I have no knowledge of these packages by reading the documentation it looks like 
it is capable of sending the request by ssh.
I just wanted to know if there is any complication in sending the calculation to 
the cluster without having to upload code and data, once the instance is active.

Thanks,
Riccardo

      
	[[alternative HTML version deleted]]
Permalink | Reply | 0 comments |
headers
clement | 10 Sep 2010 18:12
Picon

Re: Rmpi and ncdf4

Dear members,

I have started to adapt the "task pull" approach with Rmpi to 
parallelize my calculations on a NetCDF file of dimension lat[1:6], 
lon[1:6] and time[1:11323]. The basic idea is to parallelize the 
calculation by splitting the process according to each latitudinal 
coordinate i.e. for lat=1, lat=2, ..., lat=6. Please, see the code below 
and the link to the NetCDF file if you want to try : 
https://www.transferbigfiles.com/e4d7acd2-c186-4867-b4bf-82fe5aee4023?rid=jIcbZN%2blDcQz3kMeEHCQog%3d%3d

In the given example, the function works perfectly because the NetCDF 
file is quite small (~ 2 Mo). However, some difficulties occur when I 
try the function with larger NetCDF files (~ 8 Go), covering the entire 
globe, of dimensions lat[1:576], lon[1:1152] and time[1:11323]. The 
problem is that each processor (slave) needs to "connect" the NetCDF 
file before extracting the data required according to the latitudinal 
coordinate of interest. This connection takes more than 2 minutes.

I wondered if there was a possibility to avoid this problem of "time 
connection"  when a process is sent to a new slave. The best solution 
would be to open the connection to the NetCDF file in the master 
process, then to "send this connection" to the slaves. I do not know how 
it works exactly, but maybe there is something to check around the ncdf4 
package (which is based on the NetCDF4 library) to allow multiple 
connections to a single file... If you have ideas, I am opened !

Thanks in advance.

##path_job="/scratch/ctisseuil/job/Test_DWS_Rmpi"
path_job="/media/partage/Test_DWS_Rmpi"
(Continue reading)

Permalink | Reply | 3 comments |
headers
Kasper Daniel Hansen | 10 Sep 2010 18:57
Picon

Re: Rmpi and ncdf4

I suggest you start by looking through the netcdf documentation and
ask about this on their mailing lists.  It seems to me that the
problem is one of multiple processes accessing the same file
simultaneously.  I would guess that the netcdf people know whether it
is possible or not, and also what the interface should be (my guess is
that there is an interface).  The next step is then to figure out
whether ncdf4 implements this interface.

Kasper

On Fri, Sep 10, 2010 at 12:12 PM, clement
<clement.tisseuil@...> wrote:
> Dear members,
>
> I have started to adapt the "task pull" approach with Rmpi to parallelize my
> calculations on a NetCDF file of dimension lat[1:6], lon[1:6] and
> time[1:11323]. The basic idea is to parallelize the calculation by splitting
> the process according to each latitudinal coordinate i.e. for lat=1, lat=2,
> ..., lat=6. Please, see the code below and the link to the NetCDF file if
> you want to try :
> https://www.transferbigfiles.com/e4d7acd2-c186-4867-b4bf-82fe5aee4023?rid=jIcbZN%2blDcQz3kMeEHCQog%3d%3d
>
> In the given example, the function works perfectly because the NetCDF file
> is quite small (~ 2 Mo). However, some difficulties occur when I try the
> function with larger NetCDF files (~ 8 Go), covering the entire globe, of
> dimensions lat[1:576], lon[1:1152] and time[1:11323]. The problem is that
> each processor (slave) needs to "connect" the NetCDF file before extracting
> the data required according to the latitudinal coordinate of interest. This
> connection takes more than 2 minutes.
>
(Continue reading)

Permalink | Reply | 0 comments |
headers
Paul Johnson | 14 Sep 2010 20:30
Picon

Re: Rmpi and ncdf4

On Fri, Sep 10, 2010 at 11:12 AM, clement
<clement.tisseuil@...> wrote:
> Dear members,

>
>> I am using the ncdf4 package to work on General Circulation Model (GCM)
>> data (NetCDF file format) and I would like to parallelize some calculations
>> using Rmpi. Does anyone have an experience or advices in using Rmpi and
>> ncdf4 packages?
>>

Dear Clement:

Thanks for posting your code. It really helps me to learn when I can
read through what other people try.

I found myself wondering "how much" of the whole data set is used by
each slave.  Supposing the slave needs only a smaller piece,   I think
your problem would work more efficiently if you have the master load
the data one time and have it send the separate pieces to the slaves
for the work.  Well, that's what I would do because I've had very bad
experience when lots of nodes try to access the same file on NFS.  (It
causes something like a traffic jam as the processes fight over each
other).

Instead of doing system(rm ...), I'd suggest you clean files with the
file.remove function (see ?files) .  That will work across platforms,
so even people who use Windows might someday be able to run your code.

pj
(Continue reading)

Permalink | Reply | 1 comment |
headers
clement | 15 Sep 2010 09:49
Picon

Re: Rmpi and ncdf4

Dear Paul,

I do agree with your suggestion that I have successfully experimented. 
Namely, the NetCDF file is opened one time in the Master process. The 
subset of data required for the calculation is also extracted from the 
Master process before being sent to the available slave. When the 
calculation is done, results are sent to the Master process that writes 
the results to the NetCDF file. The "task pull" method (described for 
the Rmpi package at http://math.acadiau.ca/ACMMaC/Rmpi/task_pull.R) is 
particularly well adapted for this problem.

Thanks for your assistance.

Cheers

Clement

On 9/14/2010 8:30 PM, Paul Johnson wrote:
> On Fri, Sep 10, 2010 at 11:12 AM,
clement<clement.tisseuil@...>  wrote:
>    
>> Dear members,
>>      
>    
>>      
>>> I am using the ncdf4 package to work on General Circulation Model (GCM)
>>> data (NetCDF file format) and I would like to parallelize some calculations
>>> using Rmpi. Does anyone have an experience or advices in using Rmpi and
>>> ncdf4 packages?
>>>
(Continue reading)

Permalink | Reply | 0 comments |
headers
xsw1024 | 17 Sep 2010 10:42
Favicon

about the chechForRemoteError(val) of snowfall


Hello,everybody! I am a freshman to the parallel programming of R, it is said that snowfall is easy to
understand and easy to use, so I learned it for a short time.  I find it is easy to learn,but not so easy to use.
When I wrote some parallel codes to implement the kmeans,I met  some error,but I have no idea how to deal with
it, could anybody help me ,please ?

The codes are  :

library(snowfall)
sfInit(parallel=TRUE,cpus=2)
kmeans_label<-function(y){
    cl<-kmeans(x,centers)
    return(cl$cluster)
}

cutoff=0.5        #ãÐÖµ

x <- rbind(matrix(rnorm(100, sd = 0.3), ncol = 2),
           matrix(rnorm(100, mean = 1, sd = 0.3), ncol = 2))
colnames(x) <- c("x", "y")

getDistence<-function(x){
    return(x[1]**2+x[2]**2)

    }

centers<-x[1:5,]#Ëæ»úÑ¡È¡¼¸¸öµã
sfExport("x")#·¢Ë͵½¸÷¸öslave
sfExport("centers")
(Continue reading)

Permalink | Reply | 0 comments |
headers
=?UTF-8?Q? "Jens_Oehlschl=C3=A4gel" ?= | 21 Sep 2010 17:46

Re: using rbind to combine RODBC result sets using ffdf

Hi Steve,

>I have a fairly large (~4M rows, 27 double columns) data set I am attempting to 
>read in in R on Windows using RODBC and find I am running out of memory.

Fairly small under Rwin64 with enough RAM, and reasonably small for ff under RWin32.

>I thought I could easily workaround the problem by reading in smaller number of 
>rows and then rbind the results together as below:-

Fair approach, will work. So far we don't have an rbind for ffdf (but all pieces are there, see below). If we
had rbind.ffdf it ideally would cover the following situations

1) combing two (or more) ffdf into a third one
2) growing an existing ffdf by another (or more) ffdf
3) growing an existing ffdf by another (or more) data.frame

Technicaly all the pieces are there. You can 
first.ffdf <- as.ffdf(first.data.frame)
third.ffdf <- clone(first.ffdf)
and grow an ffdf by assigning nrow like in  
nrow(third.ffdf) <- nrow(third.ffdf) + nrow(second.ffdf)

Then the devil is in some details like maintaining appropriate factor levels. Example for how this works is
in read.table.ffdf. Should be easy to abstract a rbind.ffdf from this, think I will do this soon - but first
have to prepare the next release of ff which will support RWin64, sorting, ordering and using positive ff
integers as subscripts. If you abstract rbind.fdf from read.table.ffdf before I do, please let me know,
then I am happy to include in ff and give credit.

I recently had a similar request from Xiaobo Gu who uses a different data.base driver and who wrote code
(Continue reading)

Permalink | Reply | 1 comment |

Gmane