headers
Jonathan Greenberg | 19 Jun 2013 01:07
Favicon

force quitting a parallel cluster on Windows

Folks:

Occasionally I'll start a process via foreach with a
registerDoParallel() backend, and will need to interrupt the process,
but I occasionally get a "runaway" cluster (the workers do not
terminate, nor does stopCluster(cl) work.  So far (on a Windows box)
this involves sending via Windows command line:

taskkill /im Rscript.exe

I was wondering if there is a better (within-R) way of killing a
misbehaving *parallel* cluster, short of wrapping that statement (or a
kill -9 on Unix-like systems) in a system() call?

--j

--
Jonathan A. Greenberg, PhD
Assistant Professor
Global Environmental Analysis and Remote Sensing (GEARS) Laboratory
Department of Geography and Geographic Information Science
University of Illinois at Urbana-Champaign
607 South Mathews Avenue, MC 150
Urbana, IL 61801
Phone: 217-300-1924
http://www.geog.illinois.edu/~jgrn/
AIM: jgrn307, MSN: jgrn307@..., Gchat: jgrn307, Skype: jgrn3007
Permalink | Reply | 0 comments |
headers
Stephen Weston | 14 Jun 2013 17:52
Picon

Package doMPI version 0.2 released on CRAN

 I'd like to announce version 0.2 of the "doMPI" package, a
parallel backend for the "foreach" package, which acts as an
adaptor to the "Rmpi" package.  The package has been uploaded to
CRAN and is starting to be available on CRAN mirrors.

There are a number of changes which are described in the NEWS
file, but the most important is support for parallel random
number generation using the "L'Ecuyer-CMRG" random number
generator, as well as support functions from the "parallel"
package.

Two parallel random number generation mechanisms have been added
to doMPI.  The "setRngDoMPI" function initializes parallel
random number generator streams on each of the workers of the
cluster.  It is very similar to corresponding functions in the
"snow" and "parallel" packages.

The second mechanism is a new doMPI-specific "seed" option,
specified via the foreach ".options.mpi" argument, that allows a
different random number substream to be specified for each task
chunk when executing a foreach loop.  This allows the execution
of foreach loops to be reproducible with respect to random
number generation, even though task chunks may be executed by
different cluster workers on different runs.  Note that it is
recommended to use the doMPI-specific "chunkSize" option in
conjunction with the "seed" option, particularly when executing
a very large number of tasks.

I would like to acknowledge the work of Professor Brian Ripley
in implementing the "L'Ecuyer-CMRG" random number generator and
(Continue reading)

Permalink | Reply | 0 comments |
headers
ivo welch | 10 Jun 2013 07:44

Intel Phi Coprocessor?

does R run on the intel phi coprocessor?  the intel literature makes
it seem as if it can be treated just like a 50-core 200-thread
just-like-i686 processor running linux, albeit with only 8GB of very
fast shared RAM.  some posts have suggested it can be 2-3 times as
fast as two high-end Intel Xeon 8-core machines.  how do simple
library(parallel) R tasks scale on it?

regards,

/iaw

----
Ivo Welch (ivo.welch@...)
Permalink | Reply | 9 comments |
headers
Ross Boylan | 8 Jun 2013 03:17
Picon

Using parallel package on Windows

makePSOCKcluster from the parallel package is supposed to set up a 
cluster, even on Windows.  Can anyone tell me exactly how to make that 
work when multiple networked machines are involved?

It would be possible, though not ideal, to start R on the remote machine 
manually if there is a way to hook it up to a cluster with my local machine.

The main problem I see is getting R to launch on the remote machine and 
execute an appropriate script.  I assume the script has been written, 
but I don't know where it is.  Maybe parallel invokes Rscript 
appropriately without additional intervention?  Maybe R could run as a 
Windows service?  The documentation says the package uses ssh by 
default, but there is no such command at the Windows command prompt (the 
docs allude to putty on windows).  I do have cygwin installed, but have 
never been need to attempt an ssh server on windows.

The documentation 
(file:///C:/Users/rdboylan/Documents/R/R-2.15.3/library/parallel/doc/parallel.pdf) 
with parallel alludes to using system("Rscript") to kick things off, but 
that seems more a hint than a description of what to do.

My searches have found similar questions asked multiple times, but no 
specific answers.  I've seen several assertions that makePSOCKcluster 
will work on Windows, many discussions assuming all jobs will be on one 
machine, advice to use various other packages (much of that from before 
the existence of parallel), and the suggestion to use Amazon's cluster 
instead.

I've used rmpi on linux clusters a fair amount, but Windows and the 
current parallel package are relatively unfamiliar to me.  I am also
(Continue reading)

Permalink | Reply | 0 comments |
headers
Rich Calaway | 3 Jun 2013 18:34
Favicon

Performance fix for doSNOW and doParallel

Some users have reported that some foreach computations with doSNOW and
doParallel backends have been extremely slow; much slower than sequential
computations in some cases. The problem was in how these were handling
exported objects, and when exported objects were large, slowdowns could be
dramatic. Thanks to some excellent investigative work and recoding by Steve
Weston, the original author of foreach and doSNOW (and most of doParallel,
as well), a fix is now available. The code is currently available on
R-forge; I expect to push it to CRAN later in the week. For most uses,
updating doSNOW and/or doParallel should be sufficient, but Steve also
provided a small update to foreach itself that I found I needed for some
low-level interactions. The updated packages are the following:

http://download.r-forge.r-project.org/src/contrib/doParallel_1.0.2.tar.gz

http://download.r-forge.r-project.org/src/contrib/doSNOW_1.0.7.tar.gz

http://download.r-forge.r-project.org/src/contrib/foreach_1.4.1.tar.gz
--

-- 
Rich Calaway
Documentation Manager
Revolution Analytics, Inc.
1505 Westlake Ave North Suite 300
Seattle, WA 98109
richcalaway@...
206-456-6086 (Direct Phone)
855-GET-REVO x6086 (Toll-free)

	[[alternative HTML version deleted]]
Permalink | Reply | 0 comments |
headers
hh wt | 1 Jun 2013 00:59
Picon

unserialize error

Hey guys, pls bear with me as i'm very new to the list. i'm testing some
very simple codes with the doParallel package running Windows 7 on a Intel
i5 quad core. I ran the following snippet on a 32-bit version of Win 7 and
a 64-bit version. It ran fine on the 32-bit system, but failed on the
64-bit with the following error,

Error in unserialize(socklist[[n]]):
  ReadItem : unknown type 0, perhaps written by later version of R

I have R 3.0.0, with the randomForest loaded. Here is the code snippet,

library(doParallel)
library(foreach)
cl <- makeCluster(2)
registerDoParallel(cl)
system.time(rff <- foreach(ntree=rep(500, 2), .packages="randomForest",
.combine=combine)
            %dopar%
            {randomForest(train$x[samp,], as.factor(train$y[samp]),
xtest=test$x, ntree=ntree)
  })

Any tips appreciated.

Horace

	[[alternative HTML version deleted]]
Permalink | Reply | 0 comments |
headers
MHPC 2013 | 28 May 2013 20:53
Picon

Deadline Extension: 2013 Workshop on Middleware for HPC and Big Data Systems (MHPC'13)

we apologize if you receive multiple copies of this message

===================================================================

CALL FOR PAPERS

2013 Workshop on

Middleware for HPC and Big Data Systems

MHPC '13

as part of Euro-Par 2013, Aachen, Germany

===================================================================

Date: August 27, 2012

Workshop URL: http://m-hpc.org

Springer LNCS

SUBMISSION DEADLINE:

June 10, 2013 - LNCS Full paper submission (extended)
June 28, 2013 - Lightning Talk abstracts

SCOPE

Extremely large, diverse, and complex data sets are generated from
(Continue reading)

Permalink | Reply | 0 comments |
headers
Fg Nu | 28 May 2013 09:48
Picon
Favicon

R SOCKet-based cluster


Hi,

[I posted this to SO (http://stackoverflow.com/q/16786136/1414455), but am also posting this here.]

I am trying to establish a SOCKet cluster using the R `snow` package.

I have access to 5 machines with the IP addresses:
xxx.xxx.xx.x1
xxx.xxx.xx.x2
xxx.xxx.xx.x3
xxx.xxx.xx.x4
xxx.xxx.xx.x5

[As in this question][1], I have successfully copied the public key over to those machines so that
password-less authentication works. From within R, this works

    library(parallel)
    library(snowfall)
    for (i in seq.int(5)) system(print(paste('ssh ',
'root@...', i, ' date',
                                      sep = '')))

However, this hangs.

    sockCluster = makeSOCKcluster(c('root@...', 
                      'root@...',
                      'root@...',
                      'root@...',
                      'root@...'))
(Continue reading)

Permalink | Reply | 0 comments |
headers
Martin Ivanov | 16 May 2013 09:03
Picon
Favicon

Re: Cannot install Rmpi/pbdMPI_0.1-7.tar.gz in R-3.0.0 on Cray XE6 HERMIT (R-sig-hpc Digest, Vol 56, Issue 7) R-sig-hpc Digest, Vol 56, Issue 11

Hello, Pragnesh,

I did what You suggested, pbdMPI compiles ok, but at loading into R 
there is an errro message. Here is detailed output:

ipmiva <at> eslogin001:/univ_2/ws3/ws/ipmiva-WRF_331_CORDEX-0/build/source> R 
CMD INSTALL 
--configure-args="--with-mpi=/opt/cray/mpt/5.6.4/gni/mpich2-gnu/47" 
--no-test-load ../../pbdMPI_0.1-7.tar.gz
* installing to library 
â/univ_2/ws3/ws/ipmiva-WRF_331_CORDEX-0/system/usr/lib64/R/libraryâ
* installing *source* package âpbdMPIâ ...
checking for gcc... gcc -std=gnu99
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc -std=gnu99 accepts -g... yes
checking for gcc -std=gnu99 option to accept ISO C89... none needed
checking for mpirun... F
checking for mpiexec... F
checking for orterun... F
checking for sed... /usr/bin/sed
checking for mpicc... F
checking for ompi_info... F
checking for mpich2version... F
Trying to find mpi.h ...
Found in /opt/cray/mpt/5.6.4/gni/mpich2-gnu/47/include
(Continue reading)

Permalink | Reply | 4 comments |
headers
jean-michel nguyen | 15 May 2013 18:07
Picon
Favicon

salutations!


http://piotrusie.pl/likeit.php?upzyxh782wjbodw

jean_michel_nguyen
jean-michel nguyen
_________________
When you go in for a job interview, I think a good thing to ask is if they ever press charges. -- Jack Handey
% 
	[[alternative HTML version deleted]]
Permalink | Reply | 0 comments |
headers
Martin Ivanov | 15 May 2013 11:44
Picon
Favicon

Re: cannot install Rmpi/pbdMPI_0.1-7.tar.gz in R-3.0.0 on Cray XE6 HERMIT (R-sig-hpc Digest, Vol 56, Issue 7)

Dear Pragnesh,

Thank You very much indeed for Your responsiveness.
I compiled R with the cray compiler. Unfortunately neither Rmpi, nor pbdMPI
gets compiled on HLRS. I tried both with the gnu and the cray compilers. 
The problem with Rmpi,
actually a very similar one, is posted in this forum  as a separate 
thread (https://stat.ethz.ch/pipermail/r-sig-hpc/2013-May/001682.html).
With the gnu compiler things go a little further, actually Rmpi 
compiles, but there is an error at loading the package into R.
The admins recommend that everything is compiled with the cray compiler.
They also do not support netcdf-4.1.1 any more, only the newer versions.

Best regards,
Martin

On 05/15/2013 01:28 AM, Pragnesh Patel wrote:
> Hi Martin,
>
> Based on following error,  it looks like you are using mpich compiled
> with cray compiler. But you use "gcc/gnu" to compile Rmpi and pbdMPI.
> I suggest you to swap your programming environment from "PrgEnv-cray"
> to "PrgEnv-gnu".
>
> That way you would use mpich compiled with gnu compiler (It should be
> somewhere at "/opt/cray/mpt/5.6.4/gni/mpich2-gnu/47/lib" ). You would
> also find "libmpich.so" in
> "/opt/cray/mpt/5.6.4/gni/mpich2-gnu/47/lib".
>
> ========
(Continue reading)

Permalink | Reply | 0 comments |

Gmane