headers
Jonathan Taylor | 2 Oct 19:07
Picon
Favicon

pushing/pulling objects between engines

Is it possible to send information between two engines using the 
RemoteController?
I can imagine wanting to do something like the following between two 
engines i and j

a_i, a_j = a_j, a_i

(not proper python syntax of course).

I didn't see an example of this in the tutorial.... other than

 >>> a_i, a_j = rc.pull(i, 'a'), rc.pull(j, 'a')
 >>> rc.push(j, 'a'=a_i); rc.push(i, 'a'=a_j)

Any suggestions?

Thanks,

Jonathan
Permalink | Reply | 2 comments |
headers
Fernando Perez | 2 Oct 22:41
Picon
Gravatar

Re: pushing/pulling objects between engines

Hi Jonathan,

On 10/2/07, Jonathan Taylor <jonathan.taylor <at> stanford.edu> wrote:
> Is it possible to send information between two engines using the
> RemoteController?
> I can imagine wanting to do something like the following between two
> engines i and j
>
> a_i, a_j = a_j, a_i
>
> (not proper python syntax of course).
>
> I didn't see an example of this in the tutorial.... other than
>
>  >>> a_i, a_j = rc.pull(i, 'a'), rc.pull(j, 'a')
>  >>> rc.push(j, 'a'=a_i); rc.push(i, 'a'=a_j)
>
> Any suggestions?

At the moment unfortunately that's it, but with a caveat (see below).
Currently the engines all connect to the controller explicitly, and
they don't listen on any other ports than that connection, so they
have no mechanism for sending directly data to one another.

The caveat is that the system is fully compatible with MPI, so you can
start your group of engines   as part of a single MPI world, and at
any point tell your engines to communicate with one another via MPI.

We need to look into cleaner ways to integrate inter-engine
communications in the future, but for now the MPI solution might do
(Continue reading)

Permalink | Reply | 0 comments |
headers
Brian Granger | 2 Oct 23:16
Picon

Re: pushing/pulling objects between engines

What Fernando said is true.  It is worth noting that one of the key
reasons that we don't have a custom way of sending objects directly
between engines is that MPI is the best way of doing this (it is much
better than anything we would come up with).

Thus, if you really need to do this, I would use mpi4py in conjunction
with ipython1.  Mpi4py is a absolutely fantastic project and ipython1
has full support for it.  For instance, if you have mpi4py installed,
you can start the engines using

mpirun -n 4 ipengine --mpi=mpi4py

>From that point on, you can send python command to the engines (using
execute/executeAll) that contain calls to mpi4py.  The benefit of this
approach is that if your cluster has a fast interconnect (like
infiniband) it will be used to send the objects between the engines.

If you go this route, let us know if you run into problems.

Cheers,

Brian

On 10/2/07, Jonathan Taylor <jonathan.taylor <at> stanford.edu> wrote:
> Is it possible to send information between two engines using the
> RemoteController?
> I can imagine wanting to do something like the following between two
> engines i and j
>
> a_i, a_j = a_j, a_i
(Continue reading)

Permalink | Reply | 0 comments |
headers
Brian Granger | 4 Oct 23:21
Picon

Enhancements to ipython1: pushing/pulling functions

Hi,

I have just checked in some changes to ipython1 that make it possible
to push and pull functions to and from engines:

In [1]: import ipython1.kernel.api as kernel
In [2]: rc = kernel.RemoteController(('127.0.0.1',10105))
In [3]: def f(x): return 2.0*x
   ...:

In [4]: rc.pushAll(f=f)                            # push the actual function!
Out[4]: [None, None, None, None]

In [5]: rc.executeAll('y=f(10)')             # now call the function
In [6]: rc.pullAll('y')
Out[6]: [20.0, 20.0, 20.0, 20.0]

We have also adapted the rc.map/mapAll and parallelize methods to take
functions as well:

In [7]: rc.mapAll(lambda x: x*x, range(16))
Out[7]: [0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100, 121, 144, 169, 196, 225]

Please try these new features out and let us know how they work.

Some limitations:

1.  Only functions and lambdas can be pushed and pulled.   Instances
of classes that are callables don't work unless the class is already
defined on the engines.
(Continue reading)

Permalink | Reply | 0 comments |
headers
David Strozzi | 6 Oct 21:57
Picon

dumb question on installing ipython1

Folks,

This is a fairly dumb question, reflecting more my python ignorance
probably, but...

I'm having trouble installing ipython1 on opensuse 10.3 (just
released) on amd64 hardware.  I svn'ed the most recent ipython1, and
">python setup.py install" gives an error (see below).  My
site-packages is in /usr/lib64/python/site-packages, but setup.py
wants to use /usr/local/lib64/....  I tried changing this via
--install-dir, which gives an error, and --prefix, which gives a
mangled dirname still w/ /usr/local/lib64 at the front.

How can I fix this?

Is something setup wrong in distutils (not that I'd know exactly how
to fix that)?  I noticed numpy has a distutils subdir with
/usr/local/lib64/ in some file, according to grep.

Thanks,
David

***************

> python setup.py install
running install
Checking .pth file support in /usr/local/lib64/python2.5/site-packages/
error: can't create or remove files in install directory

The following error occurred while trying to add or remove files in the
(Continue reading)

Permalink | Reply | 503 comments |
headers
LUK ShunTim | 7 Oct 16:03
Picon

Re: dumb question on installing ipython1

David Strozzi wrote:
> Folks,
> 
> This is a fairly dumb question, reflecting more my python ignorance
> probably, but...
> 
> I'm having trouble installing ipython1 on opensuse 10.3 (just
> released) on amd64 hardware.  I svn'ed the most recent ipython1, and
> ">python setup.py install" gives an error (see below).  My
> site-packages is in /usr/lib64/python/site-packages, but setup.py
> wants to use /usr/local/lib64/....  I tried changing this via
> --install-dir, which gives an error, and --prefix, which gives a
> mangled dirname still w/ /usr/local/lib64 at the front.
> 
> How can I fix this?
> 
> Is something setup wrong in distutils (not that I'd know exactly how
> to fix that)?  I noticed numpy has a distutils subdir with
> /usr/local/lib64/ in some file, according to grep.
> 
> Thanks,
> David
> 
> ***************
> 
>> python setup.py install
> running install
> Checking .pth file support in /usr/local/lib64/python2.5/site-packages/
> error: can't create or remove files in install directory
>
(Continue reading)

Permalink | Reply | 502 comments |
headers
David Strozzi | 7 Oct 21:17
Picon

Re: dumb question on installing ipython1

Hi,

Thanks.  I guess my fear is that since all my other python packages are in
/usr/lib64/python2.5/site-packages    [no local]
maybe something will get confused.  I took your advice nonetheless,
and 'trial ipython1' passes all tests.

My next dumb question is, when I run ipython, how do I know if it's
using the ipython1 core/kernel?  As a normal user, from the prompt I
can run ipython, I get "IPython 0.8.2....".  There's no ipython1 on my
paths.

Thanks again,
Dave

On 10/7/07, LUK ShunTim <shuntim.luk <at> polyu.edu.hk> wrote:
> David Strozzi wrote:
> > Folks,
> >
> > This is a fairly dumb question, reflecting more my python ignorance
> > probably, but...
> >
> > I'm having trouble installing ipython1 on opensuse 10.3 (just
> > released) on amd64 hardware.  I svn'ed the most recent ipython1, and
> > ">python setup.py install" gives an error (see below).  My
> > site-packages is in /usr/lib64/python/site-packages, but setup.py
> > wants to use /usr/local/lib64/....  I tried changing this via
> > --install-dir, which gives an error, and --prefix, which gives a
> > mangled dirname still w/ /usr/local/lib64 at the front.
> >
(Continue reading)

Permalink | Reply | 3 comments |
headers
LUK ShunTim | 8 Oct 16:29
Picon

Re: dumb question on installing ipython1

David Strozzi wrote:
> Hi,
> 
> Thanks.  I guess my fear is that since all my other python packages are in
> /usr/lib64/python2.5/site-packages    [no local]
> maybe something will get confused.  I took your advice nonetheless,
> and 'trial ipython1' passes all tests.
> 
> My next dumb question is, when I run ipython, how do I know if it's
> using the ipython1 core/kernel?  As a normal user, from the prompt I
> can run ipython, I get "IPython 0.8.2....".  There's no ipython1 on my
> paths.
> 
> Thanks again,
> Dave

Hello Dave,

Ipython1 != ipython. :-) I'm learning it by using it for parallel
computing. It's still a work in progress I believe. If you want the
interactive python shell, install ipython instead.

Regards,
ST
--
Permalink | Reply | 1 comment |
headers
Brian Granger | 8 Oct 18:07
Picon

Re: dumb question on installing ipython1

> I'm having trouble installing ipython1 on opensuse 10.3 (just
> released) on amd64 hardware.  I svn'ed the most recent ipython1, and
> ">python setup.py install" gives an error (see below).  My
> site-packages is in /usr/lib64/python/site-packages, but setup.py
> wants to use /usr/local/lib64/....  I tried changing this via
> --install-dir, which gives an error, and --prefix, which gives a
> mangled dirname still w/ /usr/local/lib64 at the front.

I am not very familiar with how suse manages /usr and /usr/local.  Are
you sure that you don't have 2 versions of Python installed (one in
/usr and one in /usr/local)?  Maybe do a "locate python" and which
python to make sure you are using the right python?

> ***************
>
> > python setup.py install
> running install
> Checking .pth file support in /usr/local/lib64/python2.5/site-packages/
> error: can't create or remove files in install directory

This almost looks like a permissions problem.  What if you run:

$ sudo python setup.py install

> The following error occurred while trying to add or remove files in the
> installation directory:
>
>     [Errno 2] No such file or directory:
> '/usr/local/lib64/python2.5/site-packages/test-easy-install-16651.pth'
>
(Continue reading)

Permalink | Reply | 0 comments |
headers
Brian Granger | 8 Oct 18:10
Picon

Re: dumb question on installing ipython1

ipython and ipython1 are separate downloads and installs.  Here are
the details of the full installation:

http://projects.scipy.org/ipython/ipython/browser/ipython/branches/saw/INSTALL

When you type ipython at the command line, you are getting regular old
IPython (just the enhanced interactive shell).  The ipython1 stuff is
a separate import.  See this page for more details on basic usage of
ipython1:

http://ipython.scipy.org/moin/Parallel_Computing_With_IPython1

Let me know how it goes.

Brian

On 10/7/07, David Strozzi <david.strozzi <at> gmail.com> wrote:
> Hi,
>
> Thanks.  I guess my fear is that since all my other python packages are in
> /usr/lib64/python2.5/site-packages    [no local]
> maybe something will get confused.  I took your advice nonetheless,
> and 'trial ipython1' passes all tests.
>
> My next dumb question is, when I run ipython, how do I know if it's
> using the ipython1 core/kernel?  As a normal user, from the prompt I
> can run ipython, I get "IPython 0.8.2....".  There's no ipython1 on my
> paths.
>
> Thanks again,
(Continue reading)

Permalink | Reply | 0 comments |

Gmane