headers
mark starnes | 1 Nov 12:05
Picon

Re: Time taken to push. Adjusting MPI parameter for processor affinity. No effect.

Hi everyone,

A short update.  This multi-processor, shared memory machine shows no performance change
with the code snippet above, when I set 'processor affinty' to 1.  I got the tip from,

http://www.open-mpi.org/faq/?category=tuning

and remain interested in the results anyone else gets / tips for improving push times.

Best regards,

Mark.
Permalink | Reply | 7 comments |
headers
Brian Granger | 3 Nov 06:14
Picon

Re: Time taken to push. Adjusting MPI parameter for processor affinity. No effect.

Mark,

I don't see the "code snippet above." or your original question.
Could you repost so we can help?

But, from what you have written here, it looks like you are wanting
push to be faster?  One thing to keep in mind is that push/pull in
IPython don't use MPI, so no MPI tuning will improve their
performance.  But, we may have some ideas for you if you can give us a
better idea of what you are trying to do.

Thanks,

Brian

On Sat, Nov 1, 2008 at 3:05 AM, mark starnes <m.starnes05 <at> imperial.ac.uk> wrote:
> Hi everyone,
>
> A short update.  This multi-processor, shared memory machine shows no performance change
> with the code snippet above, when I set 'processor affinty' to 1.  I got the tip from,
>
> http://www.open-mpi.org/faq/?category=tuning
>
>
> and remain interested in the results anyone else gets / tips for improving push times.
>
> Best regards,
>
> Mark.
> _______________________________________________
(Continue reading)

Permalink | Reply | 1 comment |
headers
mark starnes | 3 Nov 13:21
Picon

Re: Time taken to push. Adjusting MPI parameter for processor affinity. No effect.

Hi Brain, thanks for the response.

I've included my message trail below, in chronological order.  It's becoming more interesting now, as
I'm about to start pushing 500Mb matrices around!

Best regards,

Mark.

----------
Hi everyone,

I'm pushing a string with 174E6 characters to the cores on my multi-core processor, and it's taking a long
time; more than
a few minutes per core.  I was expecting this to be fast.  Is this the behaviour I should expect?  If not, is there
a way to
accelerate the push?

Thanks in advance,

Mark.
------------
Hello again,

I got some results using the following snippet:

import time
from IPython.kernel import client
mec = client.MultiEngineClient()
res = []
(Continue reading)

Permalink | Reply | 0 comments |
headers
Gabriel Beckers | 4 Nov 17:48
Picon
Favicon

Re: threading and ipython -pylab problem

Many thanks Gaël, sorry for the noise. I am migrating my matplotlib
stuff to chaco, but are doing it step by step. So I indeed will need
both for a while. 'ipython -pylab -wthread' works perfectly. I don't
know why just the -pylab worked on the older configuration. Perhaps due
to some setting in matplotlibrc.

Best, Gabriel

On Tue, 2008-11-04 at 17:36 +0100, Gael Varoquaux wrote:
> On Tue, Nov 04, 2008 at 05:36:35PM +0100, Gabriel Beckers wrote:
> > Probably the wx is not a good idea?
> 
> You  cannot run Wx and GTK mainloops in the same process. If you want to
> use pylab and wx code, you need to you the WxAgg backend. If you run
> 'ipython -pylab -wthread', it should select the right backend for you
> automatically.
> 
> Gaël

_______________________________________________
IPython-user mailing list
IPython-user <at> scipy.org
http://lists.ipython.scipy.org/mailman/listinfo/ipython-user
Permalink | Reply | 0 comments |
headers
Brian Granger | 5 Nov 17:48
Picon

Re: Time taken to push. Adjusting MPI parameter for processor affinity. No effect.

Mark,

You are comparing apples (IPython.kernel) and oranges (PyPar+MPI).
PyPar uses MPI which is a peer to peer message passing architecture.
IPython uses a completely different architecture and doesn't use MPI
for its implementation.  Thus, it _should_ be slower than MPI.  I will
explain a bit more to clarify things...

* The IPythons architecture is not peer to peer.  Instead, all
processes connect to a central process (the IPython controller) which
manages the computation.  This architecture is required when you want
to do things interactively and be able to disconnect/reconnect to a
running parallel job.  Here is what things look like in IPython

Client
|
|
Controller--------Engine
|              \
|               \
Engine     Engine

All of these connections are handled using regular TCP sockets.

* We would like to optimize push/pull and other operations in IPython.
 At the same time, if you are sending really large objects from the
client to the engines, the best solution is for you to re-work your
algorithm to avoid this data transfer.  The same is true of using
PyPar+MPI.  Here are some tips:
(Continue reading)

Permalink | Reply | 4 comments |
headers
Lev Givon | 6 Nov 18:07
Favicon

engine registration failure on MacOSX

When I recently tried to run an ipython 0.9.1 controller and engine on
a MacOSX 10.5.5 system, I observed that the engine fails to register
with the controller when I attempt to run ipengine (the below assumes
the controller was already started):

2008-11-06 12:05:39-0500 [-] Log opened.
2008-11-06 12:05:40-0500 [-] Using furl file: /Users/lev/.ipython/security/ipcontroller-engine.furl
2008-11-06 12:06:10-0500 [-] 'engine registration failed:'
2008-11-06 12:06:10-0500 [-] Unhandled Error
        Traceback (most recent call last):
        Failure: twisted.internet.error.TimeoutError: User timeout
        caused connection failure.

2008-11-06 12:06:10-0500 [-] Main loop terminated.

This occurs regardless of whether I attempt to start the cluster with
ipcluster or whether I start an ipcontroller manually and then try to
start an ipengine instance (I made sure to clear out the contents of
~/.ipython/security before starting the processes). Why would this be
occurring? Could it have to do with my firewall settings? If so, would
I need to allow incoming connections from the host itself?

I'm running python 2.5.2, twisted 8.1.0, foolscap 0.3.2, pyopenssl
0.7, and zopeinterface 3.4.1 (all installed through Fink).

     	 	       	     	  	    	    L.G.
Permalink | Reply | 1 comment |
headers
Lev Givon | 6 Nov 19:04
Favicon

Re: engine registration failure on MacOSX

Received from Lev Givon on Thu, Nov 06, 2008 at 12:07:35PM EST:
> When I recently tried to run an ipython 0.9.1 controller and engine on
> a MacOSX 10.5.5 system, I observed that the engine fails to register
> with the controller when I attempt to run ipengine (the below assumes
> the controller was already started):
> 
> 2008-11-06 12:05:39-0500 [-] Log opened.
> 2008-11-06 12:05:40-0500 [-] Using furl file: /Users/lev/.ipython/security/ipcontroller-engine.furl
> 2008-11-06 12:06:10-0500 [-] 'engine registration failed:'
> 2008-11-06 12:06:10-0500 [-] Unhandled Error
>         Traceback (most recent call last):
>         Failure: twisted.internet.error.TimeoutError: User timeout
>         caused connection failure.
> 
> 2008-11-06 12:06:10-0500 [-] Main loop terminated.
> 
> This occurs regardless of whether I attempt to start the cluster with
> ipcluster or whether I start an ipcontroller manually and then try to
> start an ipengine instance (I made sure to clear out the contents of
> ~/.ipython/security before starting the processes). Why would this be
> occurring? Could it have to do with my firewall settings? If so, would
> I need to allow incoming connections from the host itself?
> 
> I'm running python 2.5.2, twisted 8.1.0, foolscap 0.3.2, pyopenssl
> 0.7, and zopeinterface 3.4.1 (all installed through Fink).
> 
>      	 	       	     	  	    	    L.G.

Found the answer: my firewall was blocking IP connections from
127.0.0.1.
(Continue reading)

Permalink | Reply | 0 comments |
headers
Prabhu Ramachandran | 9 Nov 07:57
Picon
Gravatar

Problem with %timeit and ascii codec

Hi,

I'm running IPython 0.9.1 and am running into a unicode encoding problem 
I haven't seen in earlier versions:

In [1]: import numpy

In [2]: a = numpy.linspace(0, 1, 10)

In [3]: timeit a.sum()
---------------------------------------------------------------------------
UnicodeEncodeError                        Traceback (most recent call last)

/home/prabhu/usr/virtualenv/ets_trunk/lib/python2.5/site-packages/IPython/iplib.pyc 
in ipmagic(self, arg_s)
     951         else:
     952             magic_args = self.var_expand(magic_args,1)
--> 953             return fn(magic_args)
     954
     955     def ipalias(self,arg_s):

/home/prabhu/usr/virtualenv/ets_trunk/lib/python2.5/site-packages/IPython/Magic.py 
in magic_timeit(self, parameter_s)
    1829 
precision,
    1830                                                           best 
* scaling[order],
-> 1831 
(units[order]))
    1832         if tc > tc_min:
(Continue reading)

Permalink | Reply | 3 comments |
headers
mark starnes | 9 Nov 10:24
Picon

Re: Time taken to push. Adjusting MPI parameter for processor affinity. No effect.

Hi Brian,

I see.  I changed from a push, to a, 'save to /tmp/filename, all engines
read from /tmp/filename', arrangement, and got a factor of 1000 speed up.
Thanks!

I would be interested in knowing how to call MPI from within IPython,
could you point me to a simple example?  Is there an Mpi4Py
manual?  (I noticed the Mpi4Py site appears to be a regular
target for vandalism - how annoying).

The algorithm builds a list of finite element objects (a few 10's of
thousands, each containing shape functions etc).  The elements
then update themselves for the required analysis frequency, a system
stiffness matrix and vector is built, boundary conditions applied, then
the matrix and vector are solved.  I wanted to build the list on one processor,
push the list  to all the others, followed by different frequencies
to each processor.  The solutions would be pulled back on completion,
and new frequencies sent out, as required.  It's not fine grained
parallelisation but it used the multi-processor machine I have access to.

Optimally, I would have liked to have used a more finely grained approach;
chopping the list into different segments over all the processors, performing
the frequency update, build, applied the b/c, and solved (keeping the system
matrix spread around the processors) but haven't got as far as finding a
matrix solver that would deal with the system matrix being spread around
this way.  This approach would result in the shared memory of each node
being used more efficiently, allowing for larger jobs.  If you know of a
solver that can do this, that would be very good news!
(Continue reading)

Permalink | Reply | 3 comments |
headers
Yichun Wei | 9 Nov 11:47
Picon

How to customize ipengine at its start?

I think Min's suggestion about initializing the engines at their start
is great if it really works. Such initialization can be followed with
non-trivial MapTasks.

But my attempt to do so failed. The source code shows currently
ipengine ignore every arg passed to it. Is there an easy way (other
than via MultiEngineController) through it I can have engines
executing customized init scripts or init python statement? I see some
configuration can be down similarly to what "--mpi" option is doing,
but not as strict forward as running a init scritpt like what Min has
proposed: "ipengine -s init.py"

Any suggestions will be highly appreciated.

On Thu, Nov 6, 2008 at 10:43 PM, Min Ragan-Kelley <benjaminrk <at> gmail.com> wrote:
> ...
>
> As for handling incoming engines not being initialized, it makes more sense
> to me to use the "ipengine -s <initscript.py>" to ensure that engines are
> initialized when they connect than for tasks to assume that the engine has
> not been initialized in the event of a failure.
> ...
>
> -MinRK
Permalink | Reply | 501 comments |

Gmane