headers
Wellenreuther, Gerd | 4 Feb 10:59
Picon
Favicon

CLI for Tango: Qt-widget freezes during sleep

Dear IPython community,

We just started to write some Python-modules to create something like a
lean-but-mean user-interface on top of Tango/PyTango. One Qt4-widget is
capable of showing several motor positions, and is being updated every
200 msec. Well, at least if I call it in a separate python session.

But of course I wanted to have just one IPython instance with -q4thread
running, which should have all widgets / classes / data etc. at hand. If
I am now e.g. starting to move several motors I have to wait for them to
finish their moves, and for that purpose I am using time.sleep(). Unlike
when using different python processes, no updates take place in the
motor-widget running in the q4thread. :(

So it appears to me that with using time.sleep() I also put the q4thread
to sleep ... is this really the case, and does anyone know how to
solve/circumvent this problem?

Cheers, Gerd

P.S.: Anyone from ALBA or the ESRF reading this?
--

-- 
Dr. Gerd Wellenreuther
beamline scientist P06 "Hard X-Ray Micro/Nano-Probe"
Petra III project
HASYLAB at DESY
Notkestr. 85
22603 Hamburg

Tel.: + 49 40 8998 5701
(Continue reading)

Permalink | Reply | 3 comments |
headers
Justin MacCallum | 4 Feb 21:46

Problem calling subprocess from TaskClient

Hi,

I have a parallel scientific code that runs on clusters. It currently uses my own (badly designed) parallel
communication engine and I'm trying to transition to iPython's TaskClient interface. One part of my code
uses the subprocess module to wrap a call to a different piece of software. Unfortunately, this seems to be
causing problems for me. I'm running on OS X 10.6.2, with python 2.6.4 and ipython 0.10 as supplied by
MacPorts. The following code snippet will reproduce the problem I'm having.

-----
#!/usr/bin/env python

from IPython.kernel import client

tc = client.TaskClient()

@tc.parallel()
def remote_test(input):
    import subprocess
    # I'm obviously not wrapping ls, but I have the same problem with the real binary I'm trying to call
    subprocess.check_call('ls')
    return input

work = range(100)

results = remote_test(work)
-----

The output of this program
is:
(Continue reading)

Permalink | Reply | 1 comment |
headers
Fernando Perez | 4 Feb 21:52
Picon
Gravatar

Re: CLI for Tango: Qt-widget freezes during sleep

Hi Gerd,

On Thu, Feb 4, 2010 at 4:59 AM, Wellenreuther, Gerd
<gerd.wellenreuther <at> desy.de> wrote:
> Dear IPython community,
>
> We just started to write some Python-modules to create something like a
> lean-but-mean user-interface on top of Tango/PyTango. One Qt4-widget is
> capable of showing several motor positions, and is being updated every
> 200 msec. Well, at least if I call it in a separate python session.
>
> But of course I wanted to have just one IPython instance with -q4thread
> running, which should have all widgets / classes / data etc. at hand. If
> I am now e.g. starting to move several motors I have to wait for them to
> finish their moves, and for that purpose I am using time.sleep(). Unlike
> when using different python processes, no updates take place in the
> motor-widget running in the q4thread. :(
>
> So it appears to me that with using time.sleep() I also put the q4thread
> to sleep ... is this really the case, and does anyone know how to
> solve/circumvent this problem?

I'm afraid I don't really know for sure, Darren Dale might be able to pitch in.

But it's worth mentioning to you that in the upcoming 0.11, the gui
support has been completely overhauled.  We know there are still some
bugs in there, but it would be great if you and others using ipython
with gui tools can begin testing things out, from the bzr trunk
branch:
(Continue reading)

Permalink | Reply | 0 comments |
headers
Brian Granger | 5 Feb 01:29
Picon

Re: Problem calling subprocess from TaskClient

Justin,

I didn't know about this bug.  Thanks for the simply code that shows it. Can you file a bug report for this as a reminder for me to look at this:

https://bugs.launchpad.net/ipython

Cheers,

Brian

On Thu, Feb 4, 2010 at 12:46 PM, Justin MacCallum <justin.maccallum <at> me.com> wrote:
Hi,

I have a parallel scientific code that runs on clusters. It currently uses my own (badly designed) parallel communication engine and I'm trying to transition to iPython's TaskClient interface. One part of my code uses the subprocess module to wrap a call to a different piece of software. Unfortunately, this seems to be causing problems for me. I'm running on OS X 10.6.2, with python 2.6.4 and ipython 0.10 as supplied by MacPorts. The following code snippet will reproduce the problem I'm having.

-----
#!/usr/bin/env python

from IPython.kernel import client

tc = client.TaskClient()

<at> tc.parallel()
def remote_test(input):
   import subprocess
   # I'm obviously not wrapping ls, but I have the same problem with the real binary I'm trying to call
   subprocess.check_call('ls')
   return input

work = range(100)

results = remote_test(work)
-----


The output of this program is:

-----
/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-macosx-10.6-i386.egg/twisted/python/filepath.py:12: DeprecationWarning: the sha module is deprecated; use the hashlib module instead
 import sha
/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/IPython/kernel/clientconnector.py:43: DeprecationWarning: Importing class Tub directly from 'foolscap' is deprecated since Foolscap 0.4.3. Please import foolscap.api.Tub instead
 self.tub = Tub()
/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/IPython/kernel/taskfc.py:79: DeprecationWarning: Importing class Referenceable1 directly from 'foolscap' is deprecated since Foolscap 0.4.3. Please import foolscap.api.Referenceable instead
 class FCTaskControllerFromTaskController(Referenceable):
Traceback (most recent call last):
 File "./test.py", line 15, in <module>
   results = remote_test(work)
 File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/IPython/kernel/parallelfunction.py", line 104, in call_function
   return self.mapper.map(self.func, *sequences)
 File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/IPython/kernel/mapper.py", line 230, in map
   task_results = [self.task_controller.get_task_result(tid) for tid in task_ids]
 File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/IPython/kernel/taskclient.py", line 93, in get_task_result
   taskid, block)
 File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/IPython/kernel/twistedutil.py", line 72, in blockingCallFromThread
   return twisted.internet.threads.blockingCallFromThread(reactor, f, *a, **kw)
 File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-macosx-10.6-i386.egg/twisted/internet/threads.py", line 114, in blockingCallFromThread
   result.raiseException()
 File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-macosx-10.6-i386.egg/twisted/python/failure.py", line 326, in raiseException
   raise self.type, self.value, self.tb
OSError: [Errno 4] Interrupted system call
-----



The ipcontroller log file looks like:

-----
2010-02-04 12:42:36-0800 [-] Running task 82 on worker 1
2010-02-04 12:42:36-0800 [Negotiation,2,192.168.0.194] Task completed: 82
2010-02-04 12:42:36-0800 [Negotiation,2,192.168.0.194] distributing Tasks
2010-02-04 12:42:36-0800 [Negotiation,2,192.168.0.194] Running task 83 on worker 1
2010-02-04 12:42:36-0800 [Negotiation,2,192.168.0.194] Task 83 failed on worker 1
2010-02-04 12:42:36-0800 [-] distributing Tasks
2010-02-04 12:42:36-0800 [-] Running task 84 on worker 0
2010-02-04 12:42:36-0800 [Negotiation,0,192.168.0.194] Task 84 failed on worker 0
2010-02-04 12:42:37-0800 [-] distributing Tasks
2010-02-04 12:42:37-0800 [-] Running task 85 on worker 1
2010-02-04 12:42:37-0800 [Negotiation,2,192.168.0.194] Task 85 failed on worker 1
2010-02-04 12:42:37-0800 [-] distributing Tasks
2010-02-04 12:42:37-0800 [-] Running task 86 on worker 0
2010-02-04 12:42:37-0800 [Negotiation,0,192.168.0.194] Task 86 failed on worker 0
2010-02-04 12:42:38-0800 [-] distributing Tasks
2010-02-04 12:42:38-0800 [-] Running task 87 on worker 1
-----

Notice that a few of the tasks actually complete, while the majority fail with the strange interrupted system call error.

I'd appreciate any help anyone can offer.

Thanks,
Justin

-----
Justin MacCallum
Post Doctoral Fellow
University of California San Francisco

_______________________________________________
IPython-user mailing list
IPython-user <at> scipy.org
http://mail.scipy.org/mailman/listinfo/ipython-user

_______________________________________________
IPython-user mailing list
IPython-user <at> scipy.org
http://mail.scipy.org/mailman/listinfo/ipython-user
Permalink | Reply | 0 comments |
headers
Darren Dale | 5 Feb 16:57
Picon
Gravatar

Re: CLI for Tango: Qt-widget freezes during sleep

On Thu, Feb 4, 2010 at 4:59 AM, Wellenreuther, Gerd
<gerd.wellenreuther <at> desy.de> wrote:
> Dear IPython community,
>
> We just started to write some Python-modules to create something like a
> lean-but-mean user-interface on top of Tango/PyTango. One Qt4-widget is
> capable of showing several motor positions, and is being updated every
> 200 msec. Well, at least if I call it in a separate python session.
>
> But of course I wanted to have just one IPython instance with -q4thread
> running, which should have all widgets / classes / data etc. at hand. If
> I am now e.g. starting to move several motors I have to wait for them to
> finish their moves, and for that purpose I am using time.sleep(). Unlike
> when using different python processes, no updates take place in the
> motor-widget running in the q4thread. :(
>
> So it appears to me that with using time.sleep() I also put the q4thread
> to sleep ... is this really the case, and does anyone know how to
> solve/circumvent this problem?

I doubt that you are putting the q4thread to sleep. It's hard to
comment without additional information about the implementation. Are
you calling time.sleep() in the same thread that is taking care of
dispatching update events to your qt4 widget? Can you assemble a
*short*, *self-contained* example that reproduces the problem?

Darren
Permalink | Reply | 1 comment |
headers
Wellenreuther, Gerd | 5 Feb 17:11
Picon
Favicon

Re: CLI for Tango: Qt-widget freezes during sleep


Darren Dale wrote:
> I doubt that you are putting the q4thread to sleep. It's hard to
> comment without additional information about the implementation. Are
> you calling time.sleep() in the same thread that is taking care of
> dispatching update events to your qt4 widget? 

I am launching IPyhton with the -q4thread-option. I run a startup-script 
which is building the widget, connecting the signals (e.g. the timer for 
the automatic update) and showing the corresponding widget in the end.

But all the signalling etc. should then run in the separate q4thread, or 
am I wrong?

> Can you assemble a
> *short*, *self-contained* example that reproduces the problem?

I can, but that could take some time. Will try.

Cheers, Gerd

--

-- 
Dr. Gerd Wellenreuther
beamline scientist P06 "Hard X-Ray Micro/Nano-Probe"
Petra III project
HASYLAB at DESY
Notkestr. 85
22603 Hamburg

Tel.: + 49 40 8998 5701
Permalink | Reply | 0 comments |
headers
Hassan Rom | 6 Feb 20:09
Picon

autocompletion of normal programs

Hello,

I'm running ipython with the shell profile and it works great! The
only problem I have with it currently is that ipython doesn't do
autocompletion for programs which would otherwise autocomplete fine if
I were to run them on bash. Is there a way to magically get these
programs to autocomplete in ipython too?

Thanks,
- Hassan
Permalink | Reply | 1 comment |
headers
Ville M. Vainio | 8 Feb 08:26
Picon
Gravatar

Re: autocompletion of normal programs

On Sat, Feb 6, 2010 at 9:09 PM, Hassan Rom <putera.afandy <at> gmail.com> wrote:

> I'm running ipython with the shell profile and it works great! The
> only problem I have with it currently is that ipython doesn't do
> autocompletion for programs which would otherwise autocomplete fine if
> I were to run them on bash. Is there a way to magically get these
> programs to autocomplete in ipython too?

Are you thinking of custom "bash completions":

http://www.debian-administration.org/article/An_introduction_to_bash_completion_part_1

As of yet there is no magical way to do it, but I suppose the
machinery can be used externally (invoking bash to do the work?).

--

-- 
Ville M. Vainio
http://tinyurl.com/vainio
Permalink | Reply | 0 comments |
headers
Jon Olav Vik | 8 Feb 20:11
Picon

How is a TaskClient "fault tolerant"? And can it play nice with PBS queueing?

I'm acquainting myself with parallel IPython and have a couple of questions.

1. Could someone please explain what it means that a TaskClient is "fault 
tolerant"?
http://ipython.scipy.org/doc/stable/html/parallel/parallel_task.html

2. The task interface sounds useful for embarrassingly parallel computations. 
I'm trying to follow the instructions at 
http://ipython.scipy.org/doc/stable/html/parallel/parallel_process.html#using-
ipcluster-in-pbs-mode
(PBS is the queueing system used by the computer cluster I'm working with).

I use the command 
ipcluster pbs -n 8 --pbs-script=pbs.template &
to run the following pbs script:

#PBS -N ipython
#PBS -j oe
#PBS -l walltime=00:10:00
#PBS -l nodes=${n/8}:ppn=8
#PBS -q express
cd $$PBS_O_WORKDIR
mpiexec -n ${n} ipengine --logfile=$$PBS_O_WORKDIR/ipengine &
sleep 30
python ipar.py

...where ipar.py starts a MultiEngineClient and execute()'s commands that use 
MPI on the ipengines. (I haven't tried using it with a TaskClient yet.)

Note that I'm starting mpiexec in the background; otherwise, it would never 
finish and my Python script would never get called. Also, I'm backgrounding the 
call to ipcluster because that too never seems to finish. (Using mpiexec with 
"python ipar.py" does not seem to be required.)

However, the compute cluster's user instructions say I shouldn't start 
processes in the background, because then they escape the control of the job 
scheduler. Is there a way I can make TaskClient() work under this restriction? 
Otherwise, I'm just going to manually "killall ipcluster" etc. once my job is 
done. (Or maybe that could go as the last lines of my pbs script?)

I'm a complete newbie in this, so any hints are highly appreciated.

Best regards,
Jon Olav Vik
Permalink | Reply | 3 comments |
headers
killian koepsell | 9 Feb 02:42
Picon

problem with ipython emacs mode

Hi,

I have a problem with tab completion in the (i)python mode of emacs
and am wondering if someone came across the same problem.

The problem is the following. If I press np.<TAB>, I get the following:

In [2]: import numpy as np

In [3]:    ...: np.

After a second press of the tab button, I get:

In [2]: import numpy as np

In [3]:    ...:
In [4]: ------------------------------------------------------------np.

Here is some system information: Mac OS 10.5.8
I haven't changed the emacs version (carbon emacs 22.3.1) and the
version of the python-mode.el and ipython.el recently. However, I
upgraded from python 2.5.4 (EPD 5.0.0) to python 2.6.4 (EPD 6.0.0) and
from IPython 0.9.1 to IPython 0.10. The problem does not seem to be
specific to the IPython version, since installing IPython 0.9.1 on top
of EPD 6 did not solve it.

Thanks,
  Kilian
Permalink | Reply | 6 comments |

Gmane