T J | 1 Sep 01:41 2008
Picon

[Cython] Lists?

Hi,

New to cython, and I found the sphinx documentation very helpful.
However, I didn't find much about dictionaries and lists.  How does
cython handle these....for example, if I were to have a list of
strings....is there a way to inform cython about this so that it could
be compiled?  I'm assuming non-homogeneous lists are not candidates
for optimizations (correct?).  In general, I guess I am looking for
more discussion/help about such topics.  Any pointers would be
appreciated.

Thanks.
Greg Ewing | 1 Sep 03:23 2008
Picon
Picon

Re: [Cython] 0.9.8.1.1 and .pxd files

Stefan Behnel wrote:

> To clarify a bit, the only difference is that Cython knows both, but Python
> only recognises __init__.py.

The idea of allowing __init__.pyx was so that the main
code of a package could be written in Pyrex.

However, I've never actually tested whether Python
recognises an __init__.so file as a package main
file, so I'm not sure if this works.

--

-- 
Greg
Stefan Behnel | 1 Sep 07:42 2008
Picon

Re: [Cython] Lists?

Hi,

T J wrote:
> New to cython, and I found the sphinx documentation very helpful.
> However, I didn't find much about dictionaries and lists.  How does
> cython handle these....

It uses Python's list/dict type.

> for example, if I were to have a list of
> strings....is there a way to inform cython about this so that it could
> be compiled?

The creation of such a list runs in compiled code. But you cannot specify the
type of the list content. And I don't see a use case for that.

> I'm assuming non-homogeneous lists are not candidates
> for optimizations (correct?).

There is no difference between a list with homogeneous content and one with
mixed content. What would you like to see 'optimised'?

> In general, I guess I am looking for
> more discussion/help about such topics.  Any pointers would be
> appreciated.

Maybe you could tell us what you want to achieve? That would make it easier to
answer your question.

Stefan
(Continue reading)

Hoyt Koepke | 1 Sep 19:41 2008
Picon
Picon

Re: [Cython] Lists?

Hi,

This is actually something I am wondering about as well, so I hope
it's okay to join the discussion.  My understanding is that when you
declare a variable to have a list, tuple, or dict type, then cython
optimizes it by using faster direct ways to accessing the elements
when you use them. However, since these containers always store the
elements as PyObject* pointers which hold any type, it's too difficult
to use element type information in a robust way.  Is that correct?

What is the most "cythonic" way to mimic the list functionality when
you do have homogeneous list content?  I have run into cases in
numerical calculations where I need to be append an arbitrary number
of numbers to a list (e.g. accumulating data statistics while
processing large amounts of data).  In such contexts, using a list is
too slow. One thing I've done is use a numpy array with a significant
amount of extra space, keep a separate counter to denote the end, and
access it through the nice buffer interface.  This works pretty well,
but seems quite clunky compared to how natural it is in python.  Doing
slicing and concatenation stuff would also be nice.

Personally, what I think would be really nice is a unified wrapper to
the C++ STL containers and some of the libraries with easy conversion
functions between the parallel python containers.  I know that
wrapping an STL vector is one of the examples given, so this shouldn't
be too hard.  I don't have time for at least a few weeks, and there
are several other important things on my to-do list, but I'll try to
look into it.

That said, I am pretty new still to cython, so please correct me if my
(Continue reading)

Robert Bradshaw | 1 Sep 20:27 2008

Re: [Cython] Lists?

On Sep 1, 2008, at 10:41 AM, Hoyt Koepke wrote:

> Hi,
>
> This is actually something I am wondering about as well, so I hope
> it's okay to join the discussion.  My understanding is that when you
> declare a variable to have a list, tuple, or dict type, then cython
> optimizes it by using faster direct ways to accessing the elements
> when you use them. However, since these containers always store the
> elements as PyObject* pointers which hold any type, it's too difficult
> to use element type information in a robust way.  Is that correct?

Exactly.

> What is the most "cythonic" way to mimic the list functionality when
> you do have homogeneous list content?  I have run into cases in
> numerical calculations where I need to be append an arbitrary number
> of numbers to a list (e.g. accumulating data statistics while
> processing large amounts of data).  In such contexts, using a list is
> too slow. One thing I've done is use a numpy array with a significant
> amount of extra space, keep a separate counter to denote the end, and
> access it through the nice buffer interface.  This works pretty well,
> but seems quite clunky compared to how natural it is in python.  Doing
> slicing and concatenation stuff would also be nice.

The traditional way to do this is the same as in C, with malloc,  
realloc, etc. Then one can create actual int*, double*, etc. as in C.  
Of course then one has to worry about memory leaks and/or  
segfaults... The Numpy approach is a good one too, and there is a  
resize() method.
(Continue reading)

Christoph Würstle | 1 Sep 20:51 2008
Picon

[Cython] Cython and Gtk signals

Hi,
I'm new to cython and try to optimize a gtk program. So far I'm really 
impressed how easy it is.
So far I ask me two things:
First, does anyone has experience in the speedup of gtk apps?
And second, how can I translate my own signals to cython code?

In Python I write:

class myClass(gtk.VBox):
    __gsignals__ = {
            'mysignal' : (gobject.SIGNAL_RUN_LAST, 
gobject.TYPE_NONE,(gobject.TYPE_STRING,gobject.TYPE_STRING)),
    }

and Cython does not complain when 'compiling', but I can't connect to it:

TypeError: <GtkProgramName object at 0x8482054 (GtkProgramModul at 
0x8588100)>: unknown signal name: mysignal

Thanks a lot for your help.
Chris
Hoyt Koepke | 1 Sep 20:56 2008
Picon
Picon

Re: [Cython] Lists?

Hi,

> That would be excellent! It would necessitate using C++, which is not
> going to be the default Cython output but would be very hand for
> people wanting to use it.

Okay, encouragement is a good motivator.  I'll try to get to it.

One of my main questions when I initially thought about it is how to
handle the operator methods in c++.  When I define __getitem__, I have
to define it as a python function using def, so is it compiled
directly into c code?  To ask it another way, is there any way to
create a class A such that I can write

a = A(...)
a[i] += 1

and have the a[i] += 1 be entirely c(++) code?

--Hoyt

> Another option would be something like http://wiki.cython.org/
> enhancements/arraytypes, though this is probably further off in the
> future than wrapping several C++ containers.

Okay, I'll look into it.  Thanks.

-- Hoyt

P.S.  Robert -- I see you're at UW.  I'm starting a PhD in stats there
(Continue reading)

Stefan Behnel | 1 Sep 21:13 2008
Picon

Re: [Cython] Lists?

Hi,

Hoyt Koepke wrote:
> One of my main questions when I initially thought about it is how to
> handle the operator methods in c++.  When I define __getitem__, I have
> to define it as a python function using def, so is it compiled
> directly into c code? To ask it another way, is there any way to
> create a class A such that I can write
> 
> a = A(...)
> a[i] += 1
> 
> and have the a[i] += 1 be entirely c(++) code?

Yes, but within the limits of the Python getitem protocol. This means that the
item itself will be a Python object, and thus the "+= 1" will be executed on
two Python objects.

Stefan
Robert Bradshaw | 1 Sep 21:18 2008

Re: [Cython] Lists?

On Sep 1, 2008, at 11:56 AM, Hoyt Koepke wrote:

> Hi,
>
>> That would be excellent! It would necessitate using C++, which is not
>> going to be the default Cython output but would be very hand for
>> people wanting to use it.
>
> Okay, encouragement is a good motivator.  I'll try to get to it.
>
> One of my main questions when I initially thought about it is how to
> handle the operator methods in c++.  When I define __getitem__, I have
> to define it as a python function using def, so is it compiled
> directly into c code?

Yes. To clarify, the bodies of def methods are compiled into C code  
just as with cdef methods, and special methods are special, meaning  
the get called as C functions via the Python/C API, so they are not  
really "def" methods but nor are they really "cdef" methods.

> To ask it another way, is there any way to
> create a class A such that I can write
>
> a = A(...)
> a[i] += 1
>
> and have the a[i] += 1 be entirely c(++) code?

Almost. One can use the buffer interface for the class A, then use  
the "operator overloading" trick at the bottom of http:// 
(Continue reading)

Robert Bradshaw | 1 Sep 21:53 2008

Re: [Cython] Lists?

On Sep 1, 2008, at 12:13 PM, Stefan Behnel wrote:

> Hi,
>
> Hoyt Koepke wrote:
>> One of my main questions when I initially thought about it is how to
>> handle the operator methods in c++.  When I define __getitem__, I  
>> have
>> to define it as a python function using def, so is it compiled
>> directly into c code? To ask it another way, is there any way to
>> create a class A such that I can write
>>
>> a = A(...)
>> a[i] += 1
>>
>> and have the a[i] += 1 be entirely c(++) code?
>
> Yes, but within the limits of the Python getitem protocol. This  
> means that the
> item itself will be a Python object, and thus the "+= 1" will be  
> executed on
> two Python objects.

One *can* get around this with the buffer interface however.

- Robert


Gmane