Brett Calcott | 1 Nov 04:44 2010
Picon

[Cython] Wrapping C++ - questions and thoughts

Hi,


I'm wrapping a C++ library (http://www.box2d.org/). A python version done with swig already exists, but I'm trying to make it more pythonic/cythonic, and preparing it for a future project. It's all going well, and I'm having a fun time -- cython rocks.

I have a few questions and thoughts about wrapping. Consider wrapping something simple like this, where things are public:

class Vector2d {
public:
  float x, y
  Vector2d() : x(0.0), y(0.0) {}
  // other useful stuff here
};

I begin by doing this:
extern from "vector2d.h":
  cdef cppclass Vector2d:
     float x, y
     Vector2d()

Here are the questions:
1. Why must I use a *pointer* in my python class?

cdef class PythonVector2d:
  cdef Vector2d *obj

There is a default constructor, and I can happily declare it on the stack in a function. I'm sure I'm missing something here; is there some danger with allowing this?

2. How hard would it be to automatically forward member access to a contained cppclass? Consider the following nasty hack:

cdef class PythonVector2d:
  cdef public float x, y
  cdef Vector2d *obj
  def __cinit__(self, x, y):
    self.x = x
    self.y = y
    self.obj = <Vector 2d*>x

ok. That is a bit nasty (I'm sure someone can tell me lots of reasons why I shouldn't do this). But it really makes things easy. I don't have to declare all the properties independently. And when I need the cpp version, I just pass around pointer.

Something like this would be nice:

cdef class PythonVector2d:
  cdef Vector obj
  properties:
     public obj.x as x
     public obj.y as y

Does this look crazy?

Cheers,
Brett

    

<div>
<p>Hi,</p>
<div><br></div>
<div>I'm wrapping a C++ library (<a href="http://www.box2d.org/">http://www.box2d.org/</a>). A python version done with swig already exists, but I'm trying to make it more pythonic/cythonic, and preparing it for a future project. It's all going well, and I'm having a fun time -- cython rocks.</div>

<div><br></div>
<div>I have a few questions and thoughts about wrapping. Consider wrapping something simple like this, where things are public:</div>
<div><br></div>
<div>class Vector2d {</div>
<div>public:</div>
<div>&nbsp;&nbsp;float x, y</div>

<div>&nbsp;&nbsp;Vector2d() : x(0.0), y(0.0) {}</div>
<div>&nbsp;&nbsp;// other useful stuff here</div>
<div>};</div>
<div><br></div>
<div>I begin by doing this:</div>
<div>extern from "vector2d.h":</div>
<div>&nbsp;&nbsp;cdef cppclass Vector2d:</div>

<div>&nbsp;&nbsp; &nbsp; float x, y</div>
<div>&nbsp;&nbsp; &nbsp; Vector2d()</div>
<div><br></div>
<div>Here are the questions:</div>
<div>1. Why must I use a *pointer* in my python class?</div>
<div><br></div>
<div>cdef class PythonVector2d:</div>
<div>&nbsp;&nbsp;cdef Vector2d *obj</div>

<div><br></div>
<div>There is a default constructor, and I can happily declare it on the stack in a function. I'm sure I'm missing something here; is there some danger with allowing this?</div>
<div><br></div>
<div>2. How hard would it be to automatically forward member access to a contained cppclass? Consider the following nasty hack:</div>

<div><br></div>
<div>
<div>cdef class PythonVector2d:</div>
<div>&nbsp;&nbsp;cdef public float x, y</div>
<div>&nbsp;&nbsp;cdef Vector2d *obj</div>
</div>
<div>&nbsp;&nbsp;def __cinit__(self, x, y):</div>
<div>&nbsp;&nbsp; &nbsp;self.x = x</div>
<div>&nbsp;&nbsp; &nbsp;self.y = y</div>
<div>

&nbsp;&nbsp; &nbsp;self.obj = &lt;Vector 2d*&gt;x</div>
<div><br></div>
<div>ok. That is a bit nasty (I'm sure someone can tell me lots of reasons why I shouldn't do this). But it really makes things easy. I don't have to declare all the properties independently. And when I need the cpp version, I just pass around pointer.</div>

<div><br></div>
<div>Something like this would be nice:</div>
<div><br></div>
<div>cdef class PythonVector2d:</div>
<div>&nbsp;&nbsp;cdef Vector obj</div>
<div>&nbsp;&nbsp;properties:</div>
<div>&nbsp;&nbsp; &nbsp; public obj.x as x</div>
<div>&nbsp;&nbsp; &nbsp; public obj.y as y</div>

<div><br></div>
<div>Does this look crazy?</div>
<div><br></div>
<div>Cheers,</div>
<div>Brett</div>
<div><br></div>
<div>&nbsp;&nbsp; &nbsp;</div>
<div><br></div>
</div>
Vitja Makarov | 1 Nov 08:46 2010
Picon

[Cython] metaclasses

Hi, all!

I want to see metaclasses in cython. So I look into the code and I see
that classes are generated like this:

class Foo:
   xxx = 111

is transformed in something like:

Foo = Pyx_CreateClass('Foo', bases=(), attrs={})
# and then it does setattr()
Foo.xxx = 111

So nothing to do with metaclasses in Pyx_CreateClass() as attributes
are set after class is actually created ;(
Btw if class dict will be filled before class creation it's easy to
handle metaclass stuff in Pyx_CreateClass.
This this should be hard as PyDict_SetItem() and PyDict_GetItem()
should be used while class body creation

In python it works like this:

# based on Python/ceval.c:build_class()
def create_class(name, bases, attrs):
    if '__metaclass__' in attrs:
        metaclass = attrs.get('__metaclass__')
    elif len(bases) > 0:
        base = bases[0]
        if hasattr(base, '__class__'):
            metaclass = base.__class__
        else:
            metaclass = type(base)
    else '__metaclass__' in globals():
        metaclass = globals().get('__metaclass__')
    else:
        metaclass = type
    return metaclass(name, bases, attrs)

Another tricky way is to transform ClassDefNode, that has
__metaclass__ attribute into two classes:

class Foo(object):
   __metaclass__ = Bar
   xxx = 111

transform into:

Foo = create_class('Foo', (), {})
Foo.xxx = 111
Foo = create_class('Foo', (object,),  Foo.__dict__())

Second way is much easy to implement (I think) adding
MetaclassTransform into pipeline, but seems to be dirty hack.

I want to ask what is the best way to implement this?

vitja.
Stefan Behnel | 1 Nov 11:38 2010
Picon

Re: [Cython] metaclasses

Vitja Makarov, 01.11.2010 08:46:
> I want to see metaclasses in cython.

+1, could you open an enhancement ticket in the bug tracker? I'm surprised 
there isn't one yet.

> So I look into the code and I see that classes are generated like this:
>
> class Foo:
>     xxx = 111
>
> is transformed in something like:
>
> Foo = Pyx_CreateClass('Foo', bases=(), attrs={})
> # and then it does setattr()
> Foo.xxx = 111
>
> So nothing to do with metaclasses in Pyx_CreateClass() as attributes
> are set after class is actually created ;(
> Btw if class dict will be filled before class creation it's easy to
> handle metaclass stuff in Pyx_CreateClass.

Your description seems to indicate that you want this only for Python 
classes, which is fine. I think building the dict before building the class 
is the right way to implement this, as the metaclass is allowed to do 
whatever it likes with its input, including changes to the class dict. So 
it really needs to know the complete dict.

Also see this post:

http://thread.gmane.org/gmane.comp.python.cython.devel/10388

I don't know if the method binding problem has gone away, but it's worth a 
try. If it hasn't, it's worth fixing the way methods work, IMHO, but that 
may not be low hanging fruit.

The post is from this originating thread, where Ryan Kelly proposed ways to 
implement metaclasses:

http://thread.gmane.org/gmane.comp.python.cython.user/1652

I don't think metaclasses for cdef classes are ready to be implemented yet, 
but it should be easier for Python classes.

Also note that there is a new syntax for metaclasses in Py3, which I prefer 
over the Py2 way. So we will eventually have to support both, although the 
Py2 way is easy to emulate for Py3 code.

Stefan
Vitja Makarov | 1 Nov 12:58 2010
Picon

Re: [Cython] metaclasses

2010/11/1 Stefan Behnel <stefan_ml@...>:
> Vitja Makarov, 01.11.2010 08:46:
>> I want to see metaclasses in cython.
>
> +1, could you open an enhancement ticket in the bug tracker? I'm surprised
> there isn't one yet.
>
>

It seems that I don't have write access to trac.

>> So I look into the code and I see that classes are generated like this:
>>
>> class Foo:
>>     xxx = 111
>>
>> is transformed in something like:
>>
>> Foo = Pyx_CreateClass('Foo', bases=(), attrs={})
>> # and then it does setattr()
>> Foo.xxx = 111
>>
>> So nothing to do with metaclasses in Pyx_CreateClass() as attributes
>> are set after class is actually created ;(
>> Btw if class dict will be filled before class creation it's easy to
>> handle metaclass stuff in Pyx_CreateClass.
>
> Your description seems to indicate that you want this only for Python
> classes, which is fine. I think building the dict before building the class
> is the right way to implement this, as the metaclass is allowed to do
> whatever it likes with its input, including changes to the class dict. So
> it really needs to know the complete dict.
>
> Also see this post:
>
> http://thread.gmane.org/gmane.comp.python.cython.devel/10388
>
> I don't know if the method binding problem has gone away, but it's worth a
> try. If it hasn't, it's worth fixing the way methods work, IMHO, but that
> may not be low hanging fruit.
>
> The post is from this originating thread, where Ryan Kelly proposed ways to
> implement metaclasses:
>
> http://thread.gmane.org/gmane.comp.python.cython.user/1652
>
> I don't think metaclasses for cdef classes are ready to be implemented yet,
> but it should be easier for Python classes.
>
> Also note that there is a new syntax for metaclasses in Py3, which I prefer
> over the Py2 way. So we will eventually have to support both, although the
> Py2 way is easy to emulate for Py3 code.
>

This should be done in _Pyx_CreateClass() wrapper?

> Stefan

Yes, I'm mostly interested in metaclasses for "pure" python.
As I understand current PyClassDef implementation is mostly based on Pyrex one.

So is there a way to turn (set|get)attr into (set|get)item to write
dict instead of class object?

vitja.
Stefan Behnel | 1 Nov 17:53 2010
Picon

Re: [Cython] metaclasses

Vitja Makarov, 01.11.2010 12:58:
> 2010/11/1 Stefan Behnel:
>> Vitja Makarov, 01.11.2010 08:46:
>>> I want to see metaclasses in cython.
>> +1, could you open an enhancement ticket in the bug tracker? I'm surprised
>> there isn't one yet.
>>
> It seems that I don't have write access to trac.

It's restricted to authorised users to prevent spam. Please send a htpasswd 
line and a user name to Robert Bradshaw.

>>> So I look into the code and I see that classes are generated like this:
>>>
>>> class Foo:
>>>      xxx = 111
>>>
>>> is transformed in something like:
>>>
>>> Foo = Pyx_CreateClass('Foo', bases=(), attrs={})
>>> # and then it does setattr()
>>> Foo.xxx = 111
>>>
>>> So nothing to do with metaclasses in Pyx_CreateClass() as attributes
>>> are set after class is actually created ;(
>>> Btw if class dict will be filled before class creation it's easy to
>>> handle metaclass stuff in Pyx_CreateClass.
>>
>> Your description seems to indicate that you want this only for Python
>> classes, which is fine.
>
> Yes, I'm mostly interested in metaclasses for "pure" python. As I
> understand current PyClassDef implementation is mostly based on Pyrex
> one.

Correct.

>> I think building the dict before building the class
>> is the right way to implement this, as the metaclass is allowed to do
>> whatever it likes with its input, including changes to the class dict. So
>> it really needs to know the complete dict.
>>
>> Also note that there is a new syntax for metaclasses in Py3, which I prefer
>> over the Py2 way. So we will eventually have to support both, although the
>> Py2 way is easy to emulate for Py3 code.
>
> This should be done in _Pyx_CreateClass() wrapper?

Partly, I think. If the metaclass is defined outside of the class, e.g. 
because it was passed as a keyword argument to the class, or because there 
is a globally defined name __metaclass__, I think it can safely be passed 
as an argument to that function from the calling code. The rest, i.e. 
checking for a locally defined name in the class dict, can be done inside 
of __Pyx_CreateClass(). I don't know how Py3 handles duplicate metaclass 
definitions through keyword *and* __metaclass__, but Cython should do it in 
the same way.

> So is there a way to turn (set|get)attr into (set|get)item to write
> dict instead of class object?

The class body (see PyClassDefNode in Nodes.py) is evaluated in its own 
scope (PyClassScope in Symtab.py). It might (!) work to build a DictNode at 
the end of the class body that contains all names defined in the scope with 
their current value, so that it can be passed into __Pyx_CreateClass(). Not 
sure how to handle "del" here, but I doubt that that currently works anyway.

Does that help? Please ask back on this list when you have any questions.

Stefan
killian koepsell | 1 Nov 21:14 2010
Picon

[Cython] Fwd (from cython-user): pointers to numpy ndarray / conversion of numpy array to some type C++ array

Hi,

I am not sure if my question is appropriate for the cython-dev list.
I thought, this might be a pretty common problem and I am curious
to hear an opinion of the cython developers.

Thanks,
  Kilian

---------- Forwarded message ----------
From: killian koepsell <koepsell@...>
Date: Sat, Oct 30, 2010 at 3:14 PM
Subject: pointers to numpy ndarray / conversion of numpy array to some
type C++ array
To: cython-users@...

Hi,

I would like to access a large number of numpy ndarrays (of different
size) from cython inside some inner loop. In python I would keep these
arrays inside an object attribute, such as a dictionary or a list of
arrays. However, as I understand it, I would always incure a
significant overhead if I access a python list from within cython
(since the list members are not typed). What is the best way to attach
a number of ndarrays to a python object in a way that they are fast
accessible inside cython without going through the python layer?

I was thinking of keeping a list of pointers in a C++ vector, however
I did not manage to obtain a pointer to a buffer of type e.g.
np.ndarray[y[np.double_t, dim=2]. If I only keep a pointer to the
ndarray.data, I loose the shape and stride information. The workaround
I came up with is to create an auxiliary struct or cppclass that
contains all the required information to access a numpy array (data
pointer, shape, strides) and to keep pointers to these auxiliary
object in a C++ vector. This works fast but seems somewhat cumbersome.
In particular, I lose all the nice cython buffer indexing features and
now have to use pointer arithmetics to access the data. I am wondering
if there is a better solution. Maybe, it is possible to obtain a
pointer to the buffer objects that cython uses internally?

Or alternatively, maybe it is possible to create some type of C++
ndarrays without copying of the data and to store pointers to these
C++ arrays inside a C++ vector? Blitz, opencv, and boost come to mind,
but I haven't used either from inside cython. What would be the
best/easiest way to do that?

Thanks for any suggestions.

Kilian
Robert Bradshaw | 1 Nov 21:14 2010

Re: [Cython] metaclasses

On Mon, Nov 1, 2010 at 3:38 AM, Stefan Behnel <stefan_ml@...> wrote:
> Vitja Makarov, 01.11.2010 08:46:
>> I want to see metaclasses in cython.
>
> +1, could you open an enhancement ticket in the bug tracker? I'm surprised
> there isn't one yet.

+1 from me too, this'd be great to see. It's probably the largest
missing piece after generators.

>> So I look into the code and I see that classes are generated like this:
>>
>> class Foo:
>>     xxx = 111
>>
>> is transformed in something like:
>>
>> Foo = Pyx_CreateClass('Foo', bases=(), attrs={})
>> # and then it does setattr()
>> Foo.xxx = 111
>>
>> So nothing to do with metaclasses in Pyx_CreateClass() as attributes
>> are set after class is actually created ;(
>> Btw if class dict will be filled before class creation it's easy to
>> handle metaclass stuff in Pyx_CreateClass.
>
> Your description seems to indicate that you want this only for Python
> classes, which is fine. I think building the dict before building the class
> is the right way to implement this, as the metaclass is allowed to do
> whatever it likes with its input, including changes to the class dict. So
> it really needs to know the complete dict.

I agree, we should create the dict first. It'll take some work, but
don't think this'll be that big of a change.

> Also see this post:
>
> http://thread.gmane.org/gmane.comp.python.cython.devel/10388
>
> I don't know if the method binding problem has gone away, but it's worth a
> try. If it hasn't, it's worth fixing the way methods work, IMHO, but that
> may not be low hanging fruit.

It has been fixed, in the sense that it has been implemented. I'm it's
enabled by default yet, but that would be a question of just flipping
a (context-triggered for now) switch.
http://trac.cython.org/cython_trac/ticket/494

> The post is from this originating thread, where Ryan Kelly proposed ways to
> implement metaclasses:
>
> http://thread.gmane.org/gmane.comp.python.cython.user/1652
>
> I don't think metaclasses for cdef classes are ready to be implemented yet,
> but it should be easier for Python classes.
>
> Also note that there is a new syntax for metaclasses in Py3, which I prefer
> over the Py2 way. So we will eventually have to support both, although the
> Py2 way is easy to emulate for Py3 code.
>
> Stefan
> _______________________________________________
> Cython-dev mailing list
> Cython-dev@...
> http://codespeak.net/mailman/listinfo/cython-dev
>
Robert Bradshaw | 1 Nov 21:44 2010

Re: [Cython] auto __test__ dict

On Sun, Oct 31, 2010 at 11:38 AM, Stefan Behnel <stefan_ml@...> wrote:
> Stefan Behnel, 31.10.2010 08:27:
>> Stefan Behnel, 30.10.2010 21:16:
>>> can now write something like this for a type that you only use internally
>>> in your module:
>>>
>>>        <at> cython.final
>>>        <at> cython.internal
>>>       cdef class _MyInternalType:
>>>          ...
>>
>> One problem I spotted was the auto test dict (again). It takes the classes
>> from the module dict if they have a docstring, which now fails for internal
>> types (it would also fail for 'del'-ed types, BTW). I always wondered why
>> it used the module dict anyway - I guess it's just simplicity. Any reason
>> not to rewrite that?
>
> I simplified it now to use the compile time docstrings directly, instead of
> looking them up at runtime. I don't think this breaks anything important,
> as the docstrings of functions and methods cannot be modified anyway. On
> the plus side, it removes a *ton* of ugly C code, and also saves some time
> during module initialisation.

Excellent! I've been wanting to do this myself, eventually when I
found the time. This means that cdef methods could have tested
docstrings as well (though, of course, they couldn't be called
directly).

- Robert
Robert Bradshaw | 1 Nov 21:46 2010

Re: [Cython] Fwd (from cython-user): pointers to numpy ndarray / conversion of numpy array to some type C++ array

Cython-users is the appropriate list for this kind of thing (and a
near super-set of this list, so no need to double-post). Incidentally,
I just gave some comments on that thread over there. Definitely report
back when you get something working.

On Mon, Nov 1, 2010 at 1:14 PM, killian koepsell <koepsell@...> wrote:
> Hi,
>
> I am not sure if my question is appropriate for the cython-dev list.
> I thought, this might be a pretty common problem and I am curious
> to hear an opinion of the cython developers.
>
> Thanks,
>  Kilian
>
>
> ---------- Forwarded message ----------
> From: killian koepsell <koepsell@...>
> Date: Sat, Oct 30, 2010 at 3:14 PM
> Subject: pointers to numpy ndarray / conversion of numpy array to some
> type C++ array
> To: cython-users@...
>
>
> Hi,
>
> I would like to access a large number of numpy ndarrays (of different
> size) from cython inside some inner loop. In python I would keep these
> arrays inside an object attribute, such as a dictionary or a list of
> arrays. However, as I understand it, I would always incure a
> significant overhead if I access a python list from within cython
> (since the list members are not typed). What is the best way to attach
> a number of ndarrays to a python object in a way that they are fast
> accessible inside cython without going through the python layer?
>
> I was thinking of keeping a list of pointers in a C++ vector, however
> I did not manage to obtain a pointer to a buffer of type e.g.
> np.ndarray[y[np.double_t, dim=2]. If I only keep a pointer to the
> ndarray.data, I loose the shape and stride information. The workaround
> I came up with is to create an auxiliary struct or cppclass that
> contains all the required information to access a numpy array (data
> pointer, shape, strides) and to keep pointers to these auxiliary
> object in a C++ vector. This works fast but seems somewhat cumbersome.
> In particular, I lose all the nice cython buffer indexing features and
> now have to use pointer arithmetics to access the data. I am wondering
> if there is a better solution. Maybe, it is possible to obtain a
> pointer to the buffer objects that cython uses internally?
>
> Or alternatively, maybe it is possible to create some type of C++
> ndarrays without copying of the data and to store pointers to these
> C++ arrays inside a C++ vector? Blitz, opencv, and boost come to mind,
> but I haven't used either from inside cython. What would be the
> best/easiest way to do that?
>
> Thanks for any suggestions.
>
> Kilian
> _______________________________________________
> Cython-dev mailing list
> Cython-dev@...
> http://codespeak.net/mailman/listinfo/cython-dev
>
Vitja Makarov | 1 Nov 21:53 2010
Picon

Re: [Cython] metaclasses

Created ticket
http://trac.cython.org/cython_trac/ticket/587

2010/11/1 Robert Bradshaw <robertwb@...>:
> On Mon, Nov 1, 2010 at 3:38 AM, Stefan Behnel <stefan_ml@...> wrote:
>> Vitja Makarov, 01.11.2010 08:46:
>>> I want to see metaclasses in cython.
>>
>> +1, could you open an enhancement ticket in the bug tracker? I'm surprised
>> there isn't one yet.
>
> +1 from me too, this'd be great to see. It's probably the largest
> missing piece after generators.
>
>>> So I look into the code and I see that classes are generated like this:
>>>
>>> class Foo:
>>>     xxx = 111
>>>
>>> is transformed in something like:
>>>
>>> Foo = Pyx_CreateClass('Foo', bases=(), attrs={})
>>> # and then it does setattr()
>>> Foo.xxx = 111
>>>
>>> So nothing to do with metaclasses in Pyx_CreateClass() as attributes
>>> are set after class is actually created ;(
>>> Btw if class dict will be filled before class creation it's easy to
>>> handle metaclass stuff in Pyx_CreateClass.
>>
>> Your description seems to indicate that you want this only for Python
>> classes, which is fine. I think building the dict before building the class
>> is the right way to implement this, as the metaclass is allowed to do
>> whatever it likes with its input, including changes to the class dict. So
>> it really needs to know the complete dict.
>
> I agree, we should create the dict first. It'll take some work, but
> don't think this'll be that big of a change.
>
>> Also see this post:
>>
>> http://thread.gmane.org/gmane.comp.python.cython.devel/10388
>>
>> I don't know if the method binding problem has gone away, but it's worth a
>> try. If it hasn't, it's worth fixing the way methods work, IMHO, but that
>> may not be low hanging fruit.
>
> It has been fixed, in the sense that it has been implemented. I'm it's
> enabled by default yet, but that would be a question of just flipping
> a (context-triggered for now) switch.
> http://trac.cython.org/cython_trac/ticket/494
>
>> The post is from this originating thread, where Ryan Kelly proposed ways to
>> implement metaclasses:
>>
>> http://thread.gmane.org/gmane.comp.python.cython.user/1652
>>
>> I don't think metaclasses for cdef classes are ready to be implemented yet,
>> but it should be easier for Python classes.
>>
>> Also note that there is a new syntax for metaclasses in Py3, which I prefer
>> over the Py2 way. So we will eventually have to support both, although the
>> Py2 way is easy to emulate for Py3 code.
>>
>> Stefan
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev@...
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev@...
> http://codespeak.net/mailman/listinfo/cython-dev
>

Gmane