Ian Henriksen | 4 Jul 00:43 2015
Picon
Gravatar

[Cython] Multidimensional indexing of C++ objects

Hi everyone,
I'm a GSOC student working to make a Cython API for DyND. DyND is a relatively new n-dimensional array library in C++ that is based on NumPy. A full set of Python bindings (created using Cython) are provided as a separate package. The goal of my project is to make it so that DyND arrays can be used easily within Cython so that an n-dimensional array object can be used without any of the corresponding Python overhead.

Currently, there isn't a good way to assign to multidimensional slices within Cython. Since the indexing operator in C++ is limited to a single argument, we use the call operator to represent multidimensional indexing, and then use a proxy class to perform assignment to a slice.
Currently, in C++, assigning to a slice along the second axis of a DyND array looks like this:

a(irange(), 1).vals() = 0;

Unfortunately, in Cython, only the index operator can be used for assignment, so following the C++ syntax isn't currently possible. Does anyone know of a good way to address this? I'm willing to spend some time implementing a new feature if we can reach a consensus on a good way to deal with this. Here are some possible solutions I've thought of:

1. We could allow assignment to C++ method and function calls that return references. This has the advantage that it matches the existing syntax in C++ for dealing with C++ objects. Though Cython is a Python-like language, the ability to manipulate C++ objects directly is a key part of its feature set. Since the native way to do things like multidimensional indexing in C++ is via the call operator, it seems sensible to allow assignment to C++-level call operations in Cython as well. This could be enabled via a Cython compiler directive and be disabled by default. Using a compiler directive like this would result in an interface similar to the one already used for cdivision, wrap-around indexing, and index bounds checking. The user would avoid unexpected results by default, but be able to get the needed functionality simply by enabling it.

2. We could recommend that all assignment operations of this nature be wrapped in a fake method that wraps the assignment in it's c-level name. This has the advantage that it works in current and past versions of Cython, but it is a rather unusual hack. For example, something like the following would work right now:

# declared as a method in a pxd file:
void assign "vals() = "(int value) except +

# used in a pyx file to perform assignment to a slice of an array a:
a(irange(), 1).assign(0)

For DyND, at least for now, this would be a workable solution since the difference lies primarily in the placement of the parenthesis and the presence of the assignment operator. The syntax is less clear than it could be, but it would work. On the other hand, other libraries may not be so lucky since this involves replacing assignment to a slice with a method call. For example, the expression template libraries Eigen and Blaze-lib would encounter incompatibility to varying degrees if someone were to try using them within Cython. This method also has the disadvantage that it creates an interface that is fundamentally different from both the Python and C++ interfaces.

I have also considered, writing a proxy class that can serve as an effective temporary value while a multidimensional index is constructed from a series of calls to operator[]. This is a reasonable approach, but it leads to unnecessary code bloat. It also complicates the interface exposed to users, since operator[] would be needed for left hand values and operator() would be needed for right hand values. This would also make it so that users that want to use these C++ classes in Cython would have to include and link against another set of headers and libraries to be able to use the proxy class. The burden of maintainability for Python bindings created in this way would be greater as well. This also isn't a viable approach for using any C++ class that overloads both operators.

Another option I have considered is allowing Cython's indexing operator to dispatch to a different function. Currently, user-defined cname entries for overloaded operators are not used. If this were changed for the indexing operator, indexing could be performed at the C++ level using some other method. This doesn't look like a viable approach though, since, for this to really work, users would need some way to call different methods when a C++ object is being indexed and when it is being assigned to. Using operator[] for left-hand values and operator() for right-hand values is a possible solution, but that isn't a very consistent interface. Doing this would also increase the complexity of the existing code for indexing in the Cython compiler and could lead to name collisions for classes that overload both operator[] and operator().

Are any of these acceptable ways to go forward? Does anyone have any better ideas? My preference would definitely be toward allowing C++ calls returning references to be used as lvalues, but I'd really appreciate any alternative solutions.

Thanks!
-Ian Henriksen


<div><div dir="ltr">Hi everyone,<div>I'm a GSOC student working to make a Cython API for DyND.&nbsp;<a href="https://github.com/libdynd/libdynd" target="_blank">DyND</a>&nbsp;is a relatively new n-dimensional array library in C++ that is based on NumPy. A full set of Python bindings (created using Cython) are provided as a separate package. The goal of my project is to make it so that DyND arrays can be used easily within Cython so that an n-dimensional array object can be used without any of the corresponding Python overhead.</div>
<div><br></div>
<div>Currently, there isn't a good way to assign to multidimensional slices within Cython. Since the indexing operator in C++ is limited to a single argument, we use the call operator to represent multidimensional indexing, and then use a proxy class to perform assignment to a slice.</div>
<div>Currently, in C++, assigning to a slice along the second axis of a DyND array looks like this:</div>
<div><br></div>
<div>
<div>a(irange(), 1).vals() = 0;</div>
<div><br></div>
<div>Unfortunately, in Cython, only the index operator can be used for assignment, so following the C++ syntax isn't currently possible. Does anyone know of a good way to address this? I'm willing to spend some time implementing a new feature if we can reach a consensus on a good way to deal with this. Here are some possible solutions I've thought of:</div>
<div><br></div>
<div>1. We could allow assignment to C++ method and function calls that return references. This has the advantage that it matches the existing syntax in C++ for dealing with C++ objects. Though Cython is a Python-like language, the ability to manipulate C++ objects directly is a key part of its feature set. Since the native way to do things like multidimensional indexing in C++ is via the call operator, it seems sensible to allow assignment to C++-level call operations in Cython as well. This could be enabled via a Cython compiler directive and be disabled by default. Using a compiler directive like this would result in an interface similar to the one already used for cdivision, wrap-around indexing, and index bounds checking. The user would avoid unexpected results by default, but be able to get the needed functionality simply by enabling it.</div>
<div><br></div>
<div>2. We could recommend that all assignment operations of this nature be wrapped in a fake method that wraps the assignment in it's c-level name. This has the advantage that it works in current and past versions of Cython, but it is a rather unusual hack. For example, something like the following would work right now:</div>
<div><br></div>
<div># declared as a method in a pxd file:</div>
<div>
<span>void assign "vals() = "(int value) except&nbsp;+</span><br>
</div>
<div><span><br></span></div>
<div><span># used in a pyx file to perform assignment to a slice of an array a:</span></div>
<div><span>a(irange(), 1).assign(0)</span></div>
<div><span><br></span></div>
<div><span>For DyND, at least for now, this would be a workable solution since the difference lies primarily in the placement of the parenthesis and the presence of the assignment operator. The syntax is less clear than it could be, but it would work. On the other hand, other libraries may not be so lucky since this involves replacing assignment to a slice with a method call. For example, the expression template libraries Eigen and Blaze-lib would encounter incompatibility to varying degrees if someone were to try using them within Cython. This method also has the disadvantage that it creates an interface that is fundamentally different from both the Python and C++ interfaces.</span></div>
<div><span><br></span></div>
<div>
<span>I have also considered, writing a proxy class that can serve as an effective temporary value while a multidimensional index is constructed from a series of calls to operator[]. This is a reasonable approach, but it leads to unnecessary code bloat. It also complicates the interface exposed to users, since</span>&nbsp;operator[] would be needed for left hand values and operator() would be needed for right hand values. This would also make it so that users that want to use these C++ classes in Cython would have to include and link against another set of headers and libraries to be able to use the proxy class.<span>&nbsp;The burden of maintainability for Python bindings created in this way would be greater as well. This also isn't a viable approach for using any C++ class that overloads both operators.</span>
</div>
<div><span><br></span></div>
<div>Another option I have considered is allowing Cython's indexing operator to dispatch to a different function. Currently, user-defined cname entries for overloaded operators are not used. If this were changed for the indexing operator, indexing could be performed at the C++ level using some other method. This doesn't look like a viable approach though, since, for this to really work, users would need some way to call different methods when a C++ object is being indexed and when it is being assigned to. Using operator[] for left-hand values and operator() for right-hand values is a possible solution, but that isn't a very consistent interface. Doing this would also increase the complexity of the existing code for indexing in the Cython compiler and could lead to name collisions for classes that overload both operator[] and operator().<span><br></span>
</div>
<div><span><br></span></div>
<div><span>Are any of these acceptable ways to go forward? Does anyone have any better ideas? My preference would definitely be toward allowing C++ calls returning references to be used as lvalues, but I'd really appreciate any alternative solutions.</span></div>
<div><span><br></span></div>
<div><span>Thanks!</span></div>
<div><span>-Ian Henriksen</span></div>
<div><br></div>
<div><br></div>
</div>
</div></div>
Stephen LARROQUE | 3 Jul 03:16 2015
Picon

Re: [Cython] cython-devel Digest, Vol 53, Issue 9

Thank you very much Stefan.

I have just read the ticket, and it seems that Guido van Rossum consider this to be a feature, not a bug (???), so I guess they won't fix it.

For me it would be a nice plus to have this problem fixed, but I can live with my current workaround, so I'm not sure if Cython should spend dev time to fix that if that's expensive.

At least, I think the workaround should be mentioned in the github wiki so that others may implement finite fields extension types without MemoryError issues. Should I add a wiki page?

2015-06-19 12:00 GMT+02:00 <cython-devel-request-+ZN9ApsXKcEdnm+yROfE0A@public.gmane.org>:
Send cython-devel mailing list submissions to
        cython-devel-+ZN9ApsXKcEdnm+yROfE0A@public.gmane.org

To subscribe or unsubscribe via the World Wide Web, visit
        https://mail.python.org/mailman/listinfo/cython-devel
or, via email, send a message with subject or body 'help' to
        cython-devel-request-+ZN9ApsXKcEdnm+yROfE0A@public.gmane.org

You can reach the person managing the list at
        cython-devel-owner-+ZN9ApsXKcEdnm+yROfE0A@public.gmane.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of cython-devel digest..."


Today's Topics:

   1. Re: Bug: Extension Type inheriting from int cause a
      MemoryError (Stefan Behnel)


----------------------------------------------------------------------

Message: 1
Date: Fri, 19 Jun 2015 08:38:32 +0200
From: Stefan Behnel <stefan_ml <at> behnel.de>
To: cython-devel-+ZN9ApsXKcEdnm+yROfE0A@public.gmane.org
Subject: Re: [Cython] Bug: Extension Type inheriting from int cause a
        MemoryError
Message-ID: <5583B8E8.9050902-KjMAwuNBv5izQB+pC5nmwQ@public.gmane.org>
Content-Type: text/plain; charset=utf-8

Stephen LARROQUE schrieb am 15.06.2015 um 12:34:
> I am trying to make an extension type inheriting from int or cython.int (to
> do arithmetic operations in Galois Fields). However, this causes a
> MemoryError because it seems such extension type is not freed correctly.
> Other than that, it works perfectly well.
>
> cdef class ExtendedInt(int): pass
>
> for j in xrange(10000000):
>     ExtendedInt(j)

It looks like a bug in the "int" type in Python 2.x. Python 2.7 has this
code in intobject.c:

"""
static void
int_dealloc(PyIntObject *v)
{
    if (PyInt_CheckExact(v)) {
        Py_TYPE(v) = (struct _typeobject *)free_list;
        free_list = v;
    }
    else
        Py_TYPE(v)->tp_free((PyObject *)v);
}

static void
int_free(PyIntObject *v)
{
    Py_TYPE(v) = (struct _typeobject *)free_list;
    free_list = v;
}
"""

Your extension type automatically inherits the "int_free" slot function
from its base type, so the "else" case in "int_dealloc()" will in fact call
"int_free()" and append the object to the free list despite *not* being
exactly of type PyInt. Then, when creating a new ExtendedInt instance,
Python's int-subtype instantiation code *ignores* the free list and instead
creates a completely new object.

Thus, the free list keeps growing until it fills all memory and the process
dies. I created a CPython ticket.

https://bugs.python.org/issue24469

I guess we could hack up a work around for this in Cython somehow (Python
2.7 is an easy target being a dead end, after all), but let's wait what the
CPython devs have to say about this. Note, however, that any fix in a
future CPython 2.7.x release will not fix it in earlier Python 2.x
versions, so my guess is that we'd end up having to add a work-around on
our side anyway.

Stefan



------------------------------

Subject: Digest Footer

_______________________________________________
cython-devel mailing list
cython-devel-+ZN9ApsXKcEdnm+yROfE0A@public.gmane.org
https://mail.python.org/mailman/listinfo/cython-devel


------------------------------

End of cython-devel Digest, Vol 53, Issue 9
*******************************************

<div>
<div dir="ltr">Thank you very much Stefan.<div><br></div>
<div>I have just read the ticket, and it seems that Guido van Rossum consider this to be a feature, not a bug (???), so I guess they won't fix it.</div>
<div><br></div>
<div>For me it would be a nice plus to have this problem fixed, but I can live with my current workaround, so I'm not sure if Cython should spend dev time to fix that if that's expensive.</div>
<div><br></div>
<div>At least, I think the workaround should be mentioned in the github wiki so that others may implement finite fields extension types without MemoryError issues. Should I add a wiki page?</div>
</div>
<div class="gmail_extra">
<br><div class="gmail_quote">2015-06-19 12:00 GMT+02:00  <span dir="ltr">&lt;<a href="mailto:cython-devel-request@..." target="_blank">cython-devel-request@...</a>&gt;</span>:<br><blockquote class="gmail_quote">Send cython-devel mailing list submissions to<br>
&nbsp; &nbsp; &nbsp; &nbsp; <a href="mailto:cython-devel@...">cython-devel@...</a><br><br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
&nbsp; &nbsp; &nbsp; &nbsp; <a href="https://mail.python.org/mailman/listinfo/cython-devel" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/cython-devel</a><br>
or, via email, send a message with subject or body 'help' to<br>
&nbsp; &nbsp; &nbsp; &nbsp; <a href="mailto:cython-devel-request <at> python.org">cython-devel-request@...</a><br><br>
You can reach the person managing the list at<br>
&nbsp; &nbsp; &nbsp; &nbsp; <a href="mailto:cython-devel-owner@...">cython-devel-owner@...</a><br><br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of cython-devel digest..."<br><br><br>
Today's Topics:<br><br>
&nbsp; &nbsp;1. Re: Bug: Extension Type inheriting from int cause a<br>
&nbsp; &nbsp; &nbsp; MemoryError (Stefan Behnel)<br><br><br>
----------------------------------------------------------------------<br><br>
Message: 1<br>
Date: Fri, 19 Jun 2015 08:38:32 +0200<br>
From: Stefan Behnel &lt;<a href="mailto:stefan_ml@...">stefan_ml <at> behnel.de</a>&gt;<br>
To: <a href="mailto:cython-devel@...">cython-devel@...</a><br>
Subject: Re: [Cython] Bug: Extension Type inheriting from int cause a<br>
&nbsp; &nbsp; &nbsp; &nbsp; MemoryError<br>
Message-ID: &lt;<a href="mailto:5583B8E8.9050902@...">5583B8E8.9050902@...</a>&gt;<br>
Content-Type: text/plain; charset=utf-8<br><br>
Stephen LARROQUE schrieb am 15.06.2015 um 12:34:<br>
&gt; I am trying to make an extension type inheriting from int or <a href="http://cython.int" rel="noreferrer" target="_blank">cython.int</a> (to<br>
&gt; do arithmetic operations in Galois Fields). However, this causes a<br>
&gt; MemoryError because it seems such extension type is not freed correctly.<br>
&gt; Other than that, it works perfectly well.<br>
&gt;<br>
&gt; cdef class ExtendedInt(int): pass<br>
&gt;<br>
&gt; for j in xrange(10000000):<br>
&gt;&nbsp; &nbsp; &nbsp;ExtendedInt(j)<br><br>
It looks like a bug in the "int" type in Python 2.x. Python 2.7 has this<br>
code in intobject.c:<br><br>
"""<br>
static void<br>
int_dealloc(PyIntObject *v)<br>
{<br>
&nbsp; &nbsp; if (PyInt_CheckExact(v)) {<br>
&nbsp; &nbsp; &nbsp; &nbsp; Py_TYPE(v) = (struct _typeobject *)free_list;<br>
&nbsp; &nbsp; &nbsp; &nbsp; free_list = v;<br>
&nbsp; &nbsp; }<br>
&nbsp; &nbsp; else<br>
&nbsp; &nbsp; &nbsp; &nbsp; Py_TYPE(v)-&gt;tp_free((PyObject *)v);<br>
}<br><br>
static void<br>
int_free(PyIntObject *v)<br>
{<br>
&nbsp; &nbsp; Py_TYPE(v) = (struct _typeobject *)free_list;<br>
&nbsp; &nbsp; free_list = v;<br>
}<br>
"""<br><br>
Your extension type automatically inherits the "int_free" slot function<br>
from its base type, so the "else" case in "int_dealloc()" will in fact call<br>
"int_free()" and append the object to the free list despite *not* being<br>
exactly of type PyInt. Then, when creating a new ExtendedInt instance,<br>
Python's int-subtype instantiation code *ignores* the free list and instead<br>
creates a completely new object.<br><br>
Thus, the free list keeps growing until it fills all memory and the process<br>
dies. I created a CPython ticket.<br><br><a href="https://bugs.python.org/issue24469" rel="noreferrer" target="_blank">https://bugs.python.org/issue24469</a><br><br>
I guess we could hack up a work around for this in Cython somehow (Python<br>
2.7 is an easy target being a dead end, after all), but let's wait what the<br>
CPython devs have to say about this. Note, however, that any fix in a<br>
future CPython 2.7.x release will not fix it in earlier Python 2.x<br>
versions, so my guess is that we'd end up having to add a work-around on<br>
our side anyway.<br><br>
Stefan<br><br><br><br>
------------------------------<br><br>
Subject: Digest Footer<br><br>
_______________________________________________<br>
cython-devel mailing list<br><a href="mailto:cython-devel@...">cython-devel@...</a><br><a href="https://mail.python.org/mailman/listinfo/cython-devel" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/cython-devel</a><br><br><br>
------------------------------<br><br>
End of cython-devel Digest, Vol 53, Issue 9<br>
*******************************************<br>
</blockquote>
</div>
<br>
</div>
</div>
Robert McGibbon | 1 Jul 11:12 2015
Picon
Gravatar

[Cython] Use of long type for intermediate integral variables

Hi,

(First time poster. Apologies in advance if I'm not following the appropriate protocol, or if this has been posted already)

I noticed an issue on Windows when debugging an issue in scipy, but I think it might be a little more general.  In some places in the generated code, it looks like intermediate integral variables are declared with type long, even when long is too small to hold necessary value. For example, with the code pasted below, the value n+1 is stored in a variable of type long (using Cython 0.22.1) before being supplied to F.__getitem__.

This is especially pertinent on Windows (32 bit and 64 bit) and 32-bit linux, where longs are 32-bits, so you get an overflow for a program like the example below. The result is that it prints 1 instead of the expected value, 2**53+1 = 9007199254740993. But this same issue comes up basically whenever you do arithmetic on an array index in 64-bit Windows, for indices larger than 2**31-1, since sizeof(long) << sizeof(void*).

```
from libc.stdint cimport int64_t

class F(object):
    def __getitem__(self, i):
        print(i)

cdef int64_t n = 2**53
f = F()
f[n+1]
```
<div><div dir="ltr">Hi,<div><br></div>
<div>(First time poster. Apologies in advance if I'm not following the appropriate protocol, or if this has been posted already)</div>
<div><br></div>
<div>I noticed an issue on Windows when debugging an issue in <a href="https://github.com/scipy/scipy/issues/4907">scipy</a>, but I think it might be a little more general.&nbsp; In some places in the generated code, it looks like intermediate integral variables are declared with type long, even when long is too small to hold necessary value. For example, with the code pasted below, the value n+1 is stored in a variable of type long (using Cython 0.22.1) before being supplied to F.__getitem__.</div>
<div><br></div>
<div>This is especially pertinent on Windows (32 bit and 64 bit) and 32-bit linux, where longs are 32-bits, so you get an overflow for a program like the example below. The result is that it prints 1 instead of the expected value, 2**53+1 = 9007199254740993. But this same issue comes up basically whenever you do arithmetic on an array index in 64-bit Windows, for indices larger than 2**31-1, since sizeof(long) &lt;&lt; sizeof(void*).</div>
<div><br></div>
<div>```</div>
<div>
<div>from libc.stdint cimport int64_t</div>
<div><br></div>
<div>class F(object):</div>
<div>&nbsp; &nbsp; def __getitem__(self, i):</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; print(i)</div>
<div><br></div>
<div>cdef int64_t n = 2**53</div>
<div>f = F()</div>
<div>f[n+1]</div>
</div>
<div>```</div>
</div></div>
Michael Enßlin | 22 Jun 04:04 2015

[Cython] Feature Request: Variadic Templates

Hi,

I'd like to see support for variadic template arguments, such as this:

test.h:

template<typename RetT, typename ... VARARGS>
RetT foo(VARARGS ...);

test.pyx:

cdef extern from "test.h":
    cdef RetT foo[RetT, ... VARARGS](... VARARGS)

def test():
    cdef int i = foo[int, float, int](1, 2.5, 3)

This would allow Cython's libcpp to easily support many of C++11's new
types, including std::tuple and std::function.

Support for the latter in particular would prove quite useful in passing
around function pointers between Cython and C++.

I have tried to implement this feature myself, but I'm entirely
unfamiliar with Cython's codebase, and all my attempts ended in
unsatisfactorily hacky, unstable code.
I believe that any experienced Cython developer on this list would be
able to properly implement this in a matter of a few hours, as it seems
like a rather minor feature.

The current workaround for the non-existance of this feature involves
typedefs for every possible number of arguments, like this:

template<typename ... Ts>
struct S {};

using S0 = S<>;

template<typename T0>
using S1 = S<T0>;

template<typename T0, typename T1>
using S2 = S<T0, T1>;

template<typename T0, typename T1, typename T2>
using S3 = S<T0, T1, T2>;

then exporting all of S0, S1, ... individually in the .pxd file.

This is inconvenient, but acceptable.

Now assume that S has a member function f,

	template<typename ... Us>
	void f(Ts..., Us ...);

due to the nature of C++, the same trick as above does not work, and the
user will be forced to clutter the C++ code with a function body for
every possible len(Us). Even worse, the .pxd file will now contain a
quadratic number of wrappers (one for every possible combination of
len(Ts), len(Us)), all of them extremely prone to mistakes.

Thanks for reading,

  ~ Michael

Hi,

I'd like to see support for variadic template arguments, such as this:

test.h:

template<typename RetT, typename ... VARARGS>
RetT foo(VARARGS ...);

test.pyx:

cdef extern from "test.h":
    cdef RetT foo[RetT, ... VARARGS](... VARARGS)

def test():
    cdef int i = foo[int, float, int](1, 2.5, 3)

This would allow Cython's libcpp to easily support many of C++11's new
types, including std::tuple and std::function.

Support for the latter in particular would prove quite useful in passing
around function pointers between Cython and C++.

I have tried to implement this feature myself, but I'm entirely
unfamiliar with Cython's codebase, and all my attempts ended in
unsatisfactorily hacky, unstable code.
I believe that any experienced Cython developer on this list would be
able to properly implement this in a matter of a few hours, as it seems
like a rather minor feature.

The current workaround for the non-existance of this feature involves
typedefs for every possible number of arguments, like this:

template<typename ... Ts>
struct S {};

using S0 = S<>;

template<typename T0>
using S1 = S<T0>;

template<typename T0, typename T1>
using S2 = S<T0, T1>;

template<typename T0, typename T1, typename T2>
using S3 = S<T0, T1, T2>;

then exporting all of S0, S1, ... individually in the .pxd file.

This is inconvenient, but acceptable.

Now assume that S has a member function f,

	template<typename ... Us>
	void f(Ts..., Us ...);

due to the nature of C++, the same trick as above does not work, and the
user will be forced to clutter the C++ code with a function body for
every possible len(Us). Even worse, the .pxd file will now contain a
quadratic number of wrappers (one for every possible combination of
len(Ts), len(Us)), all of them extremely prone to mistakes.

Thanks for reading,

  ~ Michael

Stephen LARROQUE | 15 Jun 12:34 2015
Picon

[Cython] Bug: Extension Type inheriting from int cause a MemoryError

Hello,

I am trying to make an extension type inheriting from int or cython.int (to do arithmetic operations in Galois Fields). However, this causes a MemoryError because it seems such extension type is not freed correctly. Other than that, it works perfectly well.

Here is a test case to reproduce the bug on Python 2.7.9 Win32 (I'm running Windows 7, but it may happen on other platform, I did not test) with Cython v0.22 on Anaconda:

########################
# mode: run

import sys

cdef class ExtendedInt(int): pass

_ERRORS = u"""
MemoryError
"""

# Simple test case to raise a MemoryError by generating lots of ExtendedInt
total_it = 1000
for i in xrange(total_it):
    for j in xrange(10000000):
        ExtendedInt(j)
    sys.stdout.write("\rGenerating lists of ExtendedInt : %i/%i" % (i, total_it))
########################

My current workaround as advised in the cython-users list:

is to inherit from standard (object) and then reimplement all int magic methods, which is cumbersome and less optimized, but at least there's no MemoryError.

<div><div dir="ltr">Hello,<div><br></div>
<div>
<span>I am trying to make an extension type inheriting from int or&nbsp;</span><a href="http://cython.int/" target="_blank" rel="nofollow">cython.int</a>&nbsp;(to do arithmetic operations in Galois Fields). However, this causes a MemoryError because it seems such extension type is not freed correctly. Other than that, it works perfectly well.<br>
</div>
<div><br></div>
<div>
<span>Here is a test case to reproduce the bug on Python 2.7.9 Win32 (I'm running Windows 7, but it may happen on other platform, I did not test) with Cython v0.22 on Anaconda:</span><br>
</div>
<div><span><br></span></div>
<div>
<span>########################</span><br>
</div>
<div>
<div># mode: run</div>
<div><br></div>
<div>import sys</div>
<div><br></div>
<div>cdef class ExtendedInt(int): pass</div>
<div><br></div>
<div>_ERRORS = u"""</div>
<div>MemoryError</div>
<div>"""</div>
<div><br></div>
<div># Simple test case to raise a MemoryError by generating lots of ExtendedInt</div>
<div>total_it = 1000</div>
<div>for i in xrange(total_it):</div>
<div>&nbsp; &nbsp; for j in xrange(10000000):</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; ExtendedInt(j)</div>
<div>&nbsp; &nbsp; sys.stdout.write("\rGenerating lists of ExtendedInt : %i/%i" % (i, total_it))</div>
<div>########################</div>
<div><br></div>
<div>My current workaround as advised in the cython-users list:</div>
<div>
<a href="https://groups.google.com/forum/#!topic/cython-users/WJ5BJ5moAh4">https://groups.google.com/forum/#!topic/cython-users/WJ5BJ5moAh4</a><br>
</div>
<div><br></div>
<div>is to inherit from standard (object) and then reimplement all int magic methods, which is cumbersome and less optimized, but at least there's no MemoryError.</div>
<div><br></div>
</div>
</div></div>
Andrew Svetlov | 13 Jun 08:30 2015
Picon

[Cython] Bug: comprehensions clear current exception

I have an issue in aiohttp library:
https://github.com/KeepSafe/aiohttp/issues/410

The source of problem is: when I execute the following code

body = ', '.join("'{}': {!r}".format(k, v) for k, v in self.items())

in except block Cython clears exception.

Changing from comprehension to regular iteration works well:

lst = []
for k, v in self._items:
    lst.append("'{}': {!r}".format(k, v))
body = ', '.join(lst)

In pyre Python both versions don't clear current exception.

I think it's a Cython bug.

--

-- 
Thanks,
Andrew Svetlov
Michael Enßlin | 12 Jun 23:24 2015

[Cython] Feature request: Flag to enable warnings when 'extern' functions are not declared 'except +'

Hi everybody,

as you surely have guessed from the amount of Bug reports I have
recently submitted to this mailing list, I'm currently working on a
rather large Cython project.
More precisely: I'm using Cython as the glue layer between the Python
and C++ components of openage; an overview and source code may be found
here:
https://github.com/mic-e/openage/blob/pyinterface/doc/implementation/pyinterface.md
https://github.com/mic-e/openage/tree/pyinterface/cpp/pyinterface
https://github.com/mic-e/openage/tree/pyinterface/openage/cppinterface

In openage, almost all C++ functions that are exposed to Cython SHOULD
be declared as 'except +'.
Forgetting to do so can cause hard-to-debug issues, including useless
Exception messages, corruption of CPython and right-out program
termination, if those methods should throw a C++ exception.

As a solution, I'd like to see Cython warnings if an extern cimport has
no 'except' declaration.

To intentionally omit the "except +", I suggest a new except
declaration, "except -", or, even better, the C++11 keyword "noexcept"
(since precisely those methods that are declared 'noexcept' should be
cimported without 'except+').

Of course, those warnings (or errors?) should be disabled by default,
and enabled by a special command-line flag.

Any thoughts on this?

	~ mic_e
Michael Enßlin | 12 Jun 13:01 2015

[Cython] Can't call functions with multi-token template arguments such as 'char *'

Hi,

it seems to be impossible to use anything but a single word as a
template type for functions.

Classes don't suffer from this issue, as seen below.

As a workaround, complex types can be ctypedef-d to a single word (also
seen below).

$ cat t10.pyx

cdef extern from "nope.h":
    cdef cppclass bar[T]:
        void func(T arg)

    void foo[T](T arg)

ctypedef char * charptr

def test():
    # works
    cdef bar[char *] barobj
    barobj.func(NULL)

    # works
    foo[int](5)

    # works
    foo[charptr](NULL)

    # fails
    foo[char *](NULL)

$ cython --cplus t10.pyx

Error compiling Cython file:
------------------------------------------------------------
...

    # works
    foo[charptr](NULL)

    # fails
    foo[char *](NULL)
             ^
------------------------------------------------------------

t10.pyx:27:14: Expected an identifier or literal

Happy debugging,
	~ mic_e

Hi,

it seems to be impossible to use anything but a single word as a
template type for functions.

Classes don't suffer from this issue, as seen below.

As a workaround, complex types can be ctypedef-d to a single word (also
seen below).

$ cat t10.pyx

cdef extern from "nope.h":
    cdef cppclass bar[T]:
        void func(T arg)

    void foo[T](T arg)

ctypedef char * charptr

def test():
    # works
    cdef bar[char *] barobj
    barobj.func(NULL)

    # works
    foo[int](5)

    # works
    foo[charptr](NULL)

    # fails
    foo[char *](NULL)

$ cython --cplus t10.pyx

Error compiling Cython file:
------------------------------------------------------------
...

    # works
    foo[charptr](NULL)

    # fails
    foo[char *](NULL)
             ^
------------------------------------------------------------

t10.pyx:27:14: Expected an identifier or literal

Happy debugging,
	~ mic_e

Michael Enßlin | 12 Jun 12:51 2015

[Cython] Compiler crash in AnalyseExpressionsTransform

Hi,

another bug report:

mic <at> mic /tmp $ cat t11.pyx
cdef cppclass foo:
    pass

def test():
    foo()

mic <at> mic /tmp $ cython --cplus t11.pyx

Error compiling Cython file:
------------------------------------------------------------
...
cdef cppclass foo:
    pass

def test():
    foo()
      ^
------------------------------------------------------------

t11.pyx:5:7: Compiler crash in AnalyseExpressionsTransform

ModuleNode.body = StatListNode(t11.pyx:1:0)
StatListNode.stats[1] = DefNode(t11.pyx:4:0,
    modifiers = [...]/0,
    name = u'test',
    py_wrapper_required = True,
    reqd_kw_flags_cname = '0',
    used = True)
File 'Nodes.py', line 421, in analyse_expressions: StatListNode(t11.pyx:5:7)
File 'Nodes.py', line 4652, in analyse_expressions:
ExprStatNode(t11.pyx:5:7)
File 'ExprNodes.py', line 434, in analyse_expressions:
SimpleCallNode(t11.pyx:5:7,
    use_managed_ref = True)
File 'ExprNodes.py', line 4495, in analyse_types:
SimpleCallNode(t11.pyx:5:7,
    use_managed_ref = True)
File 'ExprNodes.py', line 4429, in analyse_as_type_constructor:
SimpleCallNode(t11.pyx:5:7,
    use_managed_ref = True)

Compiler crash traceback from this point on:
  File "/usr/lib/python2.7/dist-packages/Cython/Compiler/ExprNodes.py",
line 4429, in analyse_as_type_constructor
    self.function = RawCNameExprNode(self.function.pos, constructor.type)
AttributeError: 'NoneType' object has no attribute 'type'

The code  is obviously nonsensical, but Cython should produce a proper
error message instead of crashing.

Hi,

another bug report:

mic <at> mic /tmp $ cat t11.pyx
cdef cppclass foo:
    pass

def test():
    foo()

mic <at> mic /tmp $ cython --cplus t11.pyx

Error compiling Cython file:
------------------------------------------------------------
...
cdef cppclass foo:
    pass

def test():
    foo()
      ^
------------------------------------------------------------

t11.pyx:5:7: Compiler crash in AnalyseExpressionsTransform

ModuleNode.body = StatListNode(t11.pyx:1:0)
StatListNode.stats[1] = DefNode(t11.pyx:4:0,
    modifiers = [...]/0,
    name = u'test',
    py_wrapper_required = True,
    reqd_kw_flags_cname = '0',
    used = True)
File 'Nodes.py', line 421, in analyse_expressions: StatListNode(t11.pyx:5:7)
File 'Nodes.py', line 4652, in analyse_expressions:
ExprStatNode(t11.pyx:5:7)
File 'ExprNodes.py', line 434, in analyse_expressions:
SimpleCallNode(t11.pyx:5:7,
    use_managed_ref = True)
File 'ExprNodes.py', line 4495, in analyse_types:
SimpleCallNode(t11.pyx:5:7,
    use_managed_ref = True)
File 'ExprNodes.py', line 4429, in analyse_as_type_constructor:
SimpleCallNode(t11.pyx:5:7,
    use_managed_ref = True)

Compiler crash traceback from this point on:
  File "/usr/lib/python2.7/dist-packages/Cython/Compiler/ExprNodes.py",
line 4429, in analyse_as_type_constructor
    self.function = RawCNameExprNode(self.function.pos, constructor.type)
AttributeError: 'NoneType' object has no attribute 'type'

The code  is obviously nonsensical, but Cython should produce a proper
error message instead of crashing.

Michael Enßlin | 11 Jun 18:03 2015

[Cython] Cython produces invalid code for operator()

Hi guys,

have a look at this:

$ cat bar.pyx
cdef extern from "foo.h":
    cdef cppclass Foo:
        int operator() (int arg)
        int do_call (int arg)

cdef int bar(int arg):
    cdef Foo foo
    foo.do_call(arg)
    return foo(arg)

$ cython3 --cplus bar.pyx
$ cat bar.cpp

(...)

static int __pyx_f_3bar_bar(int __pyx_v_arg) {
  Foo __pyx_v_foo;
  int __pyx_r;
  __Pyx_RefNannyDeclarations
  __Pyx_RefNannySetupContext("bar", 0);

  /* "bar.pyx":9
 * cdef int bar(int arg):
 *     cdef Foo foo
 *     foo.do_call(arg)             # <<<<<<<<<<<<<<
 *     return foo(arg)
 */
  __pyx_v_foo.do_call(__pyx_v_arg);

  /* "bar.pyx":10
 *     cdef Foo foo
 *     foo.do_call(arg)
 *     return foo(arg)             # <<<<<<<<<<<<<<
 */
  __pyx_r = operator()(__pyx_v_arg);
  goto __pyx_L0;

  /* "bar.pyx":7
 *
 *
 * cdef int bar(int arg):             # <<<<<<<<<<<<<<
 *     cdef Foo foo
 *     foo.do_call(arg)
 */

  /* function exit code */
  __pyx_L0:;
  __Pyx_RefNannyFinishContext();
  return __pyx_r;
}

(...)

Note how the function invocation for "do_call" is generated correctly,
but the invocation of operator() is nonsensical.

The correct line would be this:

  __pyx_r = __pyx_v_foo(__pyx_v_arg);

Happy debugging :D

	~ mic_e

Hi guys,

have a look at this:

$ cat bar.pyx
cdef extern from "foo.h":
    cdef cppclass Foo:
        int operator() (int arg)
        int do_call (int arg)

cdef int bar(int arg):
    cdef Foo foo
    foo.do_call(arg)
    return foo(arg)

$ cython3 --cplus bar.pyx
$ cat bar.cpp

(...)

static int __pyx_f_3bar_bar(int __pyx_v_arg) {
  Foo __pyx_v_foo;
  int __pyx_r;
  __Pyx_RefNannyDeclarations
  __Pyx_RefNannySetupContext("bar", 0);

  /* "bar.pyx":9
 * cdef int bar(int arg):
 *     cdef Foo foo
 *     foo.do_call(arg)             # <<<<<<<<<<<<<<
 *     return foo(arg)
 */
  __pyx_v_foo.do_call(__pyx_v_arg);

  /* "bar.pyx":10
 *     cdef Foo foo
 *     foo.do_call(arg)
 *     return foo(arg)             # <<<<<<<<<<<<<<
 */
  __pyx_r = operator()(__pyx_v_arg);
  goto __pyx_L0;

  /* "bar.pyx":7
 *
 *
 * cdef int bar(int arg):             # <<<<<<<<<<<<<<
 *     cdef Foo foo
 *     foo.do_call(arg)
 */

  /* function exit code */
  __pyx_L0:;
  __Pyx_RefNannyFinishContext();
  return __pyx_r;
}

(...)

Note how the function invocation for "do_call" is generated correctly,
but the invocation of operator() is nonsensical.

The correct line would be this:

  __pyx_r = __pyx_v_foo(__pyx_v_arg);

Happy debugging :D

	~ mic_e

Michael Enßlin | 8 Jun 16:50 2015

[Cython] Feature request: Expose methods to fake stack traces

Hi everybody,

my C++ exceptions contain some stack trace information (starting with
__FILE__ and __LINE__), and I'd like to preserve that information in my
custom 'except+' translator.

Cython seems to have some internal methods to fake Python stack trace
objects, but research (and an unanswered question on the Users list
[https://groups.google.com/forum/#!topic/cython-users/9coFCVwigpE])
seems to imply that this functionality is not currently exposed.

I'd like to see that internal functionality exposed for usage from .pyx
files.

 ~ mic_e

Hi everybody,

my C++ exceptions contain some stack trace information (starting with
__FILE__ and __LINE__), and I'd like to preserve that information in my
custom 'except+' translator.

Cython seems to have some internal methods to fake Python stack trace
objects, but research (and an unanswered question on the Users list
[https://groups.google.com/forum/#!topic/cython-users/9coFCVwigpE])
seems to imply that this functionality is not currently exposed.

I'd like to see that internal functionality exposed for usage from .pyx
files.

 ~ mic_e


Gmane