Ian Henriksen | 20 Jul 23:06 2015
Picon

[Cython] Non-type template parameters

Hi all,
I've spent a little time working on adding support for non-type
template parameters. In doing this, it has been very easy to add
support for doing something like the following:

cdef extern from "add_const.hpp" nogil:
    int myfunc1[i](int)
def test():
    print myfunc1[2](a)

The downside of this is that it does not let the Cython compiler
distinguish between different kinds of template parameters.

Stricter checking could be added using a syntax like this:

cdef extern from "add_const.hpp" nogil:
    int myfunc1[int i](int)
def test():
    print myfunc1[2](a)

The downsides here are that the syntax doesn't really match the
existing template syntax. It will also complicate the Cython codebase
since we'll have to go to greater lengths to allow or disallow all the
different special cases for templates.

Which version would be preferable?

On a similar note, for variadic templates, would we prefer something
like

cdef extern from "my_variadic.hpp" nogil:
    T myfunc2[T,...](T, ...)

or something like:

cdef extern from "my_variadic.hpp" nogil:
    T myfunc2[T, Types...](T, Types... args)

Again, the latter syntax is more explicit, but it will require much more
complicated code in Cython. It also doesn't match the existing syntax
very well. The former syntax matches the existing syntax for templates
better, but will make it hard for Cython to raise errors early on in
compilation.

I'd greatly appreciate any input on the best syntax for either use-case.

Regards,

-Ian Henriksen
<div><div dir="ltr">Hi all,<div>I've spent a little time working on adding support for non-type</div>
<div>template parameters. In doing this, it has been very easy to add</div>
<div>support for doing something like the following:</div>
<div><br></div>
<div>
<div>cdef extern from "add_const.hpp" nogil:</div>
<div>&nbsp; &nbsp; int myfunc1[i](int)</div>
</div>
<div>def test():</div>
<div>&nbsp; &nbsp;&nbsp;print myfunc1[2](a)</div>
<div><br></div>
<div>The downside of this is that it does not let the Cython compiler</div>
<div>distinguish between different kinds of template parameters.</div>
<div><br></div>
<div>Stricter checking could be added using a syntax like this:</div>
<div><br></div>
<div>
<div>
<div>cdef extern from "add_const.hpp" nogil:</div>
<div>&nbsp; &nbsp; int myfunc1[int i](int)</div>
</div>
<div>def test():</div>
<div>&nbsp; &nbsp;&nbsp;print myfunc1[2](a)</div>
</div>
<div><br></div>
<div>The downsides here are that the syntax doesn't really match the</div>
<div>existing template syntax. It will also complicate the Cython codebase</div>
<div>since we'll have to go to greater lengths to allow or disallow all the</div>
<div>different special cases for templates.</div>
<div><br></div>
<div>Which version would be preferable?</div>
<div><br></div>
<div>On a similar note, for variadic templates, would we prefer something</div>
<div>like</div>
<div><br></div>
<div>cdef extern from "my_variadic.hpp" nogil:</div>
<div>&nbsp; &nbsp; T myfunc2[T,...](T, ...)</div>
<div><br></div>
<div>or something like:</div>
<div><br></div>
<div>cdef extern from "my_variadic.hpp" nogil:</div>
<div>&nbsp; &nbsp; T myfunc2[T, Types...](T, Types... args)</div>
<div><br></div>
<div>Again, the latter syntax is more explicit, but it will require much more</div>
<div>complicated code in Cython. It also doesn't match the existing syntax</div>
<div>very well. The former syntax matches the existing syntax for templates</div>
<div>better, but will make it hard for Cython to raise errors early on in</div>
<div>compilation.</div>
<div><br></div>
<div>I'd greatly appreciate any input on the best syntax for either use-case.</div>
<div><br></div>
<div>Regards,</div>
<div><br></div>
<div>-Ian Henriksen</div>
</div></div>
Brian McAndrews | 16 Jul 23:13 2015

[Cython] Problem with declaring a map with an enum type.

I have a C enum type that I’m attempting to put in a C++ map:

Here’s what I got:

 

colors.h:

enum Color

  {

    GREEN,BLUE,RED

  };

 

val.pyx:

from libcpp.string cimport string

from libcpp.map cimport map

 

cdef extern from "colors.h":

    cpdef enum _Color "Color":

       GREEN,BLUE,RED

 

cdef class TestMe:

    cdef public map[string, _Color] vals

 

 

I get a compile error (VSC++ 2012)  

VAL.cpp(1239) : error C2440: '=' : cannot convert from 'long' to 'Color'

        Conversion to enumeration type requires an explicit cast (static_cast, C-style cast or function-style cast)

 

The offending statement is this:

    __pyx_t_9 = PyInt_AsLong(__pyx_v_value); if (unlikely(PyErr_Occurred())) {__pyx_filename = __pyx_f[1]; __pyx_lineno = 202; __pyx_clineno = __LINE__; goto __pyx_L1_error;}

 

In the __pyx_convert_map_from_py_std_3a__3a_string__and_enum__Color function.

 

The cast from PyInt_AsLong(…) to an enum Color is not specified when using maps.

 

If I don’t put it in a map (just as a member variable), the cast is performed.

 

Has anyone seen this before?

 

Thanks,

Brian







This electronic mail message and any attached files contain information intended for the exclusive use of the individual or entity to whom it is addressed and may contain information that is proprietary, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any viewing, copying, disclosure or distribution of this information may be subject to legal restriction or sanction. Please notify the sender, by electronic mail or telephone, of any unintended recipients and delete the original message without making any copies.
<div>
<div class="WordSection1">
<p class="MsoNormal">I have a C enum type that I&rsquo;m attempting to put in a C++ map:<p></p></p>
<p class="MsoNormal">Here&rsquo;s what I got:<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">colors.h:<p></p></p>
<p class="MsoNormal">enum Color<p></p></p>
<p class="MsoNormal">&nbsp; {<p></p></p>
<p class="MsoNormal">&nbsp;&nbsp;&nbsp; GREEN,BLUE,RED<p></p></p>
<p class="MsoNormal">&nbsp; };<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">val.pyx:<p></p></p>
<p class="MsoNormal">from libcpp.string cimport string<p></p></p>
<p class="MsoNormal">from libcpp.map cimport map<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">cdef extern from "colors.h":<p></p></p>
<p class="MsoNormal">&nbsp;&nbsp;&nbsp; cpdef enum _Color "Color":<p></p></p>
<p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; GREEN,BLUE,RED<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">cdef class TestMe:<p></p></p>
<p class="MsoNormal">&nbsp;&nbsp;&nbsp; cdef public map[string, _Color] vals<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">I get a compile error (VSC++ 2012) &nbsp;<p></p></p>
<p class="MsoNormal">VAL.cpp(1239) : error C2440: '=' : cannot convert from 'long' to 'Color'<p></p></p>
<p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Conversion to enumeration type requires an explicit cast (static_cast, C-style cast or function-style cast)<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">The offending statement is this:<p></p></p>
<p class="MsoNormal">&nbsp;&nbsp;&nbsp; __pyx_t_9 = PyInt_AsLong(__pyx_v_value); if (unlikely(PyErr_Occurred())) {__pyx_filename = __pyx_f[1]; __pyx_lineno = 202; __pyx_clineno = __LINE__; goto __pyx_L1_error;}<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">In the __pyx_convert_map_from_py_std_3a__3a_string__and_enum__Color function.<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">The cast from PyInt_AsLong(&hellip;) to an enum Color is not specified when using maps.<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">If I don&rsquo;t put it in a map (just as a member variable), the cast is performed.<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">Has anyone seen this before?<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">Thanks,<p></p></p>
<p class="MsoNormal">Brian<p></p></p>
</div>
<br><br><br><br><br><br><h5>This electronic mail message and any attached files contain information intended for the exclusive use of the individual or entity to whom it is addressed and may contain information that is proprietary, confidential and/or exempt from
 disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any viewing, copying, disclosure or distribution of this information may be subject to legal restriction or sanction. Please notify the sender, by electronic
 mail or telephone, of any unintended recipients and delete the original message without making any copies.
</h5>

</div>
Alex S. | 15 Jul 13:51 2015
Picon

[Cython] Fwd: [BUG] Cython failing performance when Numpy is not installed


Hello,
being unable to acquire the Trac account (Robert Bradshaw's email is not listed on his site), I write here.
The issue was described here: http://trac.cython.org/ticket/847 but was, however, misidentified. The wrapper for a function which receives "fused slice" (D[:]) arguments automatically includes the following code:
>                 cdef type ndarray
>                 try:
>                     import numpy
>                     ndarray = numpy.ndarray
>                 except (ImportError, AttributeError, TypeError):
>                     ndarray = None
(Compiler/FusedNode.py:468).
When numpy is not installed, each import tries to search for it in the system path. (When it is, it's just cached). This severely degrades the performance.


<div><div dir="ltr">
<br><div class="gmail_quote">
<div dir="ltr">
<div>
<div>Hello,<br>being unable to acquire the Trac account (Robert Bradshaw's email is not listed on his site), I write here.<br>The issue was described here: <a href="http://trac.cython.org/ticket/847" target="_blank">http://trac.cython.org/ticket/847</a> but was, however, misidentified. The wrapper for a function which receives "fused slice" (D[:]) arguments automatically includes the following code:<br>&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; cdef type ndarray<br>&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; try:<br>&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; import numpy<br>&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ndarray = numpy.ndarray<br>&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; except (ImportError, AttributeError, TypeError):<br>&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ndarray = None<br>
</div>(Compiler/FusedNode.py:468).<br>
</div>When numpy is not installed, each import tries to search for it in the system path. (When it is, it's just cached). This severely degrades the performance.<br>
</div>
<br>
</div>
<br>
</div></div>
Ian Henriksen | 15 Jul 01:32 2015
Picon

[Cython] Handling of cascaded assignment

I came across this while working on allowing proper overloading of the
assignment operator.
The problem is, essentially, that Cython does not follow the C/C++
ordering for cascaded assignment, even when dealing with objects at
the C/C++ level. For example, calling the test function from the
following Cython file and running the following C++ file should
both give the same output:

def test():
    cdef:
        long long a = (2LL << 36) - 257
        int b
        char c
    b = c = a
    print a, b, c



#include <iostream>
int main(){
    long long a = (2LL << 36) - 257;
    int b;
    char c;
    b = c = a;
    std::cout << a << std::endl;
    std::cout << b << std::endl;
    std::cout << (int)c << std::endl;
}

The Cython version prints 137438953215 -257 -1
and the C++ version prints 137438953215 -1 -1

To mimic C/C++, it should go through the arguments from right to left
and perform each assignment operation in that order, i.e.
c = a
b = c
As it stands, the current code does:
b = a
c = a

This does appear to follow what Python does. For example, the
following gives the same output as the Cython version:

import numpy as np
a = np.array([(2 << 36) - 257], np.int64)
b = np.empty(1, np.int32)
c = np.empty(1, np.int8)
b[:] = c[:] = a
print a[0], b[0], c[0]

Is this a deliberate design decision? If this weren't already included in
the language, I'd be inclined to not allow cascaded assignment to C
objects at all since it can give such surprising results. On the other
hand I don't see any particularly clean way to deal with this
incompatibility between C/C++ and Python.

Thanks!

-Ian Henriksen
<div><div dir="ltr">
<span>I came across this while working on allowing proper overloading of the</span><div>
<span>assignment operator.</span><div>The problem is, essentially, that Cython does not follow the C/C++</div>
<div>ordering for cascaded assignment, even when dealing with objects at</div>
<div>the C/C++ level. For example, calling the test function from the</div>
<div>following Cython file and running the following C++ file should</div>
<div>both give the same output:</div>
<div><br></div>
<div>
<div>def test():</div>
<div>&nbsp; &nbsp; cdef:</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; long long a = (2LL &lt;&lt; 36) - 257</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; int b</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; char c</div>
<div>&nbsp; &nbsp; b = c = a</div>
<div>&nbsp; &nbsp; print a, b, c</div>
</div>
<div><br></div>
<div><br></div>
<div><br></div>
<div>
<div>#include &lt;iostream&gt;</div>
<div>int main(){</div>
<div>&nbsp; &nbsp; long long a = (2LL &lt;&lt; 36) - 257;</div>
<div>&nbsp; &nbsp; int b;</div>
<div>&nbsp; &nbsp; char c;</div>
<div>&nbsp; &nbsp; b = c = a;</div>
<div>&nbsp; &nbsp; std::cout &lt;&lt; a &lt;&lt; std::endl;</div>
<div>&nbsp; &nbsp; std::cout &lt;&lt; b &lt;&lt; std::endl;</div>
<div>&nbsp; &nbsp; std::cout &lt;&lt; (int)c &lt;&lt; std::endl;</div>
<div>}</div>
</div>
<div><br></div>
<div>The Cython version prints&nbsp;137438953215 -257 -1</div>
<div>and the C++ version prints&nbsp;137438953215 -1 -1</div>
<div><br></div>
<div>To mimic C/C++, it should go through the arguments from right to left</div>
<div>and perform each assignment operation in that order, i.e.</div>
<div>c = a</div>
<div>b = c</div>
<div>As it stands, the current code does:</div>
<div>b = a</div>
<div>c = a</div>
</div>
<div><br></div>
<div>This does appear to follow what Python does. For example, the</div>
<div>following gives the same output as the Cython version:</div>
<div><br></div>
<div>import numpy as np</div>
<div>a = np.array([(2 &lt;&lt; 36) - 257], np.int64)</div>
<div>b = np.empty(1, np.int32)</div>
<div>c = np.empty(1, np.int8)</div>
<div>b[:] = c[:] = a</div>
<div>print a[0], b[0], c[0]</div>
<div><br></div>
<div>Is this a deliberate design decision? If this weren't already included in</div>
<div>the language, I'd be inclined to not allow cascaded assignment to C</div>
<div>objects at all since it can give such surprising results. On the other</div>
<div>hand I don't see any particularly clean way to deal with this</div>
<div>incompatibility between C/C++ and Python.</div>
<div><br></div>
<div>Thanks!</div>
<div><br></div>
<div>-Ian Henriksen</div>
</div></div>
Mark Florisson | 14 Jul 20:34 2015
Picon

[Cython] overflow bug?

Hi,

I think this might be a bug (python 3.3 + cython 0.22.1):

    def f(term=b"12345"):
val = int('987278186585')
# The below line does not work, because it treats 1 as a constant integer
# in the C code (32 bit on my machine). Using 1L does work however.
val -= 1 << (len(term) * 8)
return val
 
print(f())

This works in pure-python, but Cython generates '1 << __pyx_t_somevar', which I think treats the '1' as an integer (causing it to overflow). Using '1L' works in the Cython code however (but that may be just my platform).
Cheers,Mark
<div><div dir="ltr">Hi,<div><br></div>
<div>I think this might be a bug (python 3.3&nbsp;+ cython 0.22.1):</div>
<div><br></div>
<div>&nbsp; &nbsp;&nbsp;<span>def</span><span> </span><span>f</span><span>(</span><span>term</span><span>=</span><span>b</span><span><span>"</span>12345<span>"</span></span><span>):</span>
</div>
<div>    val <span>=</span> <span>int</span>(<span><span>'</span>987278186585<span>'</span></span>)
</div>
<div>    <span># The below line does not work, because it treats 1 as a constant integer</span>
</div>
<div>    <span># in the C code (32 bit on my machine). Using 1L does work however.</span>
</div>
<div>    val <span>-=</span> <span>1</span> <span>&lt;&lt;</span> (<span>len</span>(term) <span>*</span> <span>8</span>)
</div>
<div>    <span>return</span> val 
</div>
<div>&nbsp;
</div>
<div>
<span>print</span>(f())</div>
<div><br></div>This works in pure-python, but Cython generates '1 &lt;&lt; __pyx_t_somevar', which I think treats the '1' as an integer (causing it to overflow). Using '1L' works in the Cython code however (but that may be just my platform).<br>Cheers,Mark </div></div>
Jeroen Demeyer | 13 Jul 15:56 2015
Picon

[Cython] Regression: cdef append() no longer works

Code like the following no longer works correctly:

cdef class Foo:
     def __init__(self):
         self.append(1)

     cdef append(self, x):
         return x

It seems that Cython now always assumes that append() is a special 
method: it incorrectly uses __Pyx_PyObject_Append.

PS: it's not really clear if bug reports like these should go on this 
mailing list or on Trac (my impression is that Cython's Trac is not 
really used). What's the recommended procedure?
Josh Ayers | 7 Jul 17:11 2015

[Cython] PyUnicode_Tailmatch

Cython devs,

In the function __Pyx_PyUnicode_Tailmatch, the return type of
PyUnicode_Tailmatch is assumed to be in int.  See line 543 of
Cython/Utility/StringTools.c.  

PyUnicode_Tailmatch actually returns a Py_ssize_t.  See:
https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_Tailmatch.

For reference, there was previously an error in the Python documentation
of this function.  The documentation mistakenly said it returned an int.
 This was fixed via Python issue 22580:
https://bugs.python.org/issue22580.

Thanks,
Josh Ayers
Ian Henriksen | 4 Jul 00:43 2015
Picon

[Cython] Multidimensional indexing of C++ objects

Hi everyone,
I'm a GSOC student working to make a Cython API for DyND. DyND is a relatively new n-dimensional array library in C++ that is based on NumPy. A full set of Python bindings (created using Cython) are provided as a separate package. The goal of my project is to make it so that DyND arrays can be used easily within Cython so that an n-dimensional array object can be used without any of the corresponding Python overhead.

Currently, there isn't a good way to assign to multidimensional slices within Cython. Since the indexing operator in C++ is limited to a single argument, we use the call operator to represent multidimensional indexing, and then use a proxy class to perform assignment to a slice.
Currently, in C++, assigning to a slice along the second axis of a DyND array looks like this:

a(irange(), 1).vals() = 0;

Unfortunately, in Cython, only the index operator can be used for assignment, so following the C++ syntax isn't currently possible. Does anyone know of a good way to address this? I'm willing to spend some time implementing a new feature if we can reach a consensus on a good way to deal with this. Here are some possible solutions I've thought of:

1. We could allow assignment to C++ method and function calls that return references. This has the advantage that it matches the existing syntax in C++ for dealing with C++ objects. Though Cython is a Python-like language, the ability to manipulate C++ objects directly is a key part of its feature set. Since the native way to do things like multidimensional indexing in C++ is via the call operator, it seems sensible to allow assignment to C++-level call operations in Cython as well. This could be enabled via a Cython compiler directive and be disabled by default. Using a compiler directive like this would result in an interface similar to the one already used for cdivision, wrap-around indexing, and index bounds checking. The user would avoid unexpected results by default, but be able to get the needed functionality simply by enabling it.

2. We could recommend that all assignment operations of this nature be wrapped in a fake method that wraps the assignment in it's c-level name. This has the advantage that it works in current and past versions of Cython, but it is a rather unusual hack. For example, something like the following would work right now:

# declared as a method in a pxd file:
void assign "vals() = "(int value) except +

# used in a pyx file to perform assignment to a slice of an array a:
a(irange(), 1).assign(0)

For DyND, at least for now, this would be a workable solution since the difference lies primarily in the placement of the parenthesis and the presence of the assignment operator. The syntax is less clear than it could be, but it would work. On the other hand, other libraries may not be so lucky since this involves replacing assignment to a slice with a method call. For example, the expression template libraries Eigen and Blaze-lib would encounter incompatibility to varying degrees if someone were to try using them within Cython. This method also has the disadvantage that it creates an interface that is fundamentally different from both the Python and C++ interfaces.

I have also considered, writing a proxy class that can serve as an effective temporary value while a multidimensional index is constructed from a series of calls to operator[]. This is a reasonable approach, but it leads to unnecessary code bloat. It also complicates the interface exposed to users, since operator[] would be needed for left hand values and operator() would be needed for right hand values. This would also make it so that users that want to use these C++ classes in Cython would have to include and link against another set of headers and libraries to be able to use the proxy class. The burden of maintainability for Python bindings created in this way would be greater as well. This also isn't a viable approach for using any C++ class that overloads both operators.

Another option I have considered is allowing Cython's indexing operator to dispatch to a different function. Currently, user-defined cname entries for overloaded operators are not used. If this were changed for the indexing operator, indexing could be performed at the C++ level using some other method. This doesn't look like a viable approach though, since, for this to really work, users would need some way to call different methods when a C++ object is being indexed and when it is being assigned to. Using operator[] for left-hand values and operator() for right-hand values is a possible solution, but that isn't a very consistent interface. Doing this would also increase the complexity of the existing code for indexing in the Cython compiler and could lead to name collisions for classes that overload both operator[] and operator().

Are any of these acceptable ways to go forward? Does anyone have any better ideas? My preference would definitely be toward allowing C++ calls returning references to be used as lvalues, but I'd really appreciate any alternative solutions.

Thanks!
-Ian Henriksen


<div><div dir="ltr">Hi everyone,<div>I'm a GSOC student working to make a Cython API for DyND.&nbsp;<a href="https://github.com/libdynd/libdynd" target="_blank">DyND</a>&nbsp;is a relatively new n-dimensional array library in C++ that is based on NumPy. A full set of Python bindings (created using Cython) are provided as a separate package. The goal of my project is to make it so that DyND arrays can be used easily within Cython so that an n-dimensional array object can be used without any of the corresponding Python overhead.</div>
<div><br></div>
<div>Currently, there isn't a good way to assign to multidimensional slices within Cython. Since the indexing operator in C++ is limited to a single argument, we use the call operator to represent multidimensional indexing, and then use a proxy class to perform assignment to a slice.</div>
<div>Currently, in C++, assigning to a slice along the second axis of a DyND array looks like this:</div>
<div><br></div>
<div>
<div>a(irange(), 1).vals() = 0;</div>
<div><br></div>
<div>Unfortunately, in Cython, only the index operator can be used for assignment, so following the C++ syntax isn't currently possible. Does anyone know of a good way to address this? I'm willing to spend some time implementing a new feature if we can reach a consensus on a good way to deal with this. Here are some possible solutions I've thought of:</div>
<div><br></div>
<div>1. We could allow assignment to C++ method and function calls that return references. This has the advantage that it matches the existing syntax in C++ for dealing with C++ objects. Though Cython is a Python-like language, the ability to manipulate C++ objects directly is a key part of its feature set. Since the native way to do things like multidimensional indexing in C++ is via the call operator, it seems sensible to allow assignment to C++-level call operations in Cython as well. This could be enabled via a Cython compiler directive and be disabled by default. Using a compiler directive like this would result in an interface similar to the one already used for cdivision, wrap-around indexing, and index bounds checking. The user would avoid unexpected results by default, but be able to get the needed functionality simply by enabling it.</div>
<div><br></div>
<div>2. We could recommend that all assignment operations of this nature be wrapped in a fake method that wraps the assignment in it's c-level name. This has the advantage that it works in current and past versions of Cython, but it is a rather unusual hack. For example, something like the following would work right now:</div>
<div><br></div>
<div># declared as a method in a pxd file:</div>
<div>
<span>void assign "vals() = "(int value) except&nbsp;+</span><br>
</div>
<div><span><br></span></div>
<div><span># used in a pyx file to perform assignment to a slice of an array a:</span></div>
<div><span>a(irange(), 1).assign(0)</span></div>
<div><span><br></span></div>
<div><span>For DyND, at least for now, this would be a workable solution since the difference lies primarily in the placement of the parenthesis and the presence of the assignment operator. The syntax is less clear than it could be, but it would work. On the other hand, other libraries may not be so lucky since this involves replacing assignment to a slice with a method call. For example, the expression template libraries Eigen and Blaze-lib would encounter incompatibility to varying degrees if someone were to try using them within Cython. This method also has the disadvantage that it creates an interface that is fundamentally different from both the Python and C++ interfaces.</span></div>
<div><span><br></span></div>
<div>
<span>I have also considered, writing a proxy class that can serve as an effective temporary value while a multidimensional index is constructed from a series of calls to operator[]. This is a reasonable approach, but it leads to unnecessary code bloat. It also complicates the interface exposed to users, since</span>&nbsp;operator[] would be needed for left hand values and operator() would be needed for right hand values. This would also make it so that users that want to use these C++ classes in Cython would have to include and link against another set of headers and libraries to be able to use the proxy class.<span>&nbsp;The burden of maintainability for Python bindings created in this way would be greater as well. This also isn't a viable approach for using any C++ class that overloads both operators.</span>
</div>
<div><span><br></span></div>
<div>Another option I have considered is allowing Cython's indexing operator to dispatch to a different function. Currently, user-defined cname entries for overloaded operators are not used. If this were changed for the indexing operator, indexing could be performed at the C++ level using some other method. This doesn't look like a viable approach though, since, for this to really work, users would need some way to call different methods when a C++ object is being indexed and when it is being assigned to. Using operator[] for left-hand values and operator() for right-hand values is a possible solution, but that isn't a very consistent interface. Doing this would also increase the complexity of the existing code for indexing in the Cython compiler and could lead to name collisions for classes that overload both operator[] and operator().<span><br></span>
</div>
<div><span><br></span></div>
<div><span>Are any of these acceptable ways to go forward? Does anyone have any better ideas? My preference would definitely be toward allowing C++ calls returning references to be used as lvalues, but I'd really appreciate any alternative solutions.</span></div>
<div><span><br></span></div>
<div><span>Thanks!</span></div>
<div><span>-Ian Henriksen</span></div>
<div><br></div>
<div><br></div>
</div>
</div></div>
Stephen LARROQUE | 3 Jul 03:16 2015
Picon

Re: [Cython] cython-devel Digest, Vol 53, Issue 9

Thank you very much Stefan.

I have just read the ticket, and it seems that Guido van Rossum consider this to be a feature, not a bug (???), so I guess they won't fix it.

For me it would be a nice plus to have this problem fixed, but I can live with my current workaround, so I'm not sure if Cython should spend dev time to fix that if that's expensive.

At least, I think the workaround should be mentioned in the github wiki so that others may implement finite fields extension types without MemoryError issues. Should I add a wiki page?

2015-06-19 12:00 GMT+02:00 <cython-devel-request-+ZN9ApsXKcEdnm+yROfE0A@public.gmane.org>:
Send cython-devel mailing list submissions to
        cython-devel-+ZN9ApsXKcEdnm+yROfE0A@public.gmane.org

To subscribe or unsubscribe via the World Wide Web, visit
        https://mail.python.org/mailman/listinfo/cython-devel
or, via email, send a message with subject or body 'help' to
        cython-devel-request-+ZN9ApsXKcEdnm+yROfE0A@public.gmane.org

You can reach the person managing the list at
        cython-devel-owner-+ZN9ApsXKcEdnm+yROfE0A@public.gmane.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of cython-devel digest..."


Today's Topics:

   1. Re: Bug: Extension Type inheriting from int cause a
      MemoryError (Stefan Behnel)


----------------------------------------------------------------------

Message: 1
Date: Fri, 19 Jun 2015 08:38:32 +0200
From: Stefan Behnel <stefan_ml <at> behnel.de>
To: cython-devel-+ZN9ApsXKcEdnm+yROfE0A@public.gmane.org
Subject: Re: [Cython] Bug: Extension Type inheriting from int cause a
        MemoryError
Message-ID: <5583B8E8.9050902-KjMAwuNBv5izQB+pC5nmwQ@public.gmane.org>
Content-Type: text/plain; charset=utf-8

Stephen LARROQUE schrieb am 15.06.2015 um 12:34:
> I am trying to make an extension type inheriting from int or cython.int (to
> do arithmetic operations in Galois Fields). However, this causes a
> MemoryError because it seems such extension type is not freed correctly.
> Other than that, it works perfectly well.
>
> cdef class ExtendedInt(int): pass
>
> for j in xrange(10000000):
>     ExtendedInt(j)

It looks like a bug in the "int" type in Python 2.x. Python 2.7 has this
code in intobject.c:

"""
static void
int_dealloc(PyIntObject *v)
{
    if (PyInt_CheckExact(v)) {
        Py_TYPE(v) = (struct _typeobject *)free_list;
        free_list = v;
    }
    else
        Py_TYPE(v)->tp_free((PyObject *)v);
}

static void
int_free(PyIntObject *v)
{
    Py_TYPE(v) = (struct _typeobject *)free_list;
    free_list = v;
}
"""

Your extension type automatically inherits the "int_free" slot function
from its base type, so the "else" case in "int_dealloc()" will in fact call
"int_free()" and append the object to the free list despite *not* being
exactly of type PyInt. Then, when creating a new ExtendedInt instance,
Python's int-subtype instantiation code *ignores* the free list and instead
creates a completely new object.

Thus, the free list keeps growing until it fills all memory and the process
dies. I created a CPython ticket.

https://bugs.python.org/issue24469

I guess we could hack up a work around for this in Cython somehow (Python
2.7 is an easy target being a dead end, after all), but let's wait what the
CPython devs have to say about this. Note, however, that any fix in a
future CPython 2.7.x release will not fix it in earlier Python 2.x
versions, so my guess is that we'd end up having to add a work-around on
our side anyway.

Stefan



------------------------------

Subject: Digest Footer

_______________________________________________
cython-devel mailing list
cython-devel-+ZN9ApsXKcEdnm+yROfE0A@public.gmane.org
https://mail.python.org/mailman/listinfo/cython-devel


------------------------------

End of cython-devel Digest, Vol 53, Issue 9
*******************************************

<div>
<div dir="ltr">Thank you very much Stefan.<div><br></div>
<div>I have just read the ticket, and it seems that Guido van Rossum consider this to be a feature, not a bug (???), so I guess they won't fix it.</div>
<div><br></div>
<div>For me it would be a nice plus to have this problem fixed, but I can live with my current workaround, so I'm not sure if Cython should spend dev time to fix that if that's expensive.</div>
<div><br></div>
<div>At least, I think the workaround should be mentioned in the github wiki so that others may implement finite fields extension types without MemoryError issues. Should I add a wiki page?</div>
</div>
<div class="gmail_extra">
<br><div class="gmail_quote">2015-06-19 12:00 GMT+02:00  <span dir="ltr">&lt;<a href="mailto:cython-devel-request@..." target="_blank">cython-devel-request@...</a>&gt;</span>:<br><blockquote class="gmail_quote">Send cython-devel mailing list submissions to<br>
&nbsp; &nbsp; &nbsp; &nbsp; <a href="mailto:cython-devel@...">cython-devel@...</a><br><br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
&nbsp; &nbsp; &nbsp; &nbsp; <a href="https://mail.python.org/mailman/listinfo/cython-devel" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/cython-devel</a><br>
or, via email, send a message with subject or body 'help' to<br>
&nbsp; &nbsp; &nbsp; &nbsp; <a href="mailto:cython-devel-request <at> python.org">cython-devel-request@...</a><br><br>
You can reach the person managing the list at<br>
&nbsp; &nbsp; &nbsp; &nbsp; <a href="mailto:cython-devel-owner@...">cython-devel-owner@...</a><br><br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of cython-devel digest..."<br><br><br>
Today's Topics:<br><br>
&nbsp; &nbsp;1. Re: Bug: Extension Type inheriting from int cause a<br>
&nbsp; &nbsp; &nbsp; MemoryError (Stefan Behnel)<br><br><br>
----------------------------------------------------------------------<br><br>
Message: 1<br>
Date: Fri, 19 Jun 2015 08:38:32 +0200<br>
From: Stefan Behnel &lt;<a href="mailto:stefan_ml@...">stefan_ml <at> behnel.de</a>&gt;<br>
To: <a href="mailto:cython-devel@...">cython-devel@...</a><br>
Subject: Re: [Cython] Bug: Extension Type inheriting from int cause a<br>
&nbsp; &nbsp; &nbsp; &nbsp; MemoryError<br>
Message-ID: &lt;<a href="mailto:5583B8E8.9050902@...">5583B8E8.9050902@...</a>&gt;<br>
Content-Type: text/plain; charset=utf-8<br><br>
Stephen LARROQUE schrieb am 15.06.2015 um 12:34:<br>
&gt; I am trying to make an extension type inheriting from int or <a href="http://cython.int" rel="noreferrer" target="_blank">cython.int</a> (to<br>
&gt; do arithmetic operations in Galois Fields). However, this causes a<br>
&gt; MemoryError because it seems such extension type is not freed correctly.<br>
&gt; Other than that, it works perfectly well.<br>
&gt;<br>
&gt; cdef class ExtendedInt(int): pass<br>
&gt;<br>
&gt; for j in xrange(10000000):<br>
&gt;&nbsp; &nbsp; &nbsp;ExtendedInt(j)<br><br>
It looks like a bug in the "int" type in Python 2.x. Python 2.7 has this<br>
code in intobject.c:<br><br>
"""<br>
static void<br>
int_dealloc(PyIntObject *v)<br>
{<br>
&nbsp; &nbsp; if (PyInt_CheckExact(v)) {<br>
&nbsp; &nbsp; &nbsp; &nbsp; Py_TYPE(v) = (struct _typeobject *)free_list;<br>
&nbsp; &nbsp; &nbsp; &nbsp; free_list = v;<br>
&nbsp; &nbsp; }<br>
&nbsp; &nbsp; else<br>
&nbsp; &nbsp; &nbsp; &nbsp; Py_TYPE(v)-&gt;tp_free((PyObject *)v);<br>
}<br><br>
static void<br>
int_free(PyIntObject *v)<br>
{<br>
&nbsp; &nbsp; Py_TYPE(v) = (struct _typeobject *)free_list;<br>
&nbsp; &nbsp; free_list = v;<br>
}<br>
"""<br><br>
Your extension type automatically inherits the "int_free" slot function<br>
from its base type, so the "else" case in "int_dealloc()" will in fact call<br>
"int_free()" and append the object to the free list despite *not* being<br>
exactly of type PyInt. Then, when creating a new ExtendedInt instance,<br>
Python's int-subtype instantiation code *ignores* the free list and instead<br>
creates a completely new object.<br><br>
Thus, the free list keeps growing until it fills all memory and the process<br>
dies. I created a CPython ticket.<br><br><a href="https://bugs.python.org/issue24469" rel="noreferrer" target="_blank">https://bugs.python.org/issue24469</a><br><br>
I guess we could hack up a work around for this in Cython somehow (Python<br>
2.7 is an easy target being a dead end, after all), but let's wait what the<br>
CPython devs have to say about this. Note, however, that any fix in a<br>
future CPython 2.7.x release will not fix it in earlier Python 2.x<br>
versions, so my guess is that we'd end up having to add a work-around on<br>
our side anyway.<br><br>
Stefan<br><br><br><br>
------------------------------<br><br>
Subject: Digest Footer<br><br>
_______________________________________________<br>
cython-devel mailing list<br><a href="mailto:cython-devel@...">cython-devel@...</a><br><a href="https://mail.python.org/mailman/listinfo/cython-devel" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/cython-devel</a><br><br><br>
------------------------------<br><br>
End of cython-devel Digest, Vol 53, Issue 9<br>
*******************************************<br>
</blockquote>
</div>
<br>
</div>
</div>
Robert McGibbon | 1 Jul 11:12 2015
Picon

[Cython] Use of long type for intermediate integral variables

Hi,

(First time poster. Apologies in advance if I'm not following the appropriate protocol, or if this has been posted already)

I noticed an issue on Windows when debugging an issue in scipy, but I think it might be a little more general.  In some places in the generated code, it looks like intermediate integral variables are declared with type long, even when long is too small to hold necessary value. For example, with the code pasted below, the value n+1 is stored in a variable of type long (using Cython 0.22.1) before being supplied to F.__getitem__.

This is especially pertinent on Windows (32 bit and 64 bit) and 32-bit linux, where longs are 32-bits, so you get an overflow for a program like the example below. The result is that it prints 1 instead of the expected value, 2**53+1 = 9007199254740993. But this same issue comes up basically whenever you do arithmetic on an array index in 64-bit Windows, for indices larger than 2**31-1, since sizeof(long) << sizeof(void*).

```
from libc.stdint cimport int64_t

class F(object):
    def __getitem__(self, i):
        print(i)

cdef int64_t n = 2**53
f = F()
f[n+1]
```
<div><div dir="ltr">Hi,<div><br></div>
<div>(First time poster. Apologies in advance if I'm not following the appropriate protocol, or if this has been posted already)</div>
<div><br></div>
<div>I noticed an issue on Windows when debugging an issue in <a href="https://github.com/scipy/scipy/issues/4907">scipy</a>, but I think it might be a little more general.&nbsp; In some places in the generated code, it looks like intermediate integral variables are declared with type long, even when long is too small to hold necessary value. For example, with the code pasted below, the value n+1 is stored in a variable of type long (using Cython 0.22.1) before being supplied to F.__getitem__.</div>
<div><br></div>
<div>This is especially pertinent on Windows (32 bit and 64 bit) and 32-bit linux, where longs are 32-bits, so you get an overflow for a program like the example below. The result is that it prints 1 instead of the expected value, 2**53+1 = 9007199254740993. But this same issue comes up basically whenever you do arithmetic on an array index in 64-bit Windows, for indices larger than 2**31-1, since sizeof(long) &lt;&lt; sizeof(void*).</div>
<div><br></div>
<div>```</div>
<div>
<div>from libc.stdint cimport int64_t</div>
<div><br></div>
<div>class F(object):</div>
<div>&nbsp; &nbsp; def __getitem__(self, i):</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; print(i)</div>
<div><br></div>
<div>cdef int64_t n = 2**53</div>
<div>f = F()</div>
<div>f[n+1]</div>
</div>
<div>```</div>
</div></div>
Michael Enßlin | 22 Jun 04:04 2015

[Cython] Feature Request: Variadic Templates

Hi,

I'd like to see support for variadic template arguments, such as this:

test.h:

template<typename RetT, typename ... VARARGS>
RetT foo(VARARGS ...);

test.pyx:

cdef extern from "test.h":
    cdef RetT foo[RetT, ... VARARGS](... VARARGS)

def test():
    cdef int i = foo[int, float, int](1, 2.5, 3)

This would allow Cython's libcpp to easily support many of C++11's new
types, including std::tuple and std::function.

Support for the latter in particular would prove quite useful in passing
around function pointers between Cython and C++.

I have tried to implement this feature myself, but I'm entirely
unfamiliar with Cython's codebase, and all my attempts ended in
unsatisfactorily hacky, unstable code.
I believe that any experienced Cython developer on this list would be
able to properly implement this in a matter of a few hours, as it seems
like a rather minor feature.

The current workaround for the non-existance of this feature involves
typedefs for every possible number of arguments, like this:

template<typename ... Ts>
struct S {};

using S0 = S<>;

template<typename T0>
using S1 = S<T0>;

template<typename T0, typename T1>
using S2 = S<T0, T1>;

template<typename T0, typename T1, typename T2>
using S3 = S<T0, T1, T2>;

then exporting all of S0, S1, ... individually in the .pxd file.

This is inconvenient, but acceptable.

Now assume that S has a member function f,

	template<typename ... Us>
	void f(Ts..., Us ...);

due to the nature of C++, the same trick as above does not work, and the
user will be forced to clutter the C++ code with a function body for
every possible len(Us). Even worse, the .pxd file will now contain a
quadratic number of wrappers (one for every possible combination of
len(Ts), len(Us)), all of them extremely prone to mistakes.

Thanks for reading,

  ~ Michael

Hi,

I'd like to see support for variadic template arguments, such as this:

test.h:

template<typename RetT, typename ... VARARGS>
RetT foo(VARARGS ...);

test.pyx:

cdef extern from "test.h":
    cdef RetT foo[RetT, ... VARARGS](... VARARGS)

def test():
    cdef int i = foo[int, float, int](1, 2.5, 3)

This would allow Cython's libcpp to easily support many of C++11's new
types, including std::tuple and std::function.

Support for the latter in particular would prove quite useful in passing
around function pointers between Cython and C++.

I have tried to implement this feature myself, but I'm entirely
unfamiliar with Cython's codebase, and all my attempts ended in
unsatisfactorily hacky, unstable code.
I believe that any experienced Cython developer on this list would be
able to properly implement this in a matter of a few hours, as it seems
like a rather minor feature.

The current workaround for the non-existance of this feature involves
typedefs for every possible number of arguments, like this:

template<typename ... Ts>
struct S {};

using S0 = S<>;

template<typename T0>
using S1 = S<T0>;

template<typename T0, typename T1>
using S2 = S<T0, T1>;

template<typename T0, typename T1, typename T2>
using S3 = S<T0, T1, T2>;

then exporting all of S0, S1, ... individually in the .pxd file.

This is inconvenient, but acceptable.

Now assume that S has a member function f,

	template<typename ... Us>
	void f(Ts..., Us ...);

due to the nature of C++, the same trick as above does not work, and the
user will be forced to clutter the C++ code with a function body for
every possible len(Us). Even worse, the .pxd file will now contain a
quadratic number of wrappers (one for every possible combination of
len(Ts), len(Us)), all of them extremely prone to mistakes.

Thanks for reading,

  ~ Michael


Gmane