Dominic Sacré | 19 Aug 00:02 2015
Picon
Picon

[Cython] Cython fails to build if pgen is available

Hi,

I'm getting the following error trying to build Cython 0.23:

Cython/Parser/Grammar: No such file or directory
Traceback (most recent call last):
  File "setup.py", line 289, in <module>
    compile_cython_modules(cython_profile, cython_compile_more,
cython_with_refnanny)
  File "setup.py", line 135, in compile_cython_modules
    os.path.join(parser_dir, 'graminit.c'),
  File
"/data/master/build/tmp/sysroots/x86_64-linux/usr/lib/python2.7/subprocess.py",
line 540, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command
'['/data/master/build/tmp/sysroots/x86_64-linux/usr/bin/python-native/pgen',
'Cython/Parser/Grammar', 'Cython/Parser/graminit.h',
'Cython/Parser/graminit.c']' returned non-zero exit status 1

The toolchain is based on OpenEmbedded, and unlike most distribution
packages, its Python installation actually contains pgen.

Where would that Cython/Parser/Grammar file come from?
Alternatively, is there a clean way (without patching setup.py) to skip
compiling the grammar?

Cheers,

Dominic
(Continue reading)

Thomas Holenstein | 17 Aug 11:29 2015
Picon

[Cython] gcc error when compiling cython

Hey all,

recently, I had a gcc compile time error with cython. The work around
is obvious, as the problem happens when a cdef-ed numpy array in a
function is unused. Nevertheless, I feel it's useful to report the
problem I had.

The following is a test case
---------------------------------------- test.pyx
import numpy as np
cimport numpy as np

def f(np.ndarray[dtype=np.int_t, ndim=2] arg1):
    cdef np.ndarray[dtype=np.int_t, ndim=2] arr
    return 3
----------------------------------------- setup.py
from distutils.core import setup
from Cython.Build import cythonize

setup(
    ext_modules = cythonize("test.pyx"),
)
----------------------------------------------------------
This gives me the error

error: ‘__pyx_pybuffernd_arr’ undeclared

Full shell output:
 ~/tmp/ctest > python setup.py build_ext --inplace
running build_ext
(Continue reading)

Jakub Wilk | 14 Aug 10:41 2015
Picon

[Cython] Cython 0.23: long + int == int

$ cython --version
Cython version 0.23
$ python2.6 --version
Python 2.6.8
$ cat testlong.pyx
def f(x):
        return 0L + x
$ python2.6 -c 'import pyximport as p; p.install(); import testlong; print type(testlong.f(42))'
<type 'int'>

I added a long to an int, so I'd expect the result to be a long.
This works correctly in Python 2.7:

$ python2.7 -c 'import pyximport as p; p.install(); import testlong; print type(testlong.f(42))'
<type 'long'>

It also worked correctly in Cython 0.22.1 + Python 2.6.

--

-- 
Jakub Wilk
Phil Thompson | 12 Aug 19:34 2015

[Cython] Cython Modules as builtins

Is there any reason why cython generated modules don’t use the fully qualified name (ie.
__Pyx_MODULE_NAME) in the call to Py_InitModule4() and the PyModuleDef structure? I want to statically
link a module and add it to Python’s inittab but things don’t work without a fully qualified name.

Thanks,
Phil
_______________________________________________
cython-devel mailing list
cython-devel <at> python.org
https://mail.python.org/mailman/listinfo/cython-devel
Ian Henriksen | 20 Jul 23:06 2015
Picon

[Cython] Non-type template parameters

Hi all,
I've spent a little time working on adding support for non-type
template parameters. In doing this, it has been very easy to add
support for doing something like the following:

cdef extern from "add_const.hpp" nogil:
    int myfunc1[i](int)
def test():
    print myfunc1[2](a)

The downside of this is that it does not let the Cython compiler
distinguish between different kinds of template parameters.

Stricter checking could be added using a syntax like this:

cdef extern from "add_const.hpp" nogil:
    int myfunc1[int i](int)
def test():
    print myfunc1[2](a)

The downsides here are that the syntax doesn't really match the
existing template syntax. It will also complicate the Cython codebase
since we'll have to go to greater lengths to allow or disallow all the
different special cases for templates.

Which version would be preferable?

On a similar note, for variadic templates, would we prefer something
like

cdef extern from "my_variadic.hpp" nogil:
    T myfunc2[T,...](T, ...)

or something like:

cdef extern from "my_variadic.hpp" nogil:
    T myfunc2[T, Types...](T, Types... args)

Again, the latter syntax is more explicit, but it will require much more
complicated code in Cython. It also doesn't match the existing syntax
very well. The former syntax matches the existing syntax for templates
better, but will make it hard for Cython to raise errors early on in
compilation.

I'd greatly appreciate any input on the best syntax for either use-case.

Regards,

-Ian Henriksen
<div><div dir="ltr">Hi all,<div>I've spent a little time working on adding support for non-type</div>
<div>template parameters. In doing this, it has been very easy to add</div>
<div>support for doing something like the following:</div>
<div><br></div>
<div>
<div>cdef extern from "add_const.hpp" nogil:</div>
<div>&nbsp; &nbsp; int myfunc1[i](int)</div>
</div>
<div>def test():</div>
<div>&nbsp; &nbsp;&nbsp;print myfunc1[2](a)</div>
<div><br></div>
<div>The downside of this is that it does not let the Cython compiler</div>
<div>distinguish between different kinds of template parameters.</div>
<div><br></div>
<div>Stricter checking could be added using a syntax like this:</div>
<div><br></div>
<div>
<div>
<div>cdef extern from "add_const.hpp" nogil:</div>
<div>&nbsp; &nbsp; int myfunc1[int i](int)</div>
</div>
<div>def test():</div>
<div>&nbsp; &nbsp;&nbsp;print myfunc1[2](a)</div>
</div>
<div><br></div>
<div>The downsides here are that the syntax doesn't really match the</div>
<div>existing template syntax. It will also complicate the Cython codebase</div>
<div>since we'll have to go to greater lengths to allow or disallow all the</div>
<div>different special cases for templates.</div>
<div><br></div>
<div>Which version would be preferable?</div>
<div><br></div>
<div>On a similar note, for variadic templates, would we prefer something</div>
<div>like</div>
<div><br></div>
<div>cdef extern from "my_variadic.hpp" nogil:</div>
<div>&nbsp; &nbsp; T myfunc2[T,...](T, ...)</div>
<div><br></div>
<div>or something like:</div>
<div><br></div>
<div>cdef extern from "my_variadic.hpp" nogil:</div>
<div>&nbsp; &nbsp; T myfunc2[T, Types...](T, Types... args)</div>
<div><br></div>
<div>Again, the latter syntax is more explicit, but it will require much more</div>
<div>complicated code in Cython. It also doesn't match the existing syntax</div>
<div>very well. The former syntax matches the existing syntax for templates</div>
<div>better, but will make it hard for Cython to raise errors early on in</div>
<div>compilation.</div>
<div><br></div>
<div>I'd greatly appreciate any input on the best syntax for either use-case.</div>
<div><br></div>
<div>Regards,</div>
<div><br></div>
<div>-Ian Henriksen</div>
</div></div>
Brian McAndrews | 16 Jul 23:13 2015

[Cython] Problem with declaring a map with an enum type.

I have a C enum type that I’m attempting to put in a C++ map:

Here’s what I got:

 

colors.h:

enum Color

  {

    GREEN,BLUE,RED

  };

 

val.pyx:

from libcpp.string cimport string

from libcpp.map cimport map

 

cdef extern from "colors.h":

    cpdef enum _Color "Color":

       GREEN,BLUE,RED

 

cdef class TestMe:

    cdef public map[string, _Color] vals

 

 

I get a compile error (VSC++ 2012)  

VAL.cpp(1239) : error C2440: '=' : cannot convert from 'long' to 'Color'

        Conversion to enumeration type requires an explicit cast (static_cast, C-style cast or function-style cast)

 

The offending statement is this:

    __pyx_t_9 = PyInt_AsLong(__pyx_v_value); if (unlikely(PyErr_Occurred())) {__pyx_filename = __pyx_f[1]; __pyx_lineno = 202; __pyx_clineno = __LINE__; goto __pyx_L1_error;}

 

In the __pyx_convert_map_from_py_std_3a__3a_string__and_enum__Color function.

 

The cast from PyInt_AsLong(…) to an enum Color is not specified when using maps.

 

If I don’t put it in a map (just as a member variable), the cast is performed.

 

Has anyone seen this before?

 

Thanks,

Brian







This electronic mail message and any attached files contain information intended for the exclusive use of the individual or entity to whom it is addressed and may contain information that is proprietary, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any viewing, copying, disclosure or distribution of this information may be subject to legal restriction or sanction. Please notify the sender, by electronic mail or telephone, of any unintended recipients and delete the original message without making any copies.
<div>
<div class="WordSection1">
<p class="MsoNormal">I have a C enum type that I&rsquo;m attempting to put in a C++ map:<p></p></p>
<p class="MsoNormal">Here&rsquo;s what I got:<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">colors.h:<p></p></p>
<p class="MsoNormal">enum Color<p></p></p>
<p class="MsoNormal">&nbsp; {<p></p></p>
<p class="MsoNormal">&nbsp;&nbsp;&nbsp; GREEN,BLUE,RED<p></p></p>
<p class="MsoNormal">&nbsp; };<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">val.pyx:<p></p></p>
<p class="MsoNormal">from libcpp.string cimport string<p></p></p>
<p class="MsoNormal">from libcpp.map cimport map<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">cdef extern from "colors.h":<p></p></p>
<p class="MsoNormal">&nbsp;&nbsp;&nbsp; cpdef enum _Color "Color":<p></p></p>
<p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; GREEN,BLUE,RED<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">cdef class TestMe:<p></p></p>
<p class="MsoNormal">&nbsp;&nbsp;&nbsp; cdef public map[string, _Color] vals<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">I get a compile error (VSC++ 2012) &nbsp;<p></p></p>
<p class="MsoNormal">VAL.cpp(1239) : error C2440: '=' : cannot convert from 'long' to 'Color'<p></p></p>
<p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Conversion to enumeration type requires an explicit cast (static_cast, C-style cast or function-style cast)<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">The offending statement is this:<p></p></p>
<p class="MsoNormal">&nbsp;&nbsp;&nbsp; __pyx_t_9 = PyInt_AsLong(__pyx_v_value); if (unlikely(PyErr_Occurred())) {__pyx_filename = __pyx_f[1]; __pyx_lineno = 202; __pyx_clineno = __LINE__; goto __pyx_L1_error;}<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">In the __pyx_convert_map_from_py_std_3a__3a_string__and_enum__Color function.<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">The cast from PyInt_AsLong(&hellip;) to an enum Color is not specified when using maps.<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">If I don&rsquo;t put it in a map (just as a member variable), the cast is performed.<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">Has anyone seen this before?<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">Thanks,<p></p></p>
<p class="MsoNormal">Brian<p></p></p>
</div>
<br><br><br><br><br><br><h5>This electronic mail message and any attached files contain information intended for the exclusive use of the individual or entity to whom it is addressed and may contain information that is proprietary, confidential and/or exempt from
 disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any viewing, copying, disclosure or distribution of this information may be subject to legal restriction or sanction. Please notify the sender, by electronic
 mail or telephone, of any unintended recipients and delete the original message without making any copies.
</h5>

</div>
Alex S. | 15 Jul 13:51 2015
Picon

[Cython] Fwd: [BUG] Cython failing performance when Numpy is not installed


Hello,
being unable to acquire the Trac account (Robert Bradshaw's email is not listed on his site), I write here.
The issue was described here: http://trac.cython.org/ticket/847 but was, however, misidentified. The wrapper for a function which receives "fused slice" (D[:]) arguments automatically includes the following code:
>                 cdef type ndarray
>                 try:
>                     import numpy
>                     ndarray = numpy.ndarray
>                 except (ImportError, AttributeError, TypeError):
>                     ndarray = None
(Compiler/FusedNode.py:468).
When numpy is not installed, each import tries to search for it in the system path. (When it is, it's just cached). This severely degrades the performance.


<div><div dir="ltr">
<br><div class="gmail_quote">
<div dir="ltr">
<div>
<div>Hello,<br>being unable to acquire the Trac account (Robert Bradshaw's email is not listed on his site), I write here.<br>The issue was described here: <a href="http://trac.cython.org/ticket/847" target="_blank">http://trac.cython.org/ticket/847</a> but was, however, misidentified. The wrapper for a function which receives "fused slice" (D[:]) arguments automatically includes the following code:<br>&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; cdef type ndarray<br>&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; try:<br>&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; import numpy<br>&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ndarray = numpy.ndarray<br>&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; except (ImportError, AttributeError, TypeError):<br>&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ndarray = None<br>
</div>(Compiler/FusedNode.py:468).<br>
</div>When numpy is not installed, each import tries to search for it in the system path. (When it is, it's just cached). This severely degrades the performance.<br>
</div>
<br>
</div>
<br>
</div></div>
Ian Henriksen | 15 Jul 01:32 2015
Picon

[Cython] Handling of cascaded assignment

I came across this while working on allowing proper overloading of the
assignment operator.
The problem is, essentially, that Cython does not follow the C/C++
ordering for cascaded assignment, even when dealing with objects at
the C/C++ level. For example, calling the test function from the
following Cython file and running the following C++ file should
both give the same output:

def test():
    cdef:
        long long a = (2LL << 36) - 257
        int b
        char c
    b = c = a
    print a, b, c



#include <iostream>
int main(){
    long long a = (2LL << 36) - 257;
    int b;
    char c;
    b = c = a;
    std::cout << a << std::endl;
    std::cout << b << std::endl;
    std::cout << (int)c << std::endl;
}

The Cython version prints 137438953215 -257 -1
and the C++ version prints 137438953215 -1 -1

To mimic C/C++, it should go through the arguments from right to left
and perform each assignment operation in that order, i.e.
c = a
b = c
As it stands, the current code does:
b = a
c = a

This does appear to follow what Python does. For example, the
following gives the same output as the Cython version:

import numpy as np
a = np.array([(2 << 36) - 257], np.int64)
b = np.empty(1, np.int32)
c = np.empty(1, np.int8)
b[:] = c[:] = a
print a[0], b[0], c[0]

Is this a deliberate design decision? If this weren't already included in
the language, I'd be inclined to not allow cascaded assignment to C
objects at all since it can give such surprising results. On the other
hand I don't see any particularly clean way to deal with this
incompatibility between C/C++ and Python.

Thanks!

-Ian Henriksen
<div><div dir="ltr">
<span>I came across this while working on allowing proper overloading of the</span><div>
<span>assignment operator.</span><div>The problem is, essentially, that Cython does not follow the C/C++</div>
<div>ordering for cascaded assignment, even when dealing with objects at</div>
<div>the C/C++ level. For example, calling the test function from the</div>
<div>following Cython file and running the following C++ file should</div>
<div>both give the same output:</div>
<div><br></div>
<div>
<div>def test():</div>
<div>&nbsp; &nbsp; cdef:</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; long long a = (2LL &lt;&lt; 36) - 257</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; int b</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; char c</div>
<div>&nbsp; &nbsp; b = c = a</div>
<div>&nbsp; &nbsp; print a, b, c</div>
</div>
<div><br></div>
<div><br></div>
<div><br></div>
<div>
<div>#include &lt;iostream&gt;</div>
<div>int main(){</div>
<div>&nbsp; &nbsp; long long a = (2LL &lt;&lt; 36) - 257;</div>
<div>&nbsp; &nbsp; int b;</div>
<div>&nbsp; &nbsp; char c;</div>
<div>&nbsp; &nbsp; b = c = a;</div>
<div>&nbsp; &nbsp; std::cout &lt;&lt; a &lt;&lt; std::endl;</div>
<div>&nbsp; &nbsp; std::cout &lt;&lt; b &lt;&lt; std::endl;</div>
<div>&nbsp; &nbsp; std::cout &lt;&lt; (int)c &lt;&lt; std::endl;</div>
<div>}</div>
</div>
<div><br></div>
<div>The Cython version prints&nbsp;137438953215 -257 -1</div>
<div>and the C++ version prints&nbsp;137438953215 -1 -1</div>
<div><br></div>
<div>To mimic C/C++, it should go through the arguments from right to left</div>
<div>and perform each assignment operation in that order, i.e.</div>
<div>c = a</div>
<div>b = c</div>
<div>As it stands, the current code does:</div>
<div>b = a</div>
<div>c = a</div>
</div>
<div><br></div>
<div>This does appear to follow what Python does. For example, the</div>
<div>following gives the same output as the Cython version:</div>
<div><br></div>
<div>import numpy as np</div>
<div>a = np.array([(2 &lt;&lt; 36) - 257], np.int64)</div>
<div>b = np.empty(1, np.int32)</div>
<div>c = np.empty(1, np.int8)</div>
<div>b[:] = c[:] = a</div>
<div>print a[0], b[0], c[0]</div>
<div><br></div>
<div>Is this a deliberate design decision? If this weren't already included in</div>
<div>the language, I'd be inclined to not allow cascaded assignment to C</div>
<div>objects at all since it can give such surprising results. On the other</div>
<div>hand I don't see any particularly clean way to deal with this</div>
<div>incompatibility between C/C++ and Python.</div>
<div><br></div>
<div>Thanks!</div>
<div><br></div>
<div>-Ian Henriksen</div>
</div></div>
Mark Florisson | 14 Jul 20:34 2015
Picon

[Cython] overflow bug?

Hi,

I think this might be a bug (python 3.3 + cython 0.22.1):

    def f(term=b"12345"):
val = int('987278186585')
# The below line does not work, because it treats 1 as a constant integer
# in the C code (32 bit on my machine). Using 1L does work however.
val -= 1 << (len(term) * 8)
return val
 
print(f())

This works in pure-python, but Cython generates '1 << __pyx_t_somevar', which I think treats the '1' as an integer (causing it to overflow). Using '1L' works in the Cython code however (but that may be just my platform).
Cheers,Mark
<div><div dir="ltr">Hi,<div><br></div>
<div>I think this might be a bug (python 3.3&nbsp;+ cython 0.22.1):</div>
<div><br></div>
<div>&nbsp; &nbsp;&nbsp;<span>def</span><span> </span><span>f</span><span>(</span><span>term</span><span>=</span><span>b</span><span><span>"</span>12345<span>"</span></span><span>):</span>
</div>
<div>    val <span>=</span> <span>int</span>(<span><span>'</span>987278186585<span>'</span></span>)
</div>
<div>    <span># The below line does not work, because it treats 1 as a constant integer</span>
</div>
<div>    <span># in the C code (32 bit on my machine). Using 1L does work however.</span>
</div>
<div>    val <span>-=</span> <span>1</span> <span>&lt;&lt;</span> (<span>len</span>(term) <span>*</span> <span>8</span>)
</div>
<div>    <span>return</span> val 
</div>
<div>&nbsp;
</div>
<div>
<span>print</span>(f())</div>
<div><br></div>This works in pure-python, but Cython generates '1 &lt;&lt; __pyx_t_somevar', which I think treats the '1' as an integer (causing it to overflow). Using '1L' works in the Cython code however (but that may be just my platform).<br>Cheers,Mark </div></div>
Jeroen Demeyer | 13 Jul 15:56 2015
Picon

[Cython] Regression: cdef append() no longer works

Code like the following no longer works correctly:

cdef class Foo:
     def __init__(self):
         self.append(1)

     cdef append(self, x):
         return x

It seems that Cython now always assumes that append() is a special 
method: it incorrectly uses __Pyx_PyObject_Append.

PS: it's not really clear if bug reports like these should go on this 
mailing list or on Trac (my impression is that Cython's Trac is not 
really used). What's the recommended procedure?
Josh Ayers | 7 Jul 17:11 2015

[Cython] PyUnicode_Tailmatch

Cython devs,

In the function __Pyx_PyUnicode_Tailmatch, the return type of
PyUnicode_Tailmatch is assumed to be in int.  See line 543 of
Cython/Utility/StringTools.c.  

PyUnicode_Tailmatch actually returns a Py_ssize_t.  See:
https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_Tailmatch.

For reference, there was previously an error in the Python documentation
of this function.  The documentation mistakenly said it returned an int.
 This was fixed via Python issue 22580:
https://bugs.python.org/issue22580.

Thanks,
Josh Ayers

Gmane