Richard Hansen | 27 Jan 23:38 2015
Picon

[Cython] bug in typed memoryviews (memoryviewslice)

Hi all,

If I create foo.pyx with the following contents:

cpdef object foo():
    cdef unsigned char[:] s = <bytearray><unsigned char *>"012345"
    return s

I notice the following behavior:

$ python -c '
import foo
m=foo.foo()
print repr(memoryview(m.base).tobytes())
print repr(memoryview(m).tobytes())
'
'012345'
'123450'

Notice how the bytes are printed in the wrong order unless I use the
original buffer object via the typed memoryview object's base attribute.

-Richard
Apps Embedded | 27 Jan 15:12 2015
Picon

[Cython] IPython with Cython support on Android

Hi,

We have developped an android app which makes it possible to use the IPython shell through Android.

https://play.google.com/store/apps/details?id=com.appsopensource.labpy
https://play.google.com/store/apps/details?id=com.appsopensource.labpypremium

We decided then to add the Cython extension to speed up script execution through the notebook for instance.

Our app bundle is covered by the Gnu GPL v3 licence.
Cython is covered by the Apache Licence version 2.0 which is compatible with the Gnu GPL v3 licence.

Our question concerns the Pyrex licence on which Cython is based. It is on OSI approved licence.

From your point of view, is the Pyrex licence GPL v3 compatible ?

Thanks for your help.

Apps Embedded team.
<div><div dir="ltr">
<div>
<div>
<div>
<div>Hi,<br><br>We have developped an android app which makes it possible to use the IPython shell through Android.<br><br><a href="https://play.google.com/store/apps/details?id=com.appsopensource.labpy">https://play.google.com/store/apps/details?id=com.appsopensource.labpy</a><br><a href="https://play.google.com/store/apps/details?id=com.appsopensource.labpypremium">https://play.google.com/store/apps/details?id=com.appsopensource.labpypremium</a><br><br>
</div>We decided then to add the Cython extension to speed up script execution through the notebook for instance.<br><br>
</div>Our app bundle is covered by the Gnu GPL v3 licence.<br>
</div>Cython is covered by the Apache Licence version 2.0 which is compatible with the Gnu GPL v3 licence.<br><br>Our question concerns the Pyrex licence on which Cython is based.  It is on OSI approved licence. <br><br>From your point of view, is the Pyrex licence GPL v3 compatible ?<br><br>
</div>Thanks for your help.<br><br>Apps Embedded team.<br>
</div></div>
Favian Contreras | 22 Jan 05:14 2015
Picon

[Cython] Variable Step Value

Hey all,

I am having some trouble figuring out how to pass in an extra IfStatNode in Optimize.py, to the code generation stage (see 1st commented block). I am also having trouble getting the temp variable of the iteration variable, so that I can adjust the temp to be the value that would be in index (see the 2nd comment block), before the iteration variable gets set.

As of now I get the correct number of steps, as long as the step value is not 0. I've tried making these fields "StatListNode"s, but that gives a messy error. Any suggestions for how to handle these 2 problems?


Thanks,
Favian Contreras
<div><div dir="ltr">
<div>
<div>
<div>
<div>Hey all,<br><br>
</div>I am having some trouble figuring out how to pass in an extra IfStatNode in Optimize.py, to the code generation stage (see 1st commented block). I am also having trouble getting the temp variable of the iteration variable, so that I can adjust the temp to be the value that would be in index (see the 2nd comment block), before the iteration variable gets set.<br><br>
</div>As of now I get the correct number of steps, as long as the step value is not 0. I've tried making these fields "StatListNode"s, but that gives a messy error. Any suggestions for how to handle these 2 problems?<br><br>
</div>
<div>commit: <a href="https://github.com/BigFav/cython/commit/015440543b051168cb34441a6a8494b6fa5bdd7c">https://github.com/BigFav/cython/commit/015440543b051168cb34441a6a8494b6fa5bdd7c</a><br>
</div>
<div><br></div>Thanks,<br>
</div>Favian Contreras<br>
</div></div>
Robert Bradshaw | 18 Jan 00:10 2015
Picon

[Cython] Cython 0.22.alpha

It's been too long since we've done a release, and the features have
started to pile up, so I'm getting the ball rolling with 0.22.alpha0.
I've uploaded a tarball to
http://cython.org/release/Cython-0.22.alpha0.tar.gz; you can also get
it from github.

Let me know what breaks/works for you, and if there's anything you
think should get in. If feedback is good I'll be cutting a beta (to
the users list) next week.

- Robert
Stefan Behnel | 20 Dec 10:14 2014
Picon

[Cython] GvR on type hinting in Python

Looks like type annotations in Python are about to leave the "maybe one
day" status:

http://thread.gmane.org/gmane.comp.python.ideas/30432

Stefan
Kevin Norris | 20 Dec 01:47 2014
Picon

[Cython] Cythonize silently ignores nonexistent files

Is this behavior intentional?

    >>> from Cython.Build import cythonize
    >>> cythonize('/this/file/doesnt/exist.pyx')
    []

It would be *really* nice if Cython would raise an exception (or
something) in this case.  I just spent 20 minutes trying to puzzle out
Cython/Setuptools compatibility when the actual problem was staring me
in the face.

--
Kevin Norris
Ben Booth | 12 Dec 23:14 2014
Picon

[Cython] cygdb bugs when debugging python 3 extensions

I'm trying to figure out how to use cygdb to debug extensions built with python 3. I've tested all the debugger commands listed in http://docs.cython.org/src/userguide/debugging.html when debugging python 2 extensions and everything works great. But some of the commands seem to be broken when debugging python 3 extensions.

Here are the debugger commands that I've confirmed as working with python 3 extensions:

cy break
cy step
cy next
cy run
cy cont
cy down
cy finish .. works but prints error message
cy exec
cy list
cy_lineno()
cy_eval()
cy_cname()

Here are the commands which fail under python 3, and the error message that is given for each:

cy up
    Python Exception <class 'gdb.error'> There is no member named length.:
    Error occurred in Python command: There is no member named length.
cy finish .. works but gives error
    Python Exception <class 'gdb.error'> There is no member named length.:
    Error occurred in Python command: There is no member named length.
cy bt
    Python Exception <class 'gdb.error'> There is no member named length.:
    Error occurred in Python command: There is no member named length.
cy print
    Python Exception <class 'gdb.error'> There is no member named ma_mask.:
    Python Exception <class 'gdb.error'> Error occurred in Python command: There is no member named ma_mask.:
    Error occurred in Python command: Error occurred in Python command: There is no member named ma_mask.
cy set x = 2
    Python Exception <class 'gdb.error'> No symbol "__pyx_v_10helloworld_x" in current context.:
    Error occurred in Python command: No symbol "__pyx_v_10helloworld_x" in current context.
cy locals
    Python Exception <class 'gdb.error'> There is no member named ma_mask.:
    Error occurred in Python command: There is no member named ma_mask.
cy globals
    Python Exception <class 'gdb.error'> There is no member named ma_mask.:
    Error occurred in Python command: There is no member named ma_mask.
cy_cvalue()
    Python Exception <class 'gdb.error'> There is no member named ma_mask.:
    Error occurred in Python convenience function: There is no member named ma_mask.

In summary:
  - The "cy set" and "cy_cname()" commands seem to be giving incorrect C names for cython variables. I did a grep of the generated .c source file for "__pyx_v_10helloworld_x", and didn't get any hits. Luckily, the "cy exec" command works, so I can use that to get the value of variables as a workaround. 
  - Any commands related to viewing or traversing the call stack seem to give an error related to a missing "length" member. 
  - Commands related to printing the value of a variable complain about a missing "ma_mask" member. 

Here are the source files I used for the test:

helloworld.py:
import helloworld

helloworld.pyx:
x = 1
y = "a"
print("Hello World 1")
print("Hello World 2")
print("Hello World 3")
print("Hello World 4")
print("Hello World 5")
print("Hello World 6")
print("Hello World 8")
print("Hello World 9")
print("Hello World 10")

setup.py:
from distutils.core import setup
from Cython.Build import cythonize
from distutils.extension import Extension
extensions = [Extension("helloworld", ["helloworld.pyx"])]
setup(ext_modules=cythonize(extensions, gdb_debug=True, output_dir="."))

To compile I ran:
python setup.py build_ext --inplace

To debug, I ran:
cygdb -vv . -- --args python helloworld.py

Here is my python version:
Python 3.4.2

I used pyenv (https://github.com/yyuu/pyenv) to build and install the latest Python 3 version with debugging symbols enabled.

Ben


<div>I'm trying to figure out how to use cygdb to debug extensions built with python 3. I've tested all the debugger commands listed in&nbsp;<a href="http://docs.cython.org/src/userguide/debugging.html" class="">http://docs.cython.org/src/userguide/debugging.html</a>&nbsp;when debugging python 2 extensions and everything works great. But some of the commands seem to be broken when debugging python 3 extensions.<div class=""><br class=""></div>
<div class="">Here are the debugger commands that I've confirmed as working with python 3 extensions:</div>
<div class=""><br class=""></div>
<div class="">
<div class="">cy break</div>
<div class="">cy step</div>
<div class="">cy next</div>
<div class="">cy run</div>
<div class="">cy cont</div>
<div class="">cy down</div>
<div class="">cy finish .. works but prints error message</div>
<div class="">cy exec</div>
<div class="">cy list</div>
<div class="">cy_lineno()</div>
<div class="">cy_eval()</div>
<div class="">cy_cname()</div>
</div>
<div class=""><br class=""></div>
<div class="">Here are the commands which fail under python 3, and the error message that is given for each:</div>
<div class=""><br class=""></div>
<div class="">
<div class="">cy up</div>
<div class="">&nbsp; &nbsp; Python Exception &lt;class 'gdb.error'&gt; There is no member named length.:</div>
<div class="">&nbsp; &nbsp; Error occurred in Python command: There is no member named length.</div>
<div class="">cy finish .. works but gives error</div>
<div class="">&nbsp; &nbsp; Python Exception &lt;class 'gdb.error'&gt; There is no member named length.:</div>
<div class="">&nbsp; &nbsp; Error occurred in Python command: There is no member named length.</div>
<div class="">cy bt</div>
<div class="">&nbsp; &nbsp; Python Exception &lt;class 'gdb.error'&gt; There is no member named length.:</div>
<div class="">&nbsp; &nbsp; Error occurred in Python command: There is no member named length.</div>
<div class="">cy print</div>
<div class="">&nbsp; &nbsp; Python Exception &lt;class 'gdb.error'&gt; There is no member named ma_mask.:</div>
<div class="">&nbsp; &nbsp; Python Exception &lt;class 'gdb.error'&gt; Error occurred in Python command: There is no member named ma_mask.:</div>
<div class="">&nbsp; &nbsp; Error occurred in Python command: Error occurred in Python command: There is no member named ma_mask.</div>
<div class="">cy set x = 2</div>
<div class="">&nbsp; &nbsp; Python Exception &lt;class 'gdb.error'&gt; No symbol "__pyx_v_10helloworld_x" in current context.:</div>
<div class="">&nbsp; &nbsp; Error occurred in Python command: No symbol "__pyx_v_10helloworld_x" in current context.</div>
<div class="">cy locals</div>
<div class="">&nbsp; &nbsp; Python Exception &lt;class 'gdb.error'&gt; There is no member named ma_mask.:</div>
<div class="">&nbsp; &nbsp; Error occurred in Python command: There is no member named ma_mask.</div>
<div class="">cy globals</div>
<div class="">&nbsp; &nbsp; Python Exception &lt;class 'gdb.error'&gt; There is no member named ma_mask.:</div>
<div class="">&nbsp; &nbsp; Error occurred in Python command: There is no member named ma_mask.</div>
<div class="">cy_cvalue()</div>
<div class="">&nbsp; &nbsp; Python Exception &lt;class 'gdb.error'&gt; There is no member named ma_mask.:</div>
<div class="">&nbsp; &nbsp; Error occurred in Python convenience function: There is no member named ma_mask.</div>
</div>
<div class=""><br class=""></div>
<div class="">In summary:</div>
<div class="">&nbsp; - The "cy set" and "cy_cname()" commands seem to be giving incorrect C names for cython variables. I did a grep of the generated .c source file for "__pyx_v_10helloworld_x", and didn't get any hits. Luckily, the "cy exec" command works, so I can use that to get the value of variables as a workaround.&nbsp;</div>
<div class="">&nbsp; - Any commands related to viewing or traversing the call stack seem to give an error related to a missing "length" member.&nbsp;</div>
<div class="">&nbsp; - Commands related to printing the value of a variable complain about a missing "ma_mask" member.&nbsp;</div>
<div class=""><br class=""></div>
<div class="">Here are the source files I used for the test:</div>
<div class=""><br class=""></div>
<div class="">helloworld.py:</div>
<div class="">import helloworld</div>
<div class=""><br class=""></div>
<div class="">helloworld.pyx:</div>
<div class="">
<div class="">x = 1</div>
<div class="">y = "a"</div>
<div class="">print("Hello World 1")</div>
<div class="">print("Hello World 2")</div>
<div class="">print("Hello World 3")</div>
<div class="">print("Hello World 4")</div>
<div class="">print("Hello World 5")</div>
<div class="">print("Hello World 6")</div>
<div class="">print("Hello World 8")</div>
<div class="">print("Hello World 9")</div>
<div class="">print("Hello World 10")</div>
</div>
<div class=""><br class=""></div>
<div class="">setup.py:</div>
<div class="">
<div class="">from distutils.core import setup</div>
<div class="">from Cython.Build import cythonize</div>
<div class="">from distutils.extension import Extension</div>
<div class="">extensions = [Extension("helloworld", ["helloworld.pyx"])]</div>
<div class="">setup(ext_modules=cythonize(extensions, gdb_debug=True, output_dir="."))</div>
</div>
<div class=""><br class=""></div>
<div class="">To compile I ran:</div>
<div class="">python setup.py build_ext --inplace</div>
<div class=""><br class=""></div>
<div class="">To debug, I ran:</div>
<div class="">cygdb -vv . -- --args python helloworld.py</div>
<div class=""><br class=""></div>
<div class="">Here is my python version:</div>
<div class="">Python 3.4.2</div>
<div class=""><br class=""></div>
<div class="">I used pyenv (<a href="https://github.com/yyuu/pyenv" class="">https://github.com/yyuu/pyenv</a>)&nbsp;to build and install the latest Python 3 version with debugging symbols enabled.</div>
<div class=""><br class=""></div>
<div class="">Ben</div>
<div class=""><br class=""></div>
<div class=""><br class=""></div>
</div>
serge Guelton | 8 Dec 20:52 2014
Picon
Picon

[Cython] Cython <> pythran collaboration

Hi Cython dev,

I am one of the two core developers of the Pythran[0] compiler.
We are planning to start a project where Cython would delegate the
generation of optimized (i.e. parallel / vectorized / without temporary
arrays) numpy expression when it finds one, instead of calling the
native API. Something like the following meaningless example:

    import numpy as np
    cimport numpy as np

    def play_with_arrays(np.ndarray arr):
        cdef int N = arr.shape[0]
        cdef np.ndarray other_arr = np.empty([N], dtype=np.int)

        other_arr = arr ** 2 + 1 # pythran would handle this

        for i in range(N):
            other_arr[i] /=  2 # cython would handle this

        return  other_arr

Everything would live in a pyx file and it should be transparent to a
cython user. Cython would drive the compilation process, using pythran
as a numpy expression optimizer.

Does that look feasible / appealing to you?

[0] http://pythonhosted.org/pythran/
Dave Hirschfeld | 8 Dec 13:22 2014
Picon

[Cython] Can't assign numpy array to memoryview

It appears that you can't assign an ndarray to a memoryview slice? Is 
this correct/expected behaviour?

Whilst there are a couple of possible workarounds it would be nice if 
this just worked as it's a slightly surprising deviation from the 
ndarray behaviour.

Broken example:
```
In [3]: %%cython
   ...: cimport cython
   ...: import numpy as np
   ...: cimport numpy as np
   ...: 
   ...: cpdef np.ndarray[np.float64_t, ndim=2] f():
   ...:     cdef np.ndarray[np.float64_t, ndim=2] x = np.ones([3, 4], 
dtype=np.float64)
   ...:     cdef double[:,:] y = np.zeros([3, 4], dtype=np.float64)
   ...:     cdef np.ndarray[np.float64_t, ndim=1] row
   ...:     cdef int idx
   ...:     for idx, row in enumerate(x):
   ...:         y[idx] = row
   ...:     return np.asarray(y)

In [4]: f()
Traceback (most recent call last):

  File "<ipython-input-4-0ec059b9bfe1>", line 1, in <module>
    f()

  File "_cython_magic_5f2586693ddbf044815dae01d800bc0c.pyx", line 5, in 
_cython_magic_5f2586693ddbf044815dae01d800bc0c.f 
(C:\Users\dhirschfeld\.ipython\cython\_cython_magic_5f2586693ddbf044815d
ae01d800bc0c.c:2086)

  File "_cython_magic_5f2586693ddbf044815dae01d800bc0c.pyx", line 11, in 
_cython_magic_5f2586693ddbf044815dae01d800bc0c.f 
(C:\Users\dhirschfeld\.ipython\cython\_cython_magic_5f2586693ddbf044815d
ae01d800bc0c.c:1935)
```

Workarounds - either:
1. Type both lhs and rhs as ndarray
2. Create a 1D memoryview to use as a temporary container and assign 
that to the memoryview slice

```
In [5]: %%cython
   ...: cimport cython
   ...: import numpy as np
   ...: cimport numpy as np
   ...: 
   ...: cpdef np.ndarray[np.float64_t, ndim=2] f():
   ...:     cdef np.ndarray[np.float64_t, ndim=2] x = np.ones([3, 4], 
dtype=np.float64)
   ...:     cdef np.ndarray[np.float64_t, ndim=2] y = np.zeros([3, 4], 
dtype=np.float64)
   ...:     cdef np.ndarray[np.float64_t, ndim=1] row
   ...:     cdef int idx
   ...:     for idx, row in enumerate(x):
   ...:         y[idx] = row
   ...:     return np.asarray(y)

In [6]: f()
Out[6]: 
array([[ 1.,  1.,  1.,  1.],
       [ 1.,  1.,  1.,  1.],
       [ 1.,  1.,  1.,  1.]])

In [7]: %%cython
   ...: cimport cython
   ...: import numpy as np
   ...: cimport numpy as np
   ...: 
   ...: cpdef np.ndarray[np.float64_t, ndim=2] f():
   ...:     cdef np.ndarray[np.float64_t, ndim=2] x = np.ones([3, 4], 
dtype=np.float64)
   ...:     cdef double[:,:] y = np.zeros([3, 4], dtype=np.float64)
   ...:     cdef np.ndarray[np.float64_t, ndim=1] row
   ...:     cdef double[:] tmp
   ...:     cdef int idx
   ...:     for idx, row in enumerate(x):
   ...:         tmp = row
   ...:         y[idx] = tmp
   ...:     return np.asarray(y)

In [8]: f()
Out[8]: 
array([[ 1.,  1.,  1.,  1.],
       [ 1.,  1.,  1.,  1.],
       [ 1.,  1.,  1.,  1.]])
```

Samuele Kaplun | 2 Dec 14:13 2014
Picon
Picon

[Cython] Possible bug (or wrong documentation) WRT "not None"

Hi!

I would like to report what I think is a possible bug (in cython or 
corresponding documentation).

According to:
<http://docs.cython.org/src/userguide/extension_types.html#extension-types-and-none>

[...]
The self parameter of a method of an extension type is guaranteed never to be 
None.
[...]

However given this snippet:
[...]
cdef class test:
    def __sub__(self, test rhs not None):
        print "self is %s" % repr(self), "rhs is %s" % repr(rhs)
[...]

If I then load it python:
In [1]: import test
In [2]: x = test.test()
In [3]: None - x
self is None rhs is <test.test object at 0x7fdd9eeb7410>

Thus showing that in this case "self" was actually initialized with None.

If I explictly add "not None" as in:
[...]
cdef class test:
    def __sub__(self not None, test rhs not None):
        print "self is %s" % repr(self), "rhs is %s" % repr(rhs)
[...]
In [3]: None - x
TypeError: Argument 'self' must not be None
[...]

Which is good.

If I instead add "test self" as in:
[...]
cdef class test:
    def __sub__(test self not None, test rhs not None):
        print "self is %s" % repr(self), "rhs is %s" % repr(rhs)
[...]
In [3]: None - x
TypeError: Argument 'self' has incorrect type (expected test.test, got 
NoneType)
[...]

What's worse is that if I use "test self" (without not None), the same 
TypeError is not raised:
[...]
cdef class test:
    def __sub__(test self, test rhs not None):
        print "self is %s" % repr(self), "rhs is %s" % repr(rhs)
[...]
In [3]: None - x
self is None rhs is <test.test object at 0x7f17b36ff3f0>

Thus we have:
self -> not respecting None-checking (in disagreement with Documentation)
self not None ->  OK
test self -> not respecting type-checking!
test self non None -> OK (respecting type-checking, hence None-checking).

Hope that helps. Cheers,
	Samuele

--

-- 
Samuele Kaplun
Invenio Developer ** <http://invenio-software.org/>

Michael Enßlin | 30 Nov 16:14 2014

[Cython] cython --cplus --embed generates invalid code

Hi,

on my system, --embed does not work with --cplus.

How to reproduce: Any valid pyx file works:

$ rm -f test.pyx; touch test.pyx
$ cython --embed --cplus test.pyx
$ g++ -c test.cpp -I/usr/include/python3.4m
test.cpp: In function ‘wchar_t* __Pyx_char2wchar(char*)’:
test.cpp:945:41: error: invalid conversion from ‘void*’ to ‘wchar_t*’
[-fpermissive]
     res = malloc(argsize*sizeof(wchar_t));
$ clang++ -c test.cpp -I/usr/include/python3.4m
test.cpp:945:9: error: assigning to 'wchar_t *' from incompatible type
'void *'
    res = malloc(argsize*sizeof(wchar_t));

The issue can easily be fixed by manually casting the malloc result to
(wchar_t *).
In C it is not recommended to cast the malloc result, while in C++ it is
required (and malloc is discouraged).

System info:

$ cython --version
Cython version 0.21.1
$ g++ --version
g++ (Debian 4.9.2-2) 4.9.2
$ python3 --version
Python 3.4.2
$ lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux unstable (sid)
Release:	unstable
Codename:	sid

~ Michael

Hi,

on my system, --embed does not work with --cplus.

How to reproduce: Any valid pyx file works:

$ rm -f test.pyx; touch test.pyx
$ cython --embed --cplus test.pyx
$ g++ -c test.cpp -I/usr/include/python3.4m
test.cpp: In function ‘wchar_t* __Pyx_char2wchar(char*)’:
test.cpp:945:41: error: invalid conversion from ‘void*’ to ‘wchar_t*’
[-fpermissive]
     res = malloc(argsize*sizeof(wchar_t));
$ clang++ -c test.cpp -I/usr/include/python3.4m
test.cpp:945:9: error: assigning to 'wchar_t *' from incompatible type
'void *'
    res = malloc(argsize*sizeof(wchar_t));

The issue can easily be fixed by manually casting the malloc result to
(wchar_t *).
In C it is not recommended to cast the malloc result, while in C++ it is
required (and malloc is discouraged).

System info:

$ cython --version
Cython version 0.21.1
$ g++ --version
g++ (Debian 4.9.2-2) 4.9.2
$ python3 --version
Python 3.4.2
$ lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux unstable (sid)
Release:	unstable
Codename:	sid

~ Michael


Gmane