Robert Bradshaw | 27 Apr 20:15 2015
Picon

[Cython] New hosting

Since Cython's inception, we've been able to take advantage of William
Stein's and University of Washington's infrastructure for hosting the
Cython project along side that of Sage. However, due to UW policy,
those days are coming to and end. Maybe it's time--Cython has come a
long way from its birth of SageX + lxml.

So the question is where to move. Our primary needs are not that
large: we've got a site, a bugtracker (trac), and a continuous build
(jenkins) currently being served (the source code and wiki have
already been migrated to github).

I would propose that we look into moving everything we can over to
github. For starters, they now offer serving simple sites from a
repository (cython.org) so that seems an obvious choice. I know their
bug tracking v1 was considered far inferior to trac, but the situation
may be better now (at least worth exploring). We also have travis.ci,
which isn't as configurable as jenkins, but may be good enough. (The
biggest deficiency is that it probably wouldn't allow for building and
testing Sage regularly, or benchmarks, this is the one thing that we
may need to find/provide custom hosting for.)

Thoughts?
Stefan Behnel | 25 Apr 17:40 2015
Picon

[Cython] release preparations for 0.22.1 and 0.23

Hi,

I think it's about time for a new release. I propose to relase 0.22.1 in
the next days, and then start the release cycle for 0.23.

Please make sure everything that you consider a safe bug fix is in the
0.22.x branch. I already copied Jeroen's latest fixes over.

Stefan
Anton D. Kachalov | 23 Apr 11:22 2015
Picon

[Cython] Dash in the executable's filename

Hello.
 
I've found that executable script with dashes in the filename lead to produce wrong cythonized source:
 
   $ touch my-great-script.py
   $ cython my-great-script.py --embed
   $ fgrep PyInit my-great-script.c
PyMODINIT_FUNC PyInit_my-great-script(void); /*proto*/
PyMODINIT_FUNC PyInit_my-great-script(void)
  __Pyx_RefNannySetupContext("PyMODINIT_FUNC PyInit_my-great-script(void)", 0);
          m = PyInit_my-great-script();
 
So, if I don't want to import my final script elsewhere, I'm free to choose any filename for it.
 
--
Anton D. Kachalov
<div>
<div>Hello.</div>
<div>&nbsp;</div>
<div>I've found that executable script with dashes in the filename lead to produce wrong cythonized source:</div>
<div>&nbsp;</div>
<div>&nbsp;&nbsp; $ touch my-great-script.py</div>
<div>&nbsp;&nbsp; $ cython my-great-script.py --embed</div>
<div>&nbsp;&nbsp; $ fgrep PyInit my-great-script.c<br>PyMODINIT_FUNC PyInit_my-great-script(void); /*proto*/<br>PyMODINIT_FUNC PyInit_my-great-script(void)<br>&nbsp; __Pyx_RefNannySetupContext("PyMODINIT_FUNC PyInit_my-great-script(void)", 0);<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; m = PyInit_my-great-script();</div>
<div>&nbsp;</div>
<div>So, if I don't want to import my final script elsewhere, I'm free to choose any filename for it.</div>
<div>&nbsp;</div>
<div>-- <br>Anton D. Kachalov</div>
</div>
Michael Enßlin | 22 Apr 22:07 2015

[Cython] Cython produces invalid C code

Hi everybody,

Cython 0.21.1, from Debian Sid, and Cython 0.22, from Gentoo, produce
invalid C Code for the following .pyx file:

$ cat test.pyx
cimport cpython

cdef extern from "test.h":
    cdef void foo(int i = 0)

def bar(self):
    foo(0)

$ cat test.h
void foo(int i);

$ cython test.pyx

$ gcc -c test.c -I/usr/include/python3.4m
test.c: In function ‘__pyx_pf_4test_bar’:
test.c:659:35: error: storage size of ‘__pyx_t_1’ isn’t known
   struct __pyx_opt_args_4test_foo __pyx_t_1;

$ clang test.c -I/usr/include/python3.4m
test.c:659:35: error: variable has incomplete type 'struct
__pyx_opt_args_4test_foo'
  struct __pyx_opt_args_4test_foo __pyx_t_1;
test.c:659:10: note: forward declaration of 'struct
__pyx_opt_args_4test_foo'
  struct __pyx_opt_args_4test_foo __pyx_t_1;

Note that this is a minimal example; removing anything from test.pyx
fixes the issue (the 'cimport' statement, the default value for int i,
and the call to foo). The issue also occurs with --cplus.

Happy debugging,
	mic_e

Hi everybody,

Cython 0.21.1, from Debian Sid, and Cython 0.22, from Gentoo, produce
invalid C Code for the following .pyx file:

$ cat test.pyx
cimport cpython

cdef extern from "test.h":
    cdef void foo(int i = 0)

def bar(self):
    foo(0)

$ cat test.h
void foo(int i);

$ cython test.pyx

$ gcc -c test.c -I/usr/include/python3.4m
test.c: In function ‘__pyx_pf_4test_bar’:
test.c:659:35: error: storage size of ‘__pyx_t_1’ isn’t known
   struct __pyx_opt_args_4test_foo __pyx_t_1;

$ clang test.c -I/usr/include/python3.4m
test.c:659:35: error: variable has incomplete type 'struct
__pyx_opt_args_4test_foo'
  struct __pyx_opt_args_4test_foo __pyx_t_1;
test.c:659:10: note: forward declaration of 'struct
__pyx_opt_args_4test_foo'
  struct __pyx_opt_args_4test_foo __pyx_t_1;

Note that this is a minimal example; removing anything from test.pyx
fixes the issue (the 'cimport' statement, the default value for int i,
and the call to foo). The issue also occurs with --cplus.

Happy debugging,
	mic_e

Stefan Behnel | 21 Apr 07:21 2015
Picon

[Cython] Generator ABC in Py3.5

Hi,

I'm currently working on making the Generator protocol an ABC in Python
3.5. That will allow Cython generators to finally stand en-par with Python
generators.

https://bugs.python.org/issue24018

Stefan
Matt Graham | 17 Apr 03:09 2015
Picon

[Cython] Possible bug: Memory leak issues when using cpython.array as memoryview and/or buffer in Python 2.7

Hello,

I'm filing this as a bug report here as the issue tracker is closed to anonymous users.

## Summary

There seems to be a memory leak issue when using the buffer and memoryview support of the cpython.array Cython interface to the inbuilt Python array.array module.

## OS / Environment

Ubuntu 14.04
Cython 0.22
Python 2.7.6/2.7.9

## Description

When allocating `cpython.array` objects using the `clone` method provided in the Cython interface to a buffer or memoryview type the memory does not seem to be freed correctly when all references to the object have been removed when running Cython code with Python 2.7 (tested with both Ubuntu provided 2.7.6 version and 2.7.9 downloaded from python.org this evening). This issue does not seem to occur when running the exact same Cython code with Python 3.4.0. 

This appears like it could potentially be a long standing bug as the same issue was mentioned on the cython-user mailing list in November 2012 ([here][1]).

Using `cpython.array` with memory views seems to be quite a typical use case: it is discussed in the documentation [here][2] and [here][3] (specifically being stated to be memory safe) and is recommended in a popular StackOverflow answer [here][4] on the best way for allocating memory for a typed memory view in Cython (based on benchmarks performed in Python 3.x).



## Steps to reproduce

Cython code is attached (also copied below as not sure whether attachment will be preserved in posting) to reproduce this error.

Running this code with Python 2.7.9 gives output


>   Memory usage: 64.16015625 MiB

>   Starting cpython array buffer initialisation loop

>   Finished cpython array buffer initialisation loop

>   Memory usage: 459.71484375 MiB

>   Starting cpython array memview initialisation loop

>   Finished cpython array memview initialisation loop

>   Memory usage: 849.55078125 MiB


and with Python 3.4.0

>   Memory usage: 52.79296875 MiB
>   Starting cpython array buffer initialisation loop
>   Finished cpython array buffer initialisation loop
>   Memory usage: 52.8125 MiB
>   Starting cpython array memview initialisation loop
>   Finished cpython array memview initialisation loop
>   Memory usage: 52.81640625 MiB

Let me know if any more details are needed.

Thanks,
Matt

```

# cython: boundscheck=False

# cython: wraparound=False


import resource

import gc


from cpython.array cimport array, clone


def init_cpython_array_buffer(int length, int loops):

    cdef int i

    cdef array[double] arr, template = array('d')

    for i in range(loops):

        arr = clone(template, length, False)


def init_cpython_array_memview(int length, int loops):

    cdef int i

    cdef double[::1] arr

    cdef array template = array('d')

    for i in range(loops):

        arr = clone(template, length, False)


mem = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024.

print('Memory usage: {0} MiB'.format(mem))


print('Starting cpython array buffer initialisation loop')

init_cpython_array_buffer(100000, 100000)

# Force a garbage collection just to be safe

gc.collect()

print('Finished cpython array buffer initialisation loop')


mem = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024.

print('Memory usage: {0} MiB'.format(mem))


print('Starting cpython array memview initialisation loop')

init_cpython_array_memview(100000, 100000)

# Force a garbage collection just to be safe

gc.collect()

print('Finished cpython array memview initialisation loop')


mem = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024.

print('Memory usage: {0} MiB'.format(mem))

```

<div><div dir="ltr">Hello,<div><br></div>
<div>I'm filing this as a bug report here as the issue tracker is closed to anonymous users.</div>
<div><br></div>
<div>## Summary</div>
<div><br></div>
<div>There seems to be a memory leak issue when using the buffer and memoryview support of the cpython.array Cython interface to the inbuilt Python array.array module.</div>
<div><br></div>
<div>## OS / Environment</div>
<div><br></div>
<div>Ubuntu 14.04</div>
<div>Cython 0.22</div>
<div>Python 2.7.6/2.7.9</div>
<div><br></div>
<div>## Description</div>
<div><br></div>
<div>When allocating `cpython.array` objects using the `clone` method provided in the Cython interface to a buffer or memoryview type the memory does not seem to be freed correctly when all references to the object have been removed when running Cython code with Python 2.7 (tested with both Ubuntu provided 2.7.6 version and 2.7.9 downloaded from <a href="http://python.org">python.org</a> this evening). This issue does not seem to occur when running the exact same Cython code with Python 3.4.0.&nbsp;</div>
<div><br></div>
<div>This appears like it could potentially be a long standing bug as the same issue was mentioned on the cython-user mailing list in November 2012 ([here][1]).</div>
<div><br></div>
<div>Using `cpython.array` with memory views seems to be quite a typical use case: it is discussed in the documentation [here][2] and [here][3] (specifically being stated to be memory safe) and is recommended in a popular StackOverflow answer [here][4] on the best way for allocating memory for a typed memory view in Cython (based on benchmarks performed in Python 3.x).</div>
<div><br></div>
<div>&nbsp; [1]: <a href="https://groups.google.com/d/msg/cython-users/CwtU_jYADgM/660O2JJuO54J">https://groups.google.com/d/msg/cython-users/CwtU_jYADgM/660O2JJuO54J</a>
</div>
<div>&nbsp; [2]: <a href="http://docs.cython.org/src/userguide/memoryviews.html#cpython-array-module">http://docs.cython.org/src/userguide/memoryviews.html#cpython-array-module</a>
</div>
<div>&nbsp; [3]:&nbsp;<a href="http://docs.cython.org/src/tutorial/array.html">http://docs.cython.org/src/tutorial/array.html</a>
</div>
<div>&nbsp; [4]:&nbsp;<a href="http://stackoverflow.com/a/21054369/4798943">http://stackoverflow.com/a/21054369/4798943</a>
</div>
<div><br></div>
<div><br></div>
<div>## Steps to reproduce</div>
<div><br></div>
<div>Cython code is attached (also copied below as not sure whether attachment will be preserved in posting) to reproduce this error.</div>
<div><br></div>
<div>Running this code with Python 2.7.9 gives output</div>
<div>
<p><br></p>
<p>&gt; &nbsp; Memory usage: 64.16015625 MiB</p>
<p>&gt; &nbsp; Starting cpython array buffer initialisation loop</p>
<p>&gt; &nbsp; Finished cpython array buffer initialisation loop</p>
<p>&gt; &nbsp; Memory usage: 459.71484375 MiB</p>
<p>&gt; &nbsp; Starting cpython array memview initialisation loop</p>
<p>&gt; &nbsp; Finished cpython array memview initialisation loop</p>
<p>&gt; &nbsp; Memory usage: 849.55078125 MiB</p>
</div>
<div><br></div>
<div>and with Python 3.4.0</div>
<div><br></div>
<div>
<div>&gt; &nbsp; Memory usage: 52.79296875 MiB</div>
<div>&gt; &nbsp; Starting cpython array buffer initialisation loop</div>
<div>&gt; &nbsp; Finished cpython array buffer initialisation loop</div>
<div>&gt; &nbsp; Memory usage: 52.8125 MiB</div>
<div>&gt; &nbsp; Starting cpython array memview initialisation loop</div>
<div>&gt; &nbsp; Finished cpython array memview initialisation loop</div>
<div>&gt; &nbsp; Memory usage: 52.81640625 MiB</div>
</div>
<div><br></div>
<div>Let me know if any more details are needed.</div>
<div><br></div>
<div>Thanks,</div>
<div>Matt</div>
<div><br></div>
<div>```</div>
<div>
<p># cython: boundscheck=False<br></p>
<p># cython: wraparound=False</p>
<p><br></p>
<p>import resource</p>
<p>import gc</p>
<p><br></p>
<p>from cpython.array cimport array, clone</p>
<p><br></p>
<p>def init_cpython_array_buffer(int length, int loops):</p>
<p>&nbsp; &nbsp; cdef int i</p>
<p>&nbsp; &nbsp; cdef array[double] arr, template = array('d')</p>
<p>&nbsp; &nbsp; for i in range(loops):</p>
<p>&nbsp; &nbsp; &nbsp; &nbsp; arr = clone(template, length, False)</p>
<p><br></p>
<p>def init_cpython_array_memview(int length, int loops):</p>
<p>&nbsp; &nbsp; cdef int i</p>
<p>&nbsp; &nbsp; cdef double[::1] arr</p>
<p>&nbsp; &nbsp; cdef array template = array('d')</p>
<p>&nbsp; &nbsp; for i in range(loops):</p>
<p>&nbsp; &nbsp; &nbsp; &nbsp; arr = clone(template, length, False)</p>
<p><br></p>
<p>mem = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024.</p>
<p>print('Memory usage: {0} MiB'.format(mem))</p>
<p><br></p>
<p>print('Starting cpython array buffer initialisation loop')</p>
<p>init_cpython_array_buffer(100000, 100000)</p>
<p># Force a garbage collection just to be safe</p>
<p>gc.collect()</p>
<p>print('Finished cpython array buffer initialisation loop')</p>
<p><br></p>
<p>mem = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024.</p>
<p>print('Memory usage: {0} MiB'.format(mem))</p>
<p><br></p>
<p>print('Starting cpython array memview initialisation loop')</p>
<p>init_cpython_array_memview(100000, 100000)</p>
<p># Force a garbage collection just to be safe</p>
<p>gc.collect()</p>
<p>print('Finished cpython array memview initialisation loop')</p>
<p><br></p>
<p>mem = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024.</p>
<p>print('Memory usage: {0} MiB'.format(mem))</p>
<p>```</p>
</div>
</div></div>
Stefan Behnel | 17 Apr 07:53 2015
Picon

[Cython] PEP 489: Redesigning extension module loading

Hi,

I've been involved in redesigning a new protocol for importing extension
modules, which might make it into CPython 3.5. I think it's now ready for
general consideration.

http://article.gmane.org/gmane.comp.python.import/508

It solves several issues with the current scheme and brings it much closer
to the behaviour of "normal" Python modules.

Stefan
Stefan Behnel | 12 Apr 21:41 2015
Picon

[Cython] CI tests with pypy/pypy3

Hi,

I set up CI tests on Jenkins for the latest pypy and pypy3 releases and
invested some time into improving the situation.

https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pypy/

I explicitly disabled a couple of tests that either crash or otherwise
won't work in pypy anyway. For example, pypy's GIL support is incomplete
and inheriting from builtin types is buggy.

https://github.com/cython/cython/blob/master/tests/pypy_bugs.txt

It doesn't look all that bad, at least for pypy2. I consider pypy3 work in
progress, but we should keep them busy with fixing bugs in cpyext. There's
always room for improvements and work-arounds on our side, too, but the
major issues usually come up on their side.

Stefan
Carlos Pita | 7 Apr 18:39 2015
Picon

[Cython] [RFE] Add dummy compiler directive decorators for pure python mode

Hi all,

pure python mode will benefit from the addition of dummy decorators
for compiler directives to Shadow.py, like in:

 <at> cython.boundscheck(False)
def f(): pass

AFAICS this is not currently valid inside the interpreter.

Cheers
--
Carlos
Carlos Pita | 6 Apr 00:49 2015
Picon

[Cython] [Bug] Coercion of struct constructor nodes

Hi all,

f and g below should behave identically, shouldn't them?

import cython

cdef struct Point:
    int x
    int y

def f():
    return Point(x=10, y=10)

def g():
    cdef Point p = Point(x=10, y=10)
    return p

But then f won't compile:

 Cannot interpret dict as type 'Python object'

DictNode.coerce_to decides it can't coerce a CStructOrUnionType to a
python object, but the underlying struct type does know how to coerce
a struct into a dict.

Here is an old post stating the same, although I don't know if it ever
entered the bug tracker.

https://www.mail-archive.com/cython-dev-F/1GfIIGwJtbRRN4PJnoQQ <at> public.gmane.org/msg09121.html

Best regards
--
Carlos
Carlos Pita | 5 Apr 21:48 2015
Picon

[Cython] [Bug] Memoryviews in pure python mode

Hi all,

I've posted about this in the user list but after thinking about it a
bit more and doing some testing, I tend to believe it's a bug.

In the following code, the cython.double[:] in  <at> cython.locals is not
recognized as a type, while g() compiles fine:

import cython
import scipy

 <at> cython.locals(x=cython.double[:])
def f():
    x = scipy.array([1,2,3], scipy.double)

def g():
    cdef double[:] x = scipy.array([1,2,3], scipy.double)

Now, one could said memoryviews aren't supported in pure python mode
(which would be a pity) but then, in the interpreter:

In [48]: cy.int[:]
Out[48]: int[:]

In [49]: type(cy.int[:])
Out[49]: Cython.Shadow._ArrayType

Shadow.py implements the machinery for the interpreter, but the
compiler is not consistently supporting it.

Best regards
--
Carlos

PS: do you know of any workaround?

Gmane