Matt Graham | 17 Apr 03:09 2015
Picon

[Cython] Possible bug: Memory leak issues when using cpython.array as memoryview and/or buffer in Python 2.7

Hello,

I'm filing this as a bug report here as the issue tracker is closed to anonymous users.

## Summary

There seems to be a memory leak issue when using the buffer and memoryview support of the cpython.array Cython interface to the inbuilt Python array.array module.

## OS / Environment

Ubuntu 14.04
Cython 0.22
Python 2.7.6/2.7.9

## Description

When allocating `cpython.array` objects using the `clone` method provided in the Cython interface to a buffer or memoryview type the memory does not seem to be freed correctly when all references to the object have been removed when running Cython code with Python 2.7 (tested with both Ubuntu provided 2.7.6 version and 2.7.9 downloaded from python.org this evening). This issue does not seem to occur when running the exact same Cython code with Python 3.4.0. 

This appears like it could potentially be a long standing bug as the same issue was mentioned on the cython-user mailing list in November 2012 ([here][1]).

Using `cpython.array` with memory views seems to be quite a typical use case: it is discussed in the documentation [here][2] and [here][3] (specifically being stated to be memory safe) and is recommended in a popular StackOverflow answer [here][4] on the best way for allocating memory for a typed memory view in Cython (based on benchmarks performed in Python 3.x).



## Steps to reproduce

Cython code is attached (also copied below as not sure whether attachment will be preserved in posting) to reproduce this error.

Running this code with Python 2.7.9 gives output


>   Memory usage: 64.16015625 MiB

>   Starting cpython array buffer initialisation loop

>   Finished cpython array buffer initialisation loop

>   Memory usage: 459.71484375 MiB

>   Starting cpython array memview initialisation loop

>   Finished cpython array memview initialisation loop

>   Memory usage: 849.55078125 MiB


and with Python 3.4.0

>   Memory usage: 52.79296875 MiB
>   Starting cpython array buffer initialisation loop
>   Finished cpython array buffer initialisation loop
>   Memory usage: 52.8125 MiB
>   Starting cpython array memview initialisation loop
>   Finished cpython array memview initialisation loop
>   Memory usage: 52.81640625 MiB

Let me know if any more details are needed.

Thanks,
Matt

```

# cython: boundscheck=False

# cython: wraparound=False


import resource

import gc


from cpython.array cimport array, clone


def init_cpython_array_buffer(int length, int loops):

    cdef int i

    cdef array[double] arr, template = array('d')

    for i in range(loops):

        arr = clone(template, length, False)


def init_cpython_array_memview(int length, int loops):

    cdef int i

    cdef double[::1] arr

    cdef array template = array('d')

    for i in range(loops):

        arr = clone(template, length, False)


mem = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024.

print('Memory usage: {0} MiB'.format(mem))


print('Starting cpython array buffer initialisation loop')

init_cpython_array_buffer(100000, 100000)

# Force a garbage collection just to be safe

gc.collect()

print('Finished cpython array buffer initialisation loop')


mem = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024.

print('Memory usage: {0} MiB'.format(mem))


print('Starting cpython array memview initialisation loop')

init_cpython_array_memview(100000, 100000)

# Force a garbage collection just to be safe

gc.collect()

print('Finished cpython array memview initialisation loop')


mem = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024.

print('Memory usage: {0} MiB'.format(mem))

```

<div><div dir="ltr">Hello,<div><br></div>
<div>I'm filing this as a bug report here as the issue tracker is closed to anonymous users.</div>
<div><br></div>
<div>## Summary</div>
<div><br></div>
<div>There seems to be a memory leak issue when using the buffer and memoryview support of the cpython.array Cython interface to the inbuilt Python array.array module.</div>
<div><br></div>
<div>## OS / Environment</div>
<div><br></div>
<div>Ubuntu 14.04</div>
<div>Cython 0.22</div>
<div>Python 2.7.6/2.7.9</div>
<div><br></div>
<div>## Description</div>
<div><br></div>
<div>When allocating `cpython.array` objects using the `clone` method provided in the Cython interface to a buffer or memoryview type the memory does not seem to be freed correctly when all references to the object have been removed when running Cython code with Python 2.7 (tested with both Ubuntu provided 2.7.6 version and 2.7.9 downloaded from <a href="http://python.org">python.org</a> this evening). This issue does not seem to occur when running the exact same Cython code with Python 3.4.0.&nbsp;</div>
<div><br></div>
<div>This appears like it could potentially be a long standing bug as the same issue was mentioned on the cython-user mailing list in November 2012 ([here][1]).</div>
<div><br></div>
<div>Using `cpython.array` with memory views seems to be quite a typical use case: it is discussed in the documentation [here][2] and [here][3] (specifically being stated to be memory safe) and is recommended in a popular StackOverflow answer [here][4] on the best way for allocating memory for a typed memory view in Cython (based on benchmarks performed in Python 3.x).</div>
<div><br></div>
<div>&nbsp; [1]: <a href="https://groups.google.com/d/msg/cython-users/CwtU_jYADgM/660O2JJuO54J">https://groups.google.com/d/msg/cython-users/CwtU_jYADgM/660O2JJuO54J</a>
</div>
<div>&nbsp; [2]: <a href="http://docs.cython.org/src/userguide/memoryviews.html#cpython-array-module">http://docs.cython.org/src/userguide/memoryviews.html#cpython-array-module</a>
</div>
<div>&nbsp; [3]:&nbsp;<a href="http://docs.cython.org/src/tutorial/array.html">http://docs.cython.org/src/tutorial/array.html</a>
</div>
<div>&nbsp; [4]:&nbsp;<a href="http://stackoverflow.com/a/21054369/4798943">http://stackoverflow.com/a/21054369/4798943</a>
</div>
<div><br></div>
<div><br></div>
<div>## Steps to reproduce</div>
<div><br></div>
<div>Cython code is attached (also copied below as not sure whether attachment will be preserved in posting) to reproduce this error.</div>
<div><br></div>
<div>Running this code with Python 2.7.9 gives output</div>
<div>
<p><br></p>
<p>&gt; &nbsp; Memory usage: 64.16015625 MiB</p>
<p>&gt; &nbsp; Starting cpython array buffer initialisation loop</p>
<p>&gt; &nbsp; Finished cpython array buffer initialisation loop</p>
<p>&gt; &nbsp; Memory usage: 459.71484375 MiB</p>
<p>&gt; &nbsp; Starting cpython array memview initialisation loop</p>
<p>&gt; &nbsp; Finished cpython array memview initialisation loop</p>
<p>&gt; &nbsp; Memory usage: 849.55078125 MiB</p>
</div>
<div><br></div>
<div>and with Python 3.4.0</div>
<div><br></div>
<div>
<div>&gt; &nbsp; Memory usage: 52.79296875 MiB</div>
<div>&gt; &nbsp; Starting cpython array buffer initialisation loop</div>
<div>&gt; &nbsp; Finished cpython array buffer initialisation loop</div>
<div>&gt; &nbsp; Memory usage: 52.8125 MiB</div>
<div>&gt; &nbsp; Starting cpython array memview initialisation loop</div>
<div>&gt; &nbsp; Finished cpython array memview initialisation loop</div>
<div>&gt; &nbsp; Memory usage: 52.81640625 MiB</div>
</div>
<div><br></div>
<div>Let me know if any more details are needed.</div>
<div><br></div>
<div>Thanks,</div>
<div>Matt</div>
<div><br></div>
<div>```</div>
<div>
<p># cython: boundscheck=False<br></p>
<p># cython: wraparound=False</p>
<p><br></p>
<p>import resource</p>
<p>import gc</p>
<p><br></p>
<p>from cpython.array cimport array, clone</p>
<p><br></p>
<p>def init_cpython_array_buffer(int length, int loops):</p>
<p>&nbsp; &nbsp; cdef int i</p>
<p>&nbsp; &nbsp; cdef array[double] arr, template = array('d')</p>
<p>&nbsp; &nbsp; for i in range(loops):</p>
<p>&nbsp; &nbsp; &nbsp; &nbsp; arr = clone(template, length, False)</p>
<p><br></p>
<p>def init_cpython_array_memview(int length, int loops):</p>
<p>&nbsp; &nbsp; cdef int i</p>
<p>&nbsp; &nbsp; cdef double[::1] arr</p>
<p>&nbsp; &nbsp; cdef array template = array('d')</p>
<p>&nbsp; &nbsp; for i in range(loops):</p>
<p>&nbsp; &nbsp; &nbsp; &nbsp; arr = clone(template, length, False)</p>
<p><br></p>
<p>mem = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024.</p>
<p>print('Memory usage: {0} MiB'.format(mem))</p>
<p><br></p>
<p>print('Starting cpython array buffer initialisation loop')</p>
<p>init_cpython_array_buffer(100000, 100000)</p>
<p># Force a garbage collection just to be safe</p>
<p>gc.collect()</p>
<p>print('Finished cpython array buffer initialisation loop')</p>
<p><br></p>
<p>mem = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024.</p>
<p>print('Memory usage: {0} MiB'.format(mem))</p>
<p><br></p>
<p>print('Starting cpython array memview initialisation loop')</p>
<p>init_cpython_array_memview(100000, 100000)</p>
<p># Force a garbage collection just to be safe</p>
<p>gc.collect()</p>
<p>print('Finished cpython array memview initialisation loop')</p>
<p><br></p>
<p>mem = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024.</p>
<p>print('Memory usage: {0} MiB'.format(mem))</p>
<p>```</p>
</div>
</div></div>
Stefan Behnel | 17 Apr 07:53 2015
Picon

[Cython] PEP 489: Redesigning extension module loading

Hi,

I've been involved in redesigning a new protocol for importing extension
modules, which might make it into CPython 3.5. I think it's now ready for
general consideration.

http://article.gmane.org/gmane.comp.python.import/508

It solves several issues with the current scheme and brings it much closer
to the behaviour of "normal" Python modules.

Stefan
Stefan Behnel | 12 Apr 21:41 2015
Picon

[Cython] CI tests with pypy/pypy3

Hi,

I set up CI tests on Jenkins for the latest pypy and pypy3 releases and
invested some time into improving the situation.

https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pypy/

I explicitly disabled a couple of tests that either crash or otherwise
won't work in pypy anyway. For example, pypy's GIL support is incomplete
and inheriting from builtin types is buggy.

https://github.com/cython/cython/blob/master/tests/pypy_bugs.txt

It doesn't look all that bad, at least for pypy2. I consider pypy3 work in
progress, but we should keep them busy with fixing bugs in cpyext. There's
always room for improvements and work-arounds on our side, too, but the
major issues usually come up on their side.

Stefan
Carlos Pita | 7 Apr 18:39 2015
Picon

[Cython] [RFE] Add dummy compiler directive decorators for pure python mode

Hi all,

pure python mode will benefit from the addition of dummy decorators
for compiler directives to Shadow.py, like in:

 <at> cython.boundscheck(False)
def f(): pass

AFAICS this is not currently valid inside the interpreter.

Cheers
--
Carlos
Carlos Pita | 6 Apr 00:49 2015
Picon

[Cython] [Bug] Coercion of struct constructor nodes

Hi all,

f and g below should behave identically, shouldn't them?

import cython

cdef struct Point:
    int x
    int y

def f():
    return Point(x=10, y=10)

def g():
    cdef Point p = Point(x=10, y=10)
    return p

But then f won't compile:

 Cannot interpret dict as type 'Python object'

DictNode.coerce_to decides it can't coerce a CStructOrUnionType to a
python object, but the underlying struct type does know how to coerce
a struct into a dict.

Here is an old post stating the same, although I don't know if it ever
entered the bug tracker.

https://www.mail-archive.com/cython-dev-F/1GfIIGwJtbRRN4PJnoQQ <at> public.gmane.org/msg09121.html

Best regards
--
Carlos
Carlos Pita | 5 Apr 21:48 2015
Picon

[Cython] [Bug] Memoryviews in pure python mode

Hi all,

I've posted about this in the user list but after thinking about it a
bit more and doing some testing, I tend to believe it's a bug.

In the following code, the cython.double[:] in  <at> cython.locals is not
recognized as a type, while g() compiles fine:

import cython
import scipy

 <at> cython.locals(x=cython.double[:])
def f():
    x = scipy.array([1,2,3], scipy.double)

def g():
    cdef double[:] x = scipy.array([1,2,3], scipy.double)

Now, one could said memoryviews aren't supported in pure python mode
(which would be a pity) but then, in the interpreter:

In [48]: cy.int[:]
Out[48]: int[:]

In [49]: type(cy.int[:])
Out[49]: Cython.Shadow._ArrayType

Shadow.py implements the machinery for the interpreter, but the
compiler is not consistently supporting it.

Best regards
--
Carlos

PS: do you know of any workaround?
Lisandro Dalcin | 29 Mar 12:17 2015
Picon

[Cython] "long long" versus "PY_LONG_LONG"

At some point Cython lost the ability of using PY_LONG_LONG instead of
"long long" in generated C code. The big offenders are the following
two files:

Cython/Utility/Overflow.c
Cython/Utility/TypeConversion.c

Do we want to continue using PY_LONG_LONG? Otherwise, what about
Microsoft compilers? Going back to PY_LONG_LONG everywhere is quite
easy right now, let me know and I'll contribute the patch for the
upcoming 0.22.1 release (if that ever happens).

--

-- 
Lisandro Dalcin
============
Research Scientist
Computer, Electrical and Mathematical Sciences & Engineering (CEMSE)
Numerical Porous Media Center (NumPor)
King Abdullah University of Science and Technology (KAUST)
http://numpor.kaust.edu.sa/

4700 King Abdullah University of Science and Technology
al-Khawarizmi Bldg (Bldg 1), Office # 4332
Thuwal 23955-6900, Kingdom of Saudi Arabia
http://www.kaust.edu.sa

Office Phone: +966 12 808-0459
Daniel Grunwald | 17 Mar 18:24 2015

[Cython] Generated function has invalid name when converting from python object to C struct defined in C++ namespace

Hello,

I have Cython code like this:
     cdef extern from "cpp_library.h" namespace "CppLibrary":
         struct SomeStruct:
             int i

         void do_with_struct(SomeStruct s)

     def run():
         do_with_struct(object())

With cython 0.21.1, invalid C++ code is generated:
    struct CppLibrary::SomeStruct;
    static struct CppLibrary::SomeStruct 
__pyx_convert__from_py_CppLibrary::SomeStruct(PyObject *);

With cython master, a compiler error occurs:
      <at> cname("__pyx_convert__from_py_CppLibrary::SomeStruct")
     cdef SomeStruct __pyx_convert__from_py_CppLibrary::SomeStruct(obj) 
except *:
                                                      ^
     FromPyStructUtility:11:50: Expected an identifier or literal

Cython should mangle the struct name to ensure the helper function has a 
valid name.

Regards,

--

-- 
Daniel Grunwald
Axivion GmbH
Nobelstr. 15
70569 Stuttgart
Deutschland
Tel: +49 711 6204378-33
Fax: +49 711 6204378-99

Geschaeftsfuehrung: Stefan Bellon, Thomas Eisenbarth, Sebastian Rummler
Sitz der Gesellschaft: Stuttgart
Registergericht: Amtsgericht Stuttgart, HRB 720590

Nathan Goldbaum | 11 Mar 06:27 2015
Picon

[Cython] Cython magic annotate option is broken under IPython 3.0

Hi all,

To reproduce this issue, be on cython 0.22 and IPython 3.0 and run the following notebook:


Googling the error returns some several year old discussions in IPython related to the autoreload magic and I'm not sure if that is a red herring here:


For how I'm working around this by calling cython at the command line in my notebook and then loading the annotated HTML with an IFrame.

Thanks for your help!
<div><div dir="ltr">
<span>Hi all,</span><div><br></div>
<div>To reproduce this issue, be on cython 0.22 and IPython 3.0 and run the following notebook:</div>
<div><br></div>
<div>
<a href="http://nbviewer.ipython.org/gist/ngoldbaum/855a629d997aa7959254" target="_blank">http://nbviewer.ipython.org/gist/ngoldbaum/855a629d997aa7959254</a><br>
</div>
<div><br></div>
<div>Googling the error returns some several year old discussions in IPython related to the autoreload magic and I'm not sure if that is a red herring here:</div>
<div><br></div>
<div>
<a href="http://mail.scipy.org/pipermail/ipython-dev/2012-May/009288.html" target="_blank">http://mail.scipy.org/pipermail/ipython-dev/2012-May/009288.html</a><br>
</div>
<div><br></div>
<div>For how I'm working around this by calling cython at the command line in my notebook and then loading the annotated HTML with an IFrame.</div>
<div><br></div>
<div>Thanks for your help!</div>
</div></div>
Jeroen Demeyer | 7 Mar 09:58 2015
Picon

[Cython] Proposal: extern implementations of functions

Hello Cython developers,

I would like to propose a new mechanism to implement functions in an 
external .c file. The goal is to solve the problem I was having in
https://groups.google.com/forum/#!topic/cython-users/TsbBNyvwZn4

With the attached patch, you can do the following:

* in foo.pxd, declare the function as usual:
cdef void myfunc()

* in foo.c, actually implement the function:
static void myfunc()
{
     /* ... */
}

* in foo.pyx, declare the function extern:
cdef extern from "foo.c":
     void myfunc()

With my patch, the extern declaration in foo.pyx will be interpreted as 
implementing the function. Therefore, myfunc() is now available in the 
module foo, and it can also be cimported by other modules.

What do you think of this proposed feature?

Jeroen.
Hello Cython developers,

I would like to propose a new mechanism to implement functions in an 
external .c file. The goal is to solve the problem I was having in
https://groups.google.com/forum/#!topic/cython-users/TsbBNyvwZn4

With the attached patch, you can do the following:

* in foo.pxd, declare the function as usual:
cdef void myfunc()

* in foo.c, actually implement the function:
static void myfunc()
{
     /* ... */
}

* in foo.pyx, declare the function extern:
cdef extern from "foo.c":
     void myfunc()

With my patch, the extern declaration in foo.pyx will be interpreted as 
implementing the function. Therefore, myfunc() is now available in the 
module foo, and it can also be cimported by other modules.

What do you think of this proposed feature?

Jeroen.
Lisandro Dalcin | 17 Mar 21:10 2015
Picon

[Cython] Sorry: 0.22.x and master broken

Folks, I'm very sorry. In an attempt to fix issues with PyPy3, I
pushed a fix to 0.22.x
(https://github.com/cython/cython/commit/8573e7898a27ef607a3264909cc3bf24d4e13959)
and merged to master, but now one test is failing with CPython 3
(https://travis-ci.org/cython/cython/jobs/54762494). I did not run the
full test suite with python3, my fault.

I'll look at this issue tomorrow and push a fix ASAP.

--

-- 
Lisandro Dalcin
============
Research Scientist
Computer, Electrical and Mathematical Sciences & Engineering (CEMSE)
Numerical Porous Media Center (NumPor)
King Abdullah University of Science and Technology (KAUST)
http://numpor.kaust.edu.sa/

4700 King Abdullah University of Science and Technology
al-Khawarizmi Bldg (Bldg 1), Office # 4332
Thuwal 23955-6900, Kingdom of Saudi Arabia
http://www.kaust.edu.sa

Office Phone: +966 12 808-0459

Gmane