Yaroslav Halchenko | 22 Jul 18:48 2014

[Cython] buffmt.__test__.wrongsize gets stuck after common_include_dir?

while trying to re-test Debian package build, in an interactive shell, I
get cython test stuck at:

...
Doctest: buffmt.__test__.wrongsize ... ok
runTest (__main__.EndToEndTest)
End-to-end basic_cythonize ... ok
runTest (__main__.EndToEndTest)
End-to-end basic_distutils ... ok
runTest (__main__.EndToEndTest)
End-to-end build_dir ... ok
runTest (__main__.EndToEndTest)
End-to-end common_include_dir ...

tests were executed using

runtests.py --no-refnanny -v -v --exclude="parallel" --work-dir=build/work-dir

any clues?  is it multiprocessing to blame again?

/bin/bash /tmp/hooks/C10shell
  /bin/bash
    /usr/bin/make -f debian/rules binary
      /usr/bin/perl -w /usr/bin/dh binary --with python2,python3 --buildsystem python_distutils
        /usr/bin/make -f debian/rules override_dh_auto_test
          /bin/sh -c set -e; for P in 2.7 3.4; do \          echo "PYTHON: $P"; \  export; \  PYTHONPATH=`/bin/ls -d
/tmp/buildd/cython-0.20.2/build/lib.*-$P` \   /usr/bin/python$P runtests.py --no-refnanny -v -v
--exclude="parallel" --work-dir=build/work-dir; \ done
            /usr/bin/python2.7 runtests.py --no-refnanny -v -v --exclude=parallel --work-dir=build/work-dir
              /bin/sh -c /usr/bin/python2.7 setup.py build_ext --inplace
(Continue reading)

Kurt Smith | 16 Jul 19:01 2014
Picon

[Cython] Automatic conversion with fixed-size C arrays

Hi devs,

Being able to convert between simple C structs and Python dictionaries is great, but it doesn't work if there is a fixed-size array field in the struct.  It seems like the following struct should be convertible safely:

cdef struct state_t:
    int i, j, k
    float x[3]
    float v[3]

Along with that, I'd expect that converting between fixed-sized C arrays and Python iterables should work as well:

def auto_convert(list ll):
    cdef int arr[10] = ll
    # ...
    return arr

If this is deemed something worth doing and if someone is willing to give pointers when I get stuck, I'm offering to put in the development time to implement this.

Thoughts?

Kurt
<div><div dir="ltr">Hi devs,<div><br></div>
<div>Being able to convert between simple C structs and Python dictionaries is great, but it doesn't work if there is a fixed-size array field in the struct. &nbsp;It seems like the following struct should be convertible safely:</div>
<div><br></div>
<div>cdef struct state_t:</div>
<div>&nbsp; &nbsp; int i, j, k</div>
<div>&nbsp; &nbsp; float x[3]</div>
<div>&nbsp; &nbsp; float v[3]</div>
<div><br></div>
<div>Along with that, I'd expect that converting between fixed-sized C arrays and Python iterables should work as well:</div>
<div><br></div>
<div>def auto_convert(list ll):</div>
<div>&nbsp; &nbsp; cdef int arr[10] = ll</div>
<div>&nbsp; &nbsp; # ...</div>
<div>&nbsp; &nbsp; return arr</div>
<div><br></div>
<div>If this is deemed something worth doing and if someone is willing to give pointers when I get stuck, I'm offering to put in the development time to implement this.</div>
<div><br></div>
<div>Thoughts?</div>
<div><br></div>
<div>Kurt</div>
</div></div>
Benjamin Lerman | 15 Jul 11:12 2014

[Cython] License information on each individual files

 Hi,

 I'm planning to use cython in chromium.

 To be able to use cython as a third-party library, we need a license
header on every files in the project (because we are creating source
packages, and need to have clear licensing).

 Would cython accept to add such a copyright header on its files?

 Thanks in advance,

--

-- 
        Benjamin
Stefan Behnel | 14 Jul 21:49 2014
Picon

[Cython] heads-up on master branch status

Hi,

by applying some initially small but perpetually growing fixes, I pushed
myself into a corner where the only escape was to rewrite BoolBinopNode
(short-circuiting and/or expressions). This was long overdue anyway and
simply wasn't done before because it's not trivial and quite risky. I think
I got it working better than before, but there are still corner cases that
require some further fighting. One of them currently crashes the pyregr
tests in Py2.7, it's this code:

    [tuple((e is None and 'X' or e) for e in t) for t in target]

Thank you, python-devs, for collecting these little gems in the test suite.
From the C code this generates, the problem seems to be related to type
coercion in closures. I'm currently preparing my participation at
EuroPython next week, so I can't tell for sure when I'll be able to dig
into this. This means that current master may stay less stable than it
should be for a couple of more days.

Stefan
Alok Singhal | 9 Jul 01:34 2014
Picon

[Cython] Compiler crash: forward declaration of cdef class with "public" attributes

Hi,

I am getting a compiler crash with the following code:

$ cat foo.pxd
cdef class Foo:
    cdef public object x
$ cat foo.pyx
cdef class Foo

cdef class Foo:
    pass
$ ./cython.py  foo.pyx

Error compiling Cython file:
------------------------------------------------------------
...
cdef class Foo
    ^
------------------------------------------------------------

foo.pyx:1:5: Compiler crash in AnalyseDeclarationsTransform

ModuleNode.body = StatListNode(foo.pyx:1:0)
StatListNode.stats[0] = CClassDefNode(foo.pyx:1:5,
    as_name = u'Foo',
    class_name = u'Foo',
    module_name = '',
    visibility = 'private')

Compiler crash traceback from this point on:
  File "/Users/alok/Downloads/repos/cython.git/Cython/Compiler/Visitor.py", line 173, in _visit
    return handler_method(obj)
  File "/Users/alok/Downloads/repos/cython.git/Cython/Compiler/ParseTreeTransforms.py", line 1478, in visit_CClassDefNode
    node.body.stats += stats
AttributeError: 'NoneType' object has no attribute 'stats'

The problem seems to be due to having a .pxd file with the attributes of the cdef class in it.  Since the attributes are declared "public", Cython is trying to generate property code for "Foo.x".  But it's trying to do that at the point of forward declaration in the .pyx file, instead of the definition of the class.

An easy fix seems to be to change AnalyseDeclarationsTransform.visit_CClassDefNode in ParseTreeTransforms.py to check for node.body not being none.  I.e., the following patch seems to work:

diff --git a/Cython/Compiler/ParseTreeTransforms.py b/Cython/Compiler/ParseTreeTransforms.py
index f8c2efa..7811d56 100644
--- a/Cython/Compiler/ParseTreeTransforms.py
+++ b/Cython/Compiler/ParseTreeTransforms.py
<at> <at> -1466,7 +1466,7 <at> <at> if VALUE is not None:
 
     def visit_CClassDefNode(self, node):
         node = self.visit_ClassDefNode(node)
-        if node.scope and node.scope.implemented:
+        if node.body and node.scope and node.scope.implemented:
             stats = []
             for entry in node.scope.var_entries:
                 if entry.needs_property:

Thanks,
Alok
<div><div dir="ltr">Hi,<br><br>I am getting a compiler crash with the following code:<br><br>$ cat foo.pxd<br>cdef class Foo:<br>&nbsp; &nbsp; cdef public object x<br>$ cat foo.pyx<br>cdef class Foo<br><br>cdef class Foo:<br>&nbsp; &nbsp; pass<br>$ ./cython.py &nbsp;foo.pyx<br><br>Error compiling Cython file:<br>------------------------------------------------------------<br>...<br>cdef class Foo<br>&nbsp; &nbsp; ^<br>------------------------------------------------------------<br><br>foo.pyx:1:5: Compiler crash in AnalyseDeclarationsTransform<br><br>ModuleNode.body = StatListNode(foo.pyx:1:0)<br>StatListNode.stats[0] = CClassDefNode(foo.pyx:1:5,<br>&nbsp; &nbsp; as_name = u'Foo',<br>&nbsp; &nbsp; class_name = u'Foo',<br>
&nbsp; &nbsp; module_name = '',<br>&nbsp; &nbsp; visibility = 'private')<br><br>Compiler crash traceback from this point on:<br>&nbsp; File "/Users/alok/Downloads/repos/cython.git/Cython/Compiler/Visitor.py", line 173, in _visit<br>
&nbsp; &nbsp; return handler_method(obj)<br>&nbsp; File "/Users/alok/Downloads/repos/cython.git/Cython/Compiler/ParseTreeTransforms.py", line 1478, in visit_CClassDefNode<br>&nbsp; &nbsp; node.body.stats += stats<br>AttributeError: 'NoneType' object has no attribute 'stats'<div>
<br>
</div>
<div>The problem seems to be due to having a .pxd file with the attributes of the cdef class in it. &nbsp;Since the attributes are declared "public", Cython is trying to generate property code for "Foo.x". &nbsp;But it's trying to do that at the point of forward declaration in the .pyx file, instead of the definition of the class.</div>
<div><br></div>
<div>An easy fix seems to be to change AnalyseDeclarationsTransform.visit_CClassDefNode in&nbsp;ParseTreeTransforms.py to check for node.body not being none. &nbsp;I.e., the following patch seems to work:</div>
<div><br></div>
<div>
<div>diff --git a/Cython/Compiler/ParseTreeTransforms.py b/Cython/Compiler/ParseTreeTransforms.py</div>
<div>index f8c2efa..7811d56 100644</div>
<div>--- a/Cython/Compiler/ParseTreeTransforms.py</div>
<div>+++ b/Cython/Compiler/ParseTreeTransforms.py</div>
<div> <at>  <at>  -1466,7 +1466,7  <at>  <at>  if VALUE is not None:</div>
<div>&nbsp;</div>
<div>&nbsp; &nbsp; &nbsp;def visit_CClassDefNode(self, node):</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;node = self.visit_ClassDefNode(node)</div>
<div>- &nbsp; &nbsp; &nbsp; &nbsp;if node.scope and node.scope.implemented:</div>
<div>+ &nbsp; &nbsp; &nbsp; &nbsp;if node.body and node.scope and node.scope.implemented:</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;stats = []</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;for entry in node.scope.var_entries:</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;if entry.needs_property:</div>
</div>
<div><br></div>
<div>Thanks,</div>
<div>Alok</div>
</div></div>
Stefan Behnel | 5 Jul 12:34 2014
Picon

[Cython] memory view creation during cascaded assignments

Hi,

I started optimising cascaded assignments a little to reduce the
unnecessary duplication of coercions. While doing that, I found this test
in memslice.pyx:

'''
def cascaded_buffer_assignment(obj):
    """
    >>> A = IntMockBuffer("A", range(6))
    >>> cascaded_buffer_assignment(A)
    acquired A
    acquired A
    released A
    released A
    """
    cdef int[:] a, b
    a = b = obj
'''

It's explicitly tested for that we create two independent memory views in
this case. Is there an actual reason for this? As long as the types of a
and b are identical, I don't see why we would want to request the buffer twice.

Stefan
Matthew Brett | 3 Jul 12:49 2014
Picon

[Cython] Travis-ci builds of OSX wheels

Hi,

We just got scikit-image OSX wheel builds more or less fully automated
on travis-ci :

https://travis-ci.org/scikit-image/scikit-image-wheels
https://github.com/scikit-image/scikit-image-wheels

Any interest in getting the same thing working for Cython?

Cheers,

Matthew
Stefan Behnel | 25 Jun 21:16 2014
Picon

[Cython] NumPy 1.7+ warnings

Hi,

recent NumPy versions are a bit unhappy about Cython code and generate
deprecation warnings about it:

.../numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning:
#warning "Using deprecated NumPy API, disable it by " "#defining
NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION"

I don't know how much work it would be to port the C code properly, but it
should be mostly about updating the implementation in Cython's
"numpy/__init__.pxd", which is dated anyway.

Given that NumPy 1.7 appeared not much more than a year ago, dropping
support for older versions sounds a bit aggressive, but there should be a
way to keep supporting them by defining some macros, similar to what we do
for CPython in ModuleSetupCode.c. Any volunteers?

The relevant bits of NumPy's C-API are described here:

http://docs.scipy.org/doc/numpy-dev/reference/c-api.array.html

My guess is that most of the work should be replacing direct access to
ndarray struct fields with C macro calls. That won't easily fix user code,
but it's the first major step on the way.

Stefan
Stefan Behnel | 16 Jun 15:58 2014
Picon

[Cython] JyNI - C-API emulation for Jython

Hi,

it looks like there is a C-API emulation for Jython on the way:

http://jyni.org/

https://github.com/Stewori/JyNI

I haven't tried it yet, and I'm sure Cython modules won't work with its
current state, but the effort I put into making them run in PyPy's cpyext
should pay off here, so we might at some point have a third backend to target.

This also reminded me of IronClad, the C-API layer for IronPython. I looked
it up, but it seems to have died some years ago, with no current effort to
improve it or even just make it work with the latest IronPython releases.

https://groups.google.com/d/msg/c-extensions-for-ironpython/SrIesUIh1Gw/cBQG1FMt0AUJ

https://code.google.com/p/ironclad/

Not sure if it's still worth trying to make that run in Cython.

Stefan
Stefan Behnel | 16 Jun 10:07 2014
Picon

[Cython] any more changes for 0.21?

Hi,

I think the master branch is good enough for at least a 0.21 alpha. Are
there any pending changes that should go in before it can be released? Any
pull requests that should be considered?

We should get this in, I think:

https://github.com/cython/cython/pull/284

And the gdb tests are still broken.

Robert, could you take a look through the Sage test failures? Most of them
might be issues in Sage rather than Cython, but it's better to make sure we
can either fix or safely ignore them.

Stefan
Emmanuel Gil Peyrot | 28 May 13:25 2014
Picon

[Cython] Wrong order of __Pyx_DECREF when calling a function with an implicit str → char* conversion.

Hi,

I was testing my cython codebase on top of pypy/cpyext, and I found a
memory corruption.  After investigating it with the pypy guys (thanks
arigato!), we identified it as a cython bug:

 cdef void test(char *string):
     print(string)

 def run(array):
     test(array[0])

This code works fine under cpython, but looking at the generated code
for the last line, the call to test is done with a pointer (__pyx_t_2)
to some potentially deallocated string (__pyx_t_1), which isn’t freed
just yet on cpython but is on pypy:

 __pyx_t_1 = __Pyx_GetItemInt(__pyx_v_array, 0, …);
 __Pyx_GOTREF(__pyx_t_1);
 __pyx_t_2 = __Pyx_PyObject_AsString(__pyx_t_1);
 __Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0;
 __pyx_f_1a_test(__pyx_t_2);

The obvious solution is to swap the last two lines, it then works fine
on pypy (although not necessarily if the test function stores the
pointer somewhere, but it’s not cython’s fault then).

This issue can also happen with an explicit cast:

 pointer = <char *>array[0]
 test(pointer)

I’m not sure if it should be addressed the same way, because that would
mean keeping a reference to array[0] for all the lifetime of the
current scope, but it could still prevent obscure bugs like the memory
corruption I had.

-- 
Emmanuel Gil Peyrot
Hi,

I was testing my cython codebase on top of pypy/cpyext, and I found a
memory corruption.  After investigating it with the pypy guys (thanks
arigato!), we identified it as a cython bug:

 cdef void test(char *string):
     print(string)

 def run(array):
     test(array[0])

This code works fine under cpython, but looking at the generated code
for the last line, the call to test is done with a pointer (__pyx_t_2)
to some potentially deallocated string (__pyx_t_1), which isn’t freed
just yet on cpython but is on pypy:

 __pyx_t_1 = __Pyx_GetItemInt(__pyx_v_array, 0, …);
 __Pyx_GOTREF(__pyx_t_1);
 __pyx_t_2 = __Pyx_PyObject_AsString(__pyx_t_1);
 __Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0;
 __pyx_f_1a_test(__pyx_t_2);

The obvious solution is to swap the last two lines, it then works fine
on pypy (although not necessarily if the test function stores the
pointer somewhere, but it’s not cython’s fault then).

This issue can also happen with an explicit cast:

 pointer = <char *>array[0]
 test(pointer)

I’m not sure if it should be addressed the same way, because that would
mean keeping a reference to array[0] for all the lifetime of the
current scope, but it could still prevent obscure bugs like the memory
corruption I had.

--

-- 
Emmanuel Gil Peyrot

Gmane