Vinay Sajip | 16 Dec 11:09 2013
Picon

[Cython] Seemingly incorrect specialization of templates

Given this code in Cython/Includes/libcpp/utility.pxd:

cdef extern from "<utility>" namespace "std":
    cdef cppclass pair[T, U]:
        T first
        U second
        # rest omitted

when the following code in Cython/Includes/libcpp/map.pxd is seen:

from utility cimport pair

cdef extern from "<map>" namespace "std":
    cdef cppclass map[T, U]:
        cppclass iterator:
            pair[T, U]& operator*() nogil
            # rest omitted

The line "pair[T, U]& operator*() nogil" causes a specialization
of pair[T, U] to be created, but using placeholders rather than actual
types. This seems wrong - can someone here confirm whether this is
intentional?

This led to a problem where the generated C++ code for a genuine
specialisation instead generated a declaration like "pair<T, U>" instead
of using the actual arguments. Given this code in
Cython/Includes/libcpp/utility.pxd:

cdef extern from "<utility>" namespace "std":
    cdef cppclass pair[T, U]:
(Continue reading)

Stefan Behnel | 29 Nov 17:03 2013
Picon

"embedsignature" and "autotestdict" features in Py3.4

Hi,

recent changes in Py3.4 affect the functionality of the two directives
"embedsignature" and "autotestdict".

The so-called "argument clinic" extracts any embedded signatures from
docstrings and moves them into a new property "__text_signature__". This is
done at runtime, i.e. when either "__doc__" or "__text_signature__" are
requested.

http://hg.python.org/cpython/file/6a3e09cd96f3/Objects/methodobject.c#l182

I personally consider this a feature (and it works nicely with the
signatures that Cython embeds), but you may or may not agree. It broke some
of our own doctests, at least, because the "__doc__" value that we tested
for was no longer the same.

Regarding the "autotestdict", CPython also got smarter and now finds
doctests embedded in C implemented functions all by itself. This is clearly
an improvement compared to the previous behaviour. However, this also means
that it now executes both the doctest of the function itself and its copy
in the generated "__test__" dict (stored in the module under that name),
meaning that it executes all function doctests twice. This also broke some
of Cython's own doctests for us, which modified global state and thus
failed the second time.

I'm yet unsure if we should try to do something about this. Currently, the
explicit doctest dict is generated by default, so all code that uses
doctests in function docstrings suffers from this. We might be able to make
Cython a bit smarter so that these docstrings are left out of the test dict
(Continue reading)

Yury V. Zaytsev | 29 Nov 14:26 2013

[Cython] 'Referenced before assignment' warning triggered due to OpenMP directives?

Hi,

I'm trying to find out whether the number of threads has been actually
set to what I wanted it to be, but the compiler tells me that I'm trying
to reference a variable before assignment. This doesn't happen when I
simply release the GIL, but only within a parallel section.

Is it a problem with Cython or I'm missing something obvious?

Thanks!

$ cat ompt.pyx 
cimport openmp
from cython.parallel import parallel, prange

cdef Py_ssize_t x = 1

with nogil, parallel():
    x = openmp.omp_get_num_threads()

print("Number of OpenMP threads is '{}'!".format(x))

$ cython ompt.pyx 

Error compiling Cython file:
------------------------------------------------------------
...
cdef Py_ssize_t x = 1

with nogil, parallel():
(Continue reading)

Daniele Nicolodi | 26 Nov 16:18 2013
Picon

[Cython] BUG: assignment to typed memoryview

Hello,

I believe there is a bug in how assignment to typed memoryviews is
handled in the compiler.  I think the following code should be valid code:

  cdef double[::1] a = np.arange(10, dtype=np.double)
  cdef double[::1] b = np.empty(a.size // 2)
  b[:] = a[::2]

However, the Cython compiler complains with:

  Memoryview 'double[:]' not conformable to memoryview 'double[::1]'.

A similar thing happens with memoryview copies:

  cdef double[::1] a = np.arange(10, dtype=np.double)
  cdef double[::1] b
  b = a[::2].copy()

In both cases I would expect Cython to copy the non contiguous elements
of the source array to the destination array (or to a newly created
array in the second case).  The copy() method is defined here
https://github.com/cython/cython/blob/master/Cython/Utility/MemoryView.pyx#L592
and (while I haven't spent much time analyzing it in detail) it seems to
do the right thing.

Indeed, if I short-circuit the src_conforms_to_dst() function
https://github.com/cython/cython/blob/master/Cython/Compiler/MemoryView.py#L141
the code compiles and runs correctly in both cases.

(Continue reading)

Vinay Sajip | 25 Nov 11:35 2013
Picon

[Cython] Cython crash when using forward declarations

The following Cython source:

cdef extern from "header":
    cdef cppclass String
    cdef cppclass List[T]
    cdef cppclass StringList(List[String])

    cdef cppclass String:
        String()
        int length()

    cdef cppclass List[T]:
        List()
        int length()

    cdef cppclass StringList(List[String]):
        StringList()
        void sort()

causes a crash:

Error compiling Cython file:
------------------------------------------------------------
...

    cdef cppclass List[T]:
        List()
        int length()

    cdef cppclass StringList(List[String]):
(Continue reading)

Stefan Behnel | 8 Nov 15:16 2013
Picon

[Cython] question on Get/SetItemInt*() utility functions

Hi,

the Get/SetItemInt helper macros usually look something like this:

"""
#define __Pyx_GetItemInt(o, i, size, to_py_func, is_list, wraparound,
boundscheck) \
    (((size) <= sizeof(Py_ssize_t)) ? \
    __Pyx_GetItemInt_Fast(o, i, is_list, wraparound, boundscheck) : \
    __Pyx_GetItemInt_Generic(o, to_py_func(i)))
"""

e.g. here:

https://github.com/cython/cython/blob/59c3408c054bbbb7334ccca7d76bed93f26057e9/Cython/Utility/ObjectHandling.c#L248

https://github.com/cython/cython/blob/59c3408c054bbbb7334ccca7d76bed93f26057e9/Cython/Utility/StringTools.c#L318

The "generic" part will almost never be called, except when the index
*type* is really huge. However, if that happens, there's a rather large
impact due to object creation, even if the value in question is tiny.

Is there a way to safely extend this compile time size check with checks
that compare the actual runtime value with PY_SSIZE_T_MIN and PY_SSIZE_T_MAX ?

Within those bounds, a cast would do the trick just fine, and we could
still use the fast path. If it fails that bounds check, however, and the
type we are processing is a builtin container, we'd already know that an
exception is going to be raised and could simply drop the generic
implementation, replacing it with an appropriate IndexError directly.
(Continue reading)

Josh Ayers | 6 Nov 19:35 2013
Picon

[Cython] Bug Report: can't assign zero length array to contiguous 2D memoryview

Assigning a NumPy array with a zero length dimension to a memoryview fails if the memoryview is declared contiguous in the zero-length dimension.  I'm getting a run-time error that the buffer is not C-contiguous or Fortran contiguous.

See examples below.

Thanks,
Josh Ayers


cdef float [:, :] arr1 = numpy.empty((2, 0), 'f4') # works
cdef float [:, :] arr2 = numpy.empty((0, 2), 'f4') # works
cdef float [:, ::1] arr3 = numpy.empty((2, 0), 'f4') # fails (buffer not C-contiguous)
cdef float [:, ::1] arr4 = numpy.empty((0, 2), 'f4') # works

cdef float [:, :] arr5 = numpy.empty((2, 0), 'f4', order='F') # works
cdef float [:, :] arr6 = numpy.empty((0, 2), 'f4', order='F') # works
cdef float [::1, :] arr7 = numpy.empty((2, 0), 'f4', order='F') # works
cdef float [::1, :] arr8 = numpy.empty((0, 2), 'f4', order='F') # fails (buffer not Fortran contiguous)

<div><div dir="ltr">
<div>
<div>
<div>Assigning a NumPy array with a zero length dimension to a memoryview fails if the memoryview is declared contiguous in the zero-length dimension.&nbsp; I'm getting a run-time error that the buffer is not C-contiguous or Fortran contiguous.<br><br>
</div>See examples below.<br><br>
</div>Thanks,<br>
</div>Josh Ayers<br><br><div><div><div><div>
<br>cdef float [:, :] arr1 = numpy.empty((2, 0), 'f4') # works<br>cdef float [:, :] arr2 = numpy.empty((0, 2), 'f4') # works<br>
cdef float [:, ::1] arr3 = numpy.empty((2, 0), 'f4') # fails (buffer not C-contiguous)<br>cdef float [:, ::1] arr4 = numpy.empty((0, 2), 'f4') # works<br><br>cdef float [:, :] arr5 = numpy.empty((2, 0), 'f4', order='F') # works<br>
cdef float [:, :] arr6 = numpy.empty((0, 2), 'f4', order='F') # works<br>cdef float [::1, :] arr7 = numpy.empty((2, 0), 'f4', order='F') # works<br>cdef float [::1, :] arr8 = numpy.empty((0, 2), 'f4', order='F') # fails (buffer not Fortran contiguous)<br><br>
</div></div></div></div>
</div></div>
James Sanders | 5 Nov 15:03 2013
Picon

[Cython] Bug report: some incorrect typed memoryview indexing operations cause crashes

If I try to compile the following with the current master version:

def buggy(int N, double[:] x):
    x[0, N-1]

I get a crash.  This only seems to happen when too many indices are used, and at least one of them is an expression involving a variable (for example, things like x[0, N] and x[0, 2-1] don't cause crashes).  Here is the full output (this probably isn't related, but I'm also a bit confused about all these "non-trivial type declarators" warnings, which keep appearing in code I thought was working correctly):

warning: View.MemoryView:410:43: Non-trivial type declarators in shared declaration.
warning: View.MemoryView:583:32: Non-trivial type declarators in shared declaration.
warning: View.MemoryView:588:32: Non-trivial type declarators in shared declaration.
warning: View.MemoryView:1020:20: Non-trivial type declarators in shared declaration.
warning: View.MemoryView:1020:28: Non-trivial type declarators in shared declaration.
warning: View.MemoryView:1020:38: Non-trivial type declarators in shared declaration.

Error compiling Cython file:
------------------------------
------------------------------
...
def buggy(int N, double[:] x):
    x[0, N-1]
         ^
------------------------------------------------------------

bug.pyx:2:10: Too many indices specified for type double[:]

Error compiling Cython file:
------------------------------------------------------------
...
def buggy(int N, double[:] x):
    x[0, N-1]
         ^
------------------------------------------------------------

bug.pyx:2:10: Compiler crash in OptimizeBuiltinCalls

ModuleNode.body = StatListNode(bug.pyx:1:0)
StatListNode.stats[0] = DefNode(bug.pyx:1:0,
    modifiers = [...]/0,
    name = 'buggy',
    num_required_args = 2,
    py_wrapper_required = True,
    reqd_kw_flags_cname = '0',
    used = True)
DefNode.body = StatListNode(bug.pyx:2:5)
StatListNode.stats[0] = ExprStatNode(bug.pyx:2:5)
ExprStatNode.expr = IndexNode(bug.pyx:2:5,
    use_managed_ref = True)
IndexNode.index = TupleNode(bug.pyx:2:5,
    is_sequence_constructor = 1,
    result_is_used = True,
    use_managed_ref = True)
TupleNode.args[1] = SubNode(bug.pyx:2:10,
    infix = True,
    operator = '-',
    result_is_used = True,
    use_managed_ref = True)

Compiler crash traceback from this point on:
  File "Visitor.py", line 170, in Cython.Compiler.Visitor.TreeVisitor._visit (/home/.../cython/Cython/Compiler/Visitor.c:4122)
  File "Visitor.py", line 505, in Cython.Compiler.Visitor.MethodDispatcherTransform.visit_BinopNode (/home/.../cython/Cython/Compiler/Visitor.c:9459)
  File "Visitor.py", line 516, in Cython.Compiler.Visitor.MethodDispatcherTransform._visit_binop_node (/home/.../cython/Cython/Compiler/Visitor.c:9613)
AttributeError: 'NoneType' object has no attribute 'is_builtin_type'
<div><div dir="ltr">If I try to compile the following with the current master version:<br><div>
<br>def buggy(int N, double[:] x):<br>&nbsp;&nbsp;&nbsp; x[0, N-1]<br><br>
</div>I
 get a crash.&nbsp; This only seems to happen when too many indices are used,
 and at least one of them is an expression involving a 
variable (for example, things like x[0, N] and x[0, 2-1] don't cause 
crashes).&nbsp; Here is the full output (this probably isn't related, but I'm also a bit confused about all 
these "non-trivial type declarators" warnings, which keep appearing in 
code I thought was working correctly):<br><br>warning: View.MemoryView:410:43: Non-trivial type declarators in shared declaration.<br>warning: View.MemoryView:583:32: Non-trivial type declarators in shared declaration.<br>warning: View.MemoryView:588:32: Non-trivial type declarators in shared declaration.<br>

warning: View.MemoryView:1020:20: Non-trivial type declarators in shared declaration.<br>warning: View.MemoryView:1020:28: Non-trivial type declarators in shared declaration.<br>warning: View.MemoryView:1020:38: Non-trivial type declarators in shared declaration.<br><br>Error compiling Cython file:<br>------------------------------<div>------------------------------<br>...<br>def buggy(int N, double[:] x):<br>&nbsp;&nbsp;&nbsp; x[0, N-1]<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ^<br>------------------------------------------------------------<br><br>bug.pyx:2:10: Too many indices specified for type double[:]<br><br>Error compiling Cython file:<br>------------------------------------------------------------<br>...<br>def buggy(int N, double[:] x):<br>&nbsp;&nbsp;&nbsp; x[0, N-1]<br>

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ^<br>------------------------------------------------------------<br><br>bug.pyx:2:10: Compiler crash in OptimizeBuiltinCalls<br><br>ModuleNode.body = StatListNode(bug.pyx:1:0)<br>StatListNode.stats[0] = DefNode(bug.pyx:1:0,<br>

&nbsp;&nbsp;&nbsp; modifiers = [...]/0,<br>&nbsp;&nbsp;&nbsp; name = 'buggy',<br>&nbsp;&nbsp;&nbsp; num_required_args = 2,<br>&nbsp;&nbsp;&nbsp; py_wrapper_required = True,<br>&nbsp;&nbsp;&nbsp; reqd_kw_flags_cname = '0',<br>&nbsp;&nbsp;&nbsp; used = True)<br>DefNode.body = StatListNode(bug.pyx:2:5)<br>

StatListNode.stats[0] = ExprStatNode(bug.pyx:2:5)<br>ExprStatNode.expr = IndexNode(bug.pyx:2:5,<br>&nbsp;&nbsp;&nbsp; use_managed_ref = True)<br>IndexNode.index = TupleNode(bug.pyx:2:5,<br>&nbsp;&nbsp;&nbsp; is_sequence_constructor = 1,<br>&nbsp;&nbsp;&nbsp; result_is_used = True,<br>

&nbsp;&nbsp;&nbsp; use_managed_ref = True)<br>TupleNode.args[1] = SubNode(bug.pyx:2:10,<br>&nbsp;&nbsp;&nbsp; infix = True,<br>&nbsp;&nbsp;&nbsp; operator = '-',<br>&nbsp;&nbsp;&nbsp; result_is_used = True,<br>&nbsp;&nbsp;&nbsp; use_managed_ref = True)<br><br>Compiler crash traceback from this point on:<br>

&nbsp; File "Visitor.py", line 170, in Cython.Compiler.Visitor.TreeVisitor._visit (/home/.../cython/Cython/Compiler/Visitor.c:4122)<br>&nbsp; File "Visitor.py", line 505, in Cython.Compiler.Visitor.MethodDispatcherTransform.visit_BinopNode (/home/.../cython/Cython/Compiler/Visitor.c:9459)<br>

&nbsp; File "Visitor.py", line 516, in Cython.Compiler.Visitor.MethodDispatcherTransform._visit_binop_node (/home/.../cython/Cython/Compiler/Visitor.c:9613)<br>AttributeError: 'NoneType' object has no attribute 'is_builtin_type'</div>
</div></div>
Frédéric Bastien | 5 Nov 18:08 2013

[Cython] Supporting new and old NumPy C API

Hi,

As you probably know, cython generated c code use the old NumPy C API,
so when we compile them with newer version of NumPy this generate
warnings.

In Theano we compile many different module during the user script. So
this cause many warning being printed to the users, so we updated
Theano to work with newer and older NumPy without warning.

As we disable the old NumPy C API, we are not able to compile the
cython generated code. So I manually updated the cython code to work
with the new and old api.

I did a post on the numpy mailing list that explain how I did this, in
case someone want to tackle this problem in Cython:
http://mail.scipy.org/pipermail/numpy-discussion/2013-November/068209.html

Frédéric Bastien
Lisandro Dalcin | 1 Nov 14:16 2013
Picon

[Cython] Files with executable bit on

This is in branch 0.19.x. These files should not have the executable
bit on. Can any of you please fix them?

[dalcinl <at> macarena cython-dev]$ ls -al $(find Cython -name '*.py') | grep rwx
-rwxr-xr-x  1 dalcinl  staff    4286 Feb 14  2013
Cython/Build/BuildExecutable.py
-rwxr-xr-x  1 dalcinl  staff  420289 Oct 15 23:39 Cython/Compiler/ExprNodes.py
-rwxr-xr-x  1 dalcinl  staff  137439 Oct 15 23:39 Cython/Compiler/PyrexTypes.py

--

-- 
Lisandro Dalcin
---------------
CIMEC (UNL/CONICET)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1016)
Tel/Fax: +54-342-4511169
Stefan Behnel | 31 Oct 18:25 2013
Picon

"Stack" checker for undefined behaviour in C code

Hi,

I just came across this paper:

http://pdos.csail.mit.edu/~xi/papers/stack-sosp13.pdf

They describe an analysis tool that checks C code for bugs that exploit
undefined behaviour and that are thus up to the mercy of compiler
assumptions and "optimisations" to do the right thing or not. They made it
available on github:

https://github.com/xiw/stack/

If anyone wants to take the time to set it up for checking some Cython
generated code, I'd be interested to see if it finds something.

Stefan

--

-- 

--- 
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


Gmane