Stefan Behnel | 8 Nov 15:16 2013
Picon

[Cython] question on Get/SetItemInt*() utility functions

Hi,

the Get/SetItemInt helper macros usually look something like this:

"""
#define __Pyx_GetItemInt(o, i, size, to_py_func, is_list, wraparound,
boundscheck) \
    (((size) <= sizeof(Py_ssize_t)) ? \
    __Pyx_GetItemInt_Fast(o, i, is_list, wraparound, boundscheck) : \
    __Pyx_GetItemInt_Generic(o, to_py_func(i)))
"""

e.g. here:

https://github.com/cython/cython/blob/59c3408c054bbbb7334ccca7d76bed93f26057e9/Cython/Utility/ObjectHandling.c#L248

https://github.com/cython/cython/blob/59c3408c054bbbb7334ccca7d76bed93f26057e9/Cython/Utility/StringTools.c#L318

The "generic" part will almost never be called, except when the index
*type* is really huge. However, if that happens, there's a rather large
impact due to object creation, even if the value in question is tiny.

Is there a way to safely extend this compile time size check with checks
that compare the actual runtime value with PY_SSIZE_T_MIN and PY_SSIZE_T_MAX ?

Within those bounds, a cast would do the trick just fine, and we could
still use the fast path. If it fails that bounds check, however, and the
type we are processing is a builtin container, we'd already know that an
exception is going to be raised and could simply drop the generic
implementation, replacing it with an appropriate IndexError directly.
(Continue reading)

Josh Ayers | 6 Nov 19:35 2013
Picon

[Cython] Bug Report: can't assign zero length array to contiguous 2D memoryview

Assigning a NumPy array with a zero length dimension to a memoryview fails if the memoryview is declared contiguous in the zero-length dimension.  I'm getting a run-time error that the buffer is not C-contiguous or Fortran contiguous.

See examples below.

Thanks,
Josh Ayers


cdef float [:, :] arr1 = numpy.empty((2, 0), 'f4') # works
cdef float [:, :] arr2 = numpy.empty((0, 2), 'f4') # works
cdef float [:, ::1] arr3 = numpy.empty((2, 0), 'f4') # fails (buffer not C-contiguous)
cdef float [:, ::1] arr4 = numpy.empty((0, 2), 'f4') # works

cdef float [:, :] arr5 = numpy.empty((2, 0), 'f4', order='F') # works
cdef float [:, :] arr6 = numpy.empty((0, 2), 'f4', order='F') # works
cdef float [::1, :] arr7 = numpy.empty((2, 0), 'f4', order='F') # works
cdef float [::1, :] arr8 = numpy.empty((0, 2), 'f4', order='F') # fails (buffer not Fortran contiguous)

<div><div dir="ltr">
<div>
<div>
<div>Assigning a NumPy array with a zero length dimension to a memoryview fails if the memoryview is declared contiguous in the zero-length dimension.&nbsp; I'm getting a run-time error that the buffer is not C-contiguous or Fortran contiguous.<br><br>
</div>See examples below.<br><br>
</div>Thanks,<br>
</div>Josh Ayers<br><br><div><div><div><div>
<br>cdef float [:, :] arr1 = numpy.empty((2, 0), 'f4') # works<br>cdef float [:, :] arr2 = numpy.empty((0, 2), 'f4') # works<br>
cdef float [:, ::1] arr3 = numpy.empty((2, 0), 'f4') # fails (buffer not C-contiguous)<br>cdef float [:, ::1] arr4 = numpy.empty((0, 2), 'f4') # works<br><br>cdef float [:, :] arr5 = numpy.empty((2, 0), 'f4', order='F') # works<br>
cdef float [:, :] arr6 = numpy.empty((0, 2), 'f4', order='F') # works<br>cdef float [::1, :] arr7 = numpy.empty((2, 0), 'f4', order='F') # works<br>cdef float [::1, :] arr8 = numpy.empty((0, 2), 'f4', order='F') # fails (buffer not Fortran contiguous)<br><br>
</div></div></div></div>
</div></div>
James Sanders | 5 Nov 15:03 2013
Picon

[Cython] Bug report: some incorrect typed memoryview indexing operations cause crashes

If I try to compile the following with the current master version:

def buggy(int N, double[:] x):
    x[0, N-1]

I get a crash.  This only seems to happen when too many indices are used, and at least one of them is an expression involving a variable (for example, things like x[0, N] and x[0, 2-1] don't cause crashes).  Here is the full output (this probably isn't related, but I'm also a bit confused about all these "non-trivial type declarators" warnings, which keep appearing in code I thought was working correctly):

warning: View.MemoryView:410:43: Non-trivial type declarators in shared declaration.
warning: View.MemoryView:583:32: Non-trivial type declarators in shared declaration.
warning: View.MemoryView:588:32: Non-trivial type declarators in shared declaration.
warning: View.MemoryView:1020:20: Non-trivial type declarators in shared declaration.
warning: View.MemoryView:1020:28: Non-trivial type declarators in shared declaration.
warning: View.MemoryView:1020:38: Non-trivial type declarators in shared declaration.

Error compiling Cython file:
------------------------------
------------------------------
...
def buggy(int N, double[:] x):
    x[0, N-1]
         ^
------------------------------------------------------------

bug.pyx:2:10: Too many indices specified for type double[:]

Error compiling Cython file:
------------------------------------------------------------
...
def buggy(int N, double[:] x):
    x[0, N-1]
         ^
------------------------------------------------------------

bug.pyx:2:10: Compiler crash in OptimizeBuiltinCalls

ModuleNode.body = StatListNode(bug.pyx:1:0)
StatListNode.stats[0] = DefNode(bug.pyx:1:0,
    modifiers = [...]/0,
    name = 'buggy',
    num_required_args = 2,
    py_wrapper_required = True,
    reqd_kw_flags_cname = '0',
    used = True)
DefNode.body = StatListNode(bug.pyx:2:5)
StatListNode.stats[0] = ExprStatNode(bug.pyx:2:5)
ExprStatNode.expr = IndexNode(bug.pyx:2:5,
    use_managed_ref = True)
IndexNode.index = TupleNode(bug.pyx:2:5,
    is_sequence_constructor = 1,
    result_is_used = True,
    use_managed_ref = True)
TupleNode.args[1] = SubNode(bug.pyx:2:10,
    infix = True,
    operator = '-',
    result_is_used = True,
    use_managed_ref = True)

Compiler crash traceback from this point on:
  File "Visitor.py", line 170, in Cython.Compiler.Visitor.TreeVisitor._visit (/home/.../cython/Cython/Compiler/Visitor.c:4122)
  File "Visitor.py", line 505, in Cython.Compiler.Visitor.MethodDispatcherTransform.visit_BinopNode (/home/.../cython/Cython/Compiler/Visitor.c:9459)
  File "Visitor.py", line 516, in Cython.Compiler.Visitor.MethodDispatcherTransform._visit_binop_node (/home/.../cython/Cython/Compiler/Visitor.c:9613)
AttributeError: 'NoneType' object has no attribute 'is_builtin_type'
<div><div dir="ltr">If I try to compile the following with the current master version:<br><div>
<br>def buggy(int N, double[:] x):<br>&nbsp;&nbsp;&nbsp; x[0, N-1]<br><br>
</div>I
 get a crash.&nbsp; This only seems to happen when too many indices are used,
 and at least one of them is an expression involving a 
variable (for example, things like x[0, N] and x[0, 2-1] don't cause 
crashes).&nbsp; Here is the full output (this probably isn't related, but I'm also a bit confused about all 
these "non-trivial type declarators" warnings, which keep appearing in 
code I thought was working correctly):<br><br>warning: View.MemoryView:410:43: Non-trivial type declarators in shared declaration.<br>warning: View.MemoryView:583:32: Non-trivial type declarators in shared declaration.<br>warning: View.MemoryView:588:32: Non-trivial type declarators in shared declaration.<br>

warning: View.MemoryView:1020:20: Non-trivial type declarators in shared declaration.<br>warning: View.MemoryView:1020:28: Non-trivial type declarators in shared declaration.<br>warning: View.MemoryView:1020:38: Non-trivial type declarators in shared declaration.<br><br>Error compiling Cython file:<br>------------------------------<div>------------------------------<br>...<br>def buggy(int N, double[:] x):<br>&nbsp;&nbsp;&nbsp; x[0, N-1]<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ^<br>------------------------------------------------------------<br><br>bug.pyx:2:10: Too many indices specified for type double[:]<br><br>Error compiling Cython file:<br>------------------------------------------------------------<br>...<br>def buggy(int N, double[:] x):<br>&nbsp;&nbsp;&nbsp; x[0, N-1]<br>

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ^<br>------------------------------------------------------------<br><br>bug.pyx:2:10: Compiler crash in OptimizeBuiltinCalls<br><br>ModuleNode.body = StatListNode(bug.pyx:1:0)<br>StatListNode.stats[0] = DefNode(bug.pyx:1:0,<br>

&nbsp;&nbsp;&nbsp; modifiers = [...]/0,<br>&nbsp;&nbsp;&nbsp; name = 'buggy',<br>&nbsp;&nbsp;&nbsp; num_required_args = 2,<br>&nbsp;&nbsp;&nbsp; py_wrapper_required = True,<br>&nbsp;&nbsp;&nbsp; reqd_kw_flags_cname = '0',<br>&nbsp;&nbsp;&nbsp; used = True)<br>DefNode.body = StatListNode(bug.pyx:2:5)<br>

StatListNode.stats[0] = ExprStatNode(bug.pyx:2:5)<br>ExprStatNode.expr = IndexNode(bug.pyx:2:5,<br>&nbsp;&nbsp;&nbsp; use_managed_ref = True)<br>IndexNode.index = TupleNode(bug.pyx:2:5,<br>&nbsp;&nbsp;&nbsp; is_sequence_constructor = 1,<br>&nbsp;&nbsp;&nbsp; result_is_used = True,<br>

&nbsp;&nbsp;&nbsp; use_managed_ref = True)<br>TupleNode.args[1] = SubNode(bug.pyx:2:10,<br>&nbsp;&nbsp;&nbsp; infix = True,<br>&nbsp;&nbsp;&nbsp; operator = '-',<br>&nbsp;&nbsp;&nbsp; result_is_used = True,<br>&nbsp;&nbsp;&nbsp; use_managed_ref = True)<br><br>Compiler crash traceback from this point on:<br>

&nbsp; File "Visitor.py", line 170, in Cython.Compiler.Visitor.TreeVisitor._visit (/home/.../cython/Cython/Compiler/Visitor.c:4122)<br>&nbsp; File "Visitor.py", line 505, in Cython.Compiler.Visitor.MethodDispatcherTransform.visit_BinopNode (/home/.../cython/Cython/Compiler/Visitor.c:9459)<br>

&nbsp; File "Visitor.py", line 516, in Cython.Compiler.Visitor.MethodDispatcherTransform._visit_binop_node (/home/.../cython/Cython/Compiler/Visitor.c:9613)<br>AttributeError: 'NoneType' object has no attribute 'is_builtin_type'</div>
</div></div>
Frédéric Bastien | 5 Nov 18:08 2013

[Cython] Supporting new and old NumPy C API

Hi,

As you probably know, cython generated c code use the old NumPy C API,
so when we compile them with newer version of NumPy this generate
warnings.

In Theano we compile many different module during the user script. So
this cause many warning being printed to the users, so we updated
Theano to work with newer and older NumPy without warning.

As we disable the old NumPy C API, we are not able to compile the
cython generated code. So I manually updated the cython code to work
with the new and old api.

I did a post on the numpy mailing list that explain how I did this, in
case someone want to tackle this problem in Cython:
http://mail.scipy.org/pipermail/numpy-discussion/2013-November/068209.html

Frédéric Bastien
Lisandro Dalcin | 1 Nov 14:16 2013
Picon

[Cython] Files with executable bit on

This is in branch 0.19.x. These files should not have the executable
bit on. Can any of you please fix them?

[dalcinl <at> macarena cython-dev]$ ls -al $(find Cython -name '*.py') | grep rwx
-rwxr-xr-x  1 dalcinl  staff    4286 Feb 14  2013
Cython/Build/BuildExecutable.py
-rwxr-xr-x  1 dalcinl  staff  420289 Oct 15 23:39 Cython/Compiler/ExprNodes.py
-rwxr-xr-x  1 dalcinl  staff  137439 Oct 15 23:39 Cython/Compiler/PyrexTypes.py

--

-- 
Lisandro Dalcin
---------------
CIMEC (UNL/CONICET)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1016)
Tel/Fax: +54-342-4511169
Stefan Behnel | 31 Oct 18:25 2013
Picon

"Stack" checker for undefined behaviour in C code

Hi,

I just came across this paper:

http://pdos.csail.mit.edu/~xi/papers/stack-sosp13.pdf

They describe an analysis tool that checks C code for bugs that exploit
undefined behaviour and that are thus up to the mercy of compiler
assumptions and "optimisations" to do the right thing or not. They made it
available on github:

https://github.com/xiw/stack/

If anyone wants to take the time to set it up for checking some Cython
generated code, I'd be interested to see if it finds something.

Stefan

--

-- 

--- 
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Robert Bradshaw | 17 Oct 08:29 2013
Picon

Cython wiki

I've cleaned out the spam and moved the wiki to
https://github.com/cython/cython/wiki Don't edit it just yet (as I
have at lest one more round of force-pushing conversion from moinmoin
files), but please let me know if something doesn't look right or it's
missing pages you expect to see. (Yes, I know some pages are failing
to render; still looking into that, but if you can see what the error
is please let me know.)

- Robert

--

-- 

--- 
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Stefan Behnel | 13 Oct 20:00 2013
Picon

Cython 0.19.2 released

Hi all,

I released Cython 0.19.2 today as a bug-fix release for the 0.19 series.

You can get it from here:

http://cython.org/release/Cython-0.19.2.tar.gz

http://cython.org/release/Cython-0.19.2.zip

Or from PyPI:
https://pypi.python.org/pypi/Cython/0.19.2

Release notes:
https://github.com/cython/cython/blob/c2ddf294e025646132f7c29fae609c2f3c010e78/CHANGES.rst

Documentation: http://docs.cython.org/

Have fun,

Stefan

--

-- 

--- 
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Stefan Behnel | 11 Oct 20:42 2013
Picon

[Cython] preparing 0.19.2 for this weekend

Hi,

I've started preparing a bug-fix release on the 0.19.x branch. If you have
any further changes that should go in, please speak up now. I'd like to get
it out by this Sunday at the latest (before PyCon-DE starts on Monday).

Stefan
Yury V. Zaytsev | 9 Oct 15:01 2013

[Cython] State of PyPy compatibility wrt. arrays, incl. NumPy

Hi,

I've been playing with my Cython-generated extension on PyPy, trying to
load it through CPyExt, and, surprisingly, after a few fixes to PyPy, it
generally seems to work.

However, it looks like the situation with whatever kind of arrays is
actually pretty gloomy at the moment.

First, PyPy supports Python array.arrays, but doesn't replicate Python
internals, so the extension even fails to load, since in PyPy
tp_basicsize == 16 and not 56.

Not sure what to make of this one... for my needs, that would be enough.
I guess PyPy developers have to be approached to see if it's practical
to expose CPython structure, or agree on a different API?

Second, they say that the new-style buffer access is broken and it's not
clear if they will support it at all & when this is going to happen.

Third, apparently, there is already some limited support for NumPy C-API
in NumPyPy, and it seems that there is interest in improving this
support to make it usable:

http://docs.scipy.org/doc/numpy/reference/c-api.array.html

Finally, GetItem on NumPyPy arrays, which will be very slow, but at
least should work, rather than not, is also broken. I think this will be
easiest of all to fix.

So my question is, has anyone been playing with PyPy recently? Is there
anyone interested in making Cython-generated code to work nicely with
CPyExt? What are the plans w.r.t. the points I mentioned?

I've noticed that PyPy jobs in Jenkins are disabled... not sure whether
this really means anything or not.

Thanks!

--

-- 
Sincerely yours,
Yury V. Zaytsev

Chuck Blake | 3 Oct 18:30 2013
Picon

Re: [Cython] Declaration syntax change

Greg Ewing wrote:
>What would be the benefit of this? You're proposing to change
>from something identical to C declaration syntax, which is
>second nature for a great many people, to something that
>looks deceptively like C syntax but isn't.
>
>I can't see that causing anything other than a massive
>amount of confusion, anguish and hair-tearing.

I would echo this, actually.  While the C declarator syntax takes
a while to get used to, there is actually a simple rationale for
why it is the way it is: just one set of operator associativity
and precedences.  The way [], *, and function call ()s bind do
not vary by context.  One doesn't have to remember multiple sets
of rules "tailored" for various contexts like declaration/typedef
definition or casting vs operator application.

I believe this is the "fact underlying" the reactions various folk
here are having of "initially seemingly clearer, but oh wait..what
about corner case?"  There is a simplicity embedded in the seemingly
backwards "operators you need to apply to get the base type" rule.

The manual translation of C headers/decls to Cython is another good
point.  I think that it's a big departure from the perhaps hard to
articulate "sweet spot" Cython/Pyrex are trying to hit straddling
the C & Python worlds.

Stefan Behnel wrote:
>Also, C is only second nature to some people. A great many people actually
>use Cython specifically to *avoid* having to write C.

This is a fair point, but Pyrex/Cython have a decade plus long
history here of being very close to C in this specific way,
which is kind of a central thing..almost like a different way
to spell the same type grammar.  It's easy after a short while
to visually see the translation.  With whole new syntax and maybe
parens in new, shiner places, I doubt seeing the mapping would be
so easy for "hard cases".  And easy cases are easy.

On a separate relase-number-tying note, I also feel like the
Cython 1.0 target (or even Pyrex 1.0) was always more about
compatibility with "whatever Python you throw at the Cython
compiler" than about finality to all the various extended
Cython syntax areas.  There are quite a few beyond-Py syntax
areas at this point.

So, *if* the conclusion is that there is a group will and solid
rationale to let Cython allow a user to use an *alternate* type
declaration sub-language/warnings/a wrapper/macro syntax, then I
don't see any 1.0-release relevance.  Cython grows just yet another
way to specify the same things at whatever rev is easy.  The 1.0
only matters if it's a strict backward incompatible proposal,
but I sense a bit of discord on this response chain about that.
Maybe changing "cdef" to be "cydef" would be a visual flag that
an entirely different syntax is afoot?  But then..sooo many ways
to specify types.

If you really want to be *that* different - arguably almost as
much as different as simply requiring Python3 and annotations
rather than having "cdef" at all, then why not go all the way
and full on require Py3?  Then a user confronts all the Py3 core
syntax change and Cy2.0 (or maybe Cy1.5 as 3.0/2 or whatever)
at the same time.  And, why, the changes are even related to
only-in-Py3 things to boot.  Just my two cents, anyway.

cb

Gmane