Stefan Behnel | 23 Jan 20:49 2014
Picon

[Cython] C code churn and benchmark diffs

Hi,

to get a better idea of what new optimisations (or C code changes in
general) actually bring, I've added a diffing step to the benchmark runner
jobs. The bigger ones now generate an additional output file
"cfiles.diff.gz" that contains a diff of the C files generated in the
current run against those of the last (successful) run. E.g. here:

https://sage.math.washington.edu:8091/hudson/view/bench/job/cython-devel-pybenchmarks-neu-py3k/

(That one should start looking a lot smaller tomorrow :)

While doing that, I noticed that there are a couple of unnecessary
differences in the C files that are due to dict iteration, which is
randomised in Py3.3+ (for security reasons). I fixed (most of?) them, so
repeated Cython runs should now generate pretty much identical output
files, if you strip the file header comment (diff can do that). The rest is
generated deterministically in source code order already.

It should be possible to do the same thing for the Sage build, I guess.
That would also be interesting to look at from time to time, especially
when things start failing for some reason.

Stefan
Syam Gadde | 21 Jan 23:00 2014

[Cython] segfault due to using DECREF instead of XDECREF

(apologies to the moderator for multiple postings -- I'm trying to figure out which of my email addresses is
actually being exported to the public!)

Hi,

It seems that cython's static code analysis is failing on the following
code.  It thinks that a particular variable (myobject2) must not be
NULL, but in fact it is, so the __Pyx_DECREF_SET segfaults.  This came
from third-party code, so I could fix it with an explicit initialization
to None, but I'd love to have Cython deal with it directly.  Thanks for
your help!

#<<<BEGIN tmpnone.py>>>
# When compiled with Cython 0.19.2 or 0.20dev, the following code
# segfaults on import

import random

def myfunc():
      ## uncommenting the following fixes things
      #myobject2 = None

      myfalse = random.random() > 2

      if myfalse:
          myobject = None
          myobject2 = None

      if not myfalse:
          # here Cython uses __Pyx_XDECREF_SET
(Continue reading)

Julian Taylor | 22 Jan 00:03 2014

[Cython] bytearray tests fail with default unsigned char

hi,
the bytearray tests are broken when chars are unsigned.

tests/run/bytearraymethods.pyx defines following function:
def bytearray_append(b, char c, int i, object o):

this gets converted to an effective __Pyx_PyInt_AsUnsignedChar which
then errors out when -1 is passed in.
chars are unsigned like they are by default on arm, s390x and powerpc.
This causes a couple build failures in debian:
https://buildd.debian.org/status/package.php?p=cython
https://buildd.debian.org/status/fetch.php?pkg=cython&arch=armel&ver=0.20-1&stamp=1390316252

the tests can be fixed by adding signed char to the interface.

to reproduce on x86 with gcc (note the -funsigned-char to change the
default):

cython tests/run/bytearraymethods.pyx
gcc -funsigned-char bytearraymethods.c -fPIC $(python-config --includes)
$(python-config --libs) -shared -O2 -o bytearraymethods.so

python -c "import doctest; import bytearraymethods;
doctest.testmod(bytearraymethods)"

Traceback (most recent call last):
  File "<string>", line 1, in <module>
NameError: name 'bytearraymethods' is not defined
root <at> ubuntu:/# python -c "import doctest; import bytearraymethods;
doctest.testmod(bytearraymethods)"
(Continue reading)

Andriy Kornatskyy | 19 Jan 21:00 2014

[Cython] Compiler crash in RemoveUnreachableCode

The cython compiler crash report below. Steps to reproduce:

1. virtualenv env
2. env/bin/easy_install cython
3. env/bin/easy_install wheezy.http

The wheezy.http has dependency on wheezy.core. If I install those two packages separately there is no
error, only if through dependencies.

Thanks.

Andriy Kornatskyy

  File "Visitor.py", line 178, in Cython.Compiler.Visitor.TreeVisitor._visit (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:4437)
  File "Visitor.py", line 137, in Cython.Compiler.Visitor.TreeVisitor._raise_compiler_error (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:3655)
Cython.Compiler.Errors.CompilerCrash:
Error compiling Cython file:
------------------------------------------------------------
...

"""
"""

__version__ = '0.1.129'
             ^
------------------------------------------------------------

src/wheezy/core/__init__.py:5:14: Compiler crash in RemoveUnreachableCode

ModuleNode.body = StatListNode(__init__.py:5:14)
(Continue reading)

Robert Byrnes | 16 Jan 18:43 2014
Picon

[Cython] assignment to reference argument bug, revisited

Are there any plans to fix the "Assignment to reference" bug mentioned here?

https://groups.google.com/forum/?fromgroups=#!topic/cython-users/j58Sp3QMrD4

> Oh, yes, of course. Good old C++, where local T& x is unassignable but
> assigning to an argument T& x is a common idiom. This is a bug in Cython.

This is still broken in 0.19.2, and a casual inspection of 0.20rc1
suggests that it is broken there as well.

Is the fix as simple as this?

--- ExprNodes.py~       2014-01-16 12:31:08.377573000 -0500
+++ ExprNodes.py        2014-01-16 12:30:56.945501000 -0500
 <at>  <at>  -1605,7 +1605,7  <at>  <at>  class NameNode(AtomicExprNode):

         if self.type.is_const:
             error(self.pos, "Assignment to const '%s'" % self.name)
-        if self.type.is_reference:
+        if self.type.is_reference and not self.entry.is_arg:
             error(self.pos, "Assignment to reference '%s'" % self.name)
         if not self.is_lvalue():
             error(self.pos, "Assignment to non-lvalue '%s'"

This seems to work for me, but ...

I wonder if it is also correct to set self.entry.cf_unused = True
for the case of assignment to reference arguments, to avoid CYTHON_UNUSED
annotations in the function definitions and unused variable warnings.

(Continue reading)

Julian Taylor | 13 Jan 19:39 2014

[Cython] remove timestamps from generated source files

Hi,
Cython currently places timestamps into all its generated C source files.
This may be ok for developers who normally only rebuild files they
changed anyway but it is a major annoyance for distribution packagers
who often have to rebuild software from scratch including re-cythonizing.
The reason this is annoying is that the adding of timestamps breaks
caching of compiler results e.g. with ccache. Due to the size of the
generated files recompiling the full sources normally takes a large
fraction of the build time of packages.

I'm not sure what purpose the timestamps really serve, the version
number also included should be sufficient information in order to deduce
its origin.

Would it be possible to change Cython to stop putting timestamps into
the source?
I would also be happy if the granularity is reduced from seconds to
hours/days or an non-default option to disable it (e.g. an environment
variable).

Additionally I think it might be useful to have a vendor id added to the
version number added to the source files. This would allow easier origin
tracking of files created with distribution patched versions of Cython.
E.g. if Debian patches 0.20 of Cython it puts in

 Generated by Cython 0.20 (Debian revision 3)

Where the vendor id is added via a Distribution patch of the package and
none is emitted for upstream builds.
(CC. Debian maintainer of Cython for comments)
(Continue reading)

Julian Taylor | 12 Jan 21:12 2014

[Cython] h5py build broken by 0.20b2

Hi,
a h5py cython file fails to build with cython git head and 0.20b2.
It works with older versions.
This file fails

The offending file seems to be:
https://github.com/h5py/h5py/blob/master/h5py/h5p.pyd

cython h5py/h5p.pyx
Error compiling Cython file:
------------------------------------------------------------
...
from _objects cimport ObjectID

# --- Base classes ---

cdef class PropID(ObjectID):
    """ Base class for all property lists """
   ^
------------------------------------------------------------
h5py/h5p.pxd:17:4: Executable statement not allowed here
...

it was introduced in cython around this commit:
b6b5152f386ddae503674cc26200a547f3b4c8b0
properly handle expressions at the beginning of func/class/etc. blocks

Is this an intentional change? I did not see anything related in the
CHANGES.rst of the 0.20.x branch.
Please advise if this is a bug/regression in cython or
(Continue reading)

Julian Taylor | 12 Jan 20:35 2014

[Cython] libimobiledevice build broken by 0.20b2

Hi,
libimobiledevice cython file fails to build with cython git head and 0.20b2.
It works with older versions, the bad commit in cython is:
commit db5d5a41b852fa32876688198e3d2d46f12de858
Author: Robert Bradshaw <robertwb@...>
Date:   Mon Dec 30 22:24:04 2013 -0800

    Fix bug when base classes were declared out-of-order.

its parent commit succeeds the build.

the file is:
https://github.com/libimobiledevice/libimobiledevice/blob/master/cython/imobiledevice.pyx

build can be run with
./autogen.sh
./configure --disable-openssl
make CYTHON=/path/to/cython

imobiledevice.c:13521:88: error: 'struct
__pyx_obj_13imobiledevice_MobileSyncClient' has no member named '__pyx_vtab'
imobiledevice.c:13644:88: error: 'struct
__pyx_obj_13imobiledevice_MobileSyncClient' has no member named '__pyx_vtab'
imobiledevice.c:13783:88: error: 'struct
__pyx_obj_13imobiledevice_MobileSyncClient' has no member named '__pyx_vtab'
imobiledevice.c:13906:88: error: 'struct
__pyx_obj_13imobiledevice_MobileSyncClient' has no member named '__pyx_vtab'
imobiledevice.c:14029:88: error: 'struct
__pyx_obj_13imobiledevice_MobileSyncClient' has no member named '__pyx_vtab'
imobiledevice.c:14345:88: error: 'struct
(Continue reading)

Simon Jagoe | 8 Jan 21:17 2014

[Cython] cdef class tp_new should call PyBaseObject_Type.tp_new?

Hi all,

I recently posted to the cython-users list about this problem. This
email is to submita potential patch to fix the issue. The issue is as
follows:

At Enthought we are busy porting Traits
(http://github.com/enthought/traits) from C to Cython.

An issue has come up that that prevents classes inheriting from a
cdef-class from playing nicely with ABCs.  It is type's tp_new that
checks for abstractmethods that have not been implemented, so if type's
tp_new is bypassed, the check is never run and you can instantiate an
abstract base class.

It is equivalent to this simple example below (taken from
http://stackoverflow.com/questions/20432335). If you replace the simple
class A with a Cython cdef class, the effect is the same.

import abc

class A(object):
    def __new__(cls):
        # self = object.__new__(cls)
        return 42

class B(A):
    __metaclass__ = abc.ABCMeta

     <at> abc.abstractmethod
(Continue reading)

[Cython] Possible bug in Python string to C++ string conversion

The following extension module crashes when passed a non-string argument rather than throwing a TypeError:

 

************* string_bug.pyx *****************

from libcpp.string cimport string

 

cdef extern from "stdio.h":

   int printf(char *format, ...) nogil

 

def blow_up(string text):

    printf(text.c_str())

 

*****************************************

 

To see this run

 

************* show_bug.py *****************

import string_bug

 

# This is O.K.

string_bug.blow_up("Testing...")

 

# This seg faults for me.

string_bug.blow_up(1)

*****************************************

 

Here is my setup.py for your convenience:

 

************** setup.py *******************

from distutils.core import setup

from distutils.extension import Extension

from Cython.Distutils import build_ext

 

setup(cmdclass={'build_ext': build_ext},

      ext_modules=[Extension("string_bug",

                             sources=["string_bug.pyx"],

                             language="c++")])

******************************************

 

Also, I have:

 

> python

Python 2.7.5 (default, May 15 2013, 22:43:36) [MSC v.1500 32 bit (Intel)] on win32

Type "help", "copyright", "credits" or "license" for more information.

 

>>> import Cython

>>> Cython.__version__

'0.19.1'

 

Cheers,

Gerard

This e-mail and any attachments are confidential, may contain legal,
professional or other privileged information, and are intended solely for the
addressee. If you are not the intended recipient, do not use the information
in this e-mail in any way, delete this e-mail and notify the sender. -EXCIP

<div>
<div class="WordSection1">
<p class="MsoNormal">The following extension module crashes when passed a non-string argument rather than throwing a TypeError:<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">************* string_bug.pyx *****************≤p></p></p>
<p class="MsoNormal">from libcpp.string cimport string<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">cdef extern from "stdio.h":<p></p></p>
<p class="MsoNormal">&nbsp;&nbsp; int printf(char *format, ...) nogil<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">def blow_up(string text):<p></p></p>
<p class="MsoNormal">&nbsp;&nbsp;&nbsp; printf(text.c_str())<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">*****************************************≤p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">To see this run<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">************* show_bug.py *****************≤p></p></p>
<p class="MsoNormal">import string_bug<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal"># This is O.K.<p></p></p>
<p class="MsoNormal">string_bug.blow_up("Testing...")<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal"># This seg faults for me.<p></p></p>
<p class="MsoNormal">string_bug.blow_up(1)<p></p></p>
<p class="MsoNormal">*****************************************≤p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">Here is my setup.py for your convenience:<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">************** setup.py *******************<p></p></p>
<p class="MsoNormal">from distutils.core import setup<p></p></p>
<p class="MsoNormal">from distutils.extension import Extension<p></p></p>
<p class="MsoNormal">from Cython.Distutils import build_ext<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">setup(cmdclass={'build_ext': build_ext},<p></p></p>
<p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ext_modules=[Extension("string_bug",<p></p></p>
<p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; sources=["string_bug.pyx"],<p></p></p>
<p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; language="c++")])<p></p></p>
<p class="MsoNormal">******************************************<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">Also, I have:<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">&gt; python<p></p></p>
<p class="MsoNormal">Python 2.7.5 (default, May 15 2013, 22:43:36) [MSC v.1500 32 bit (Intel)] on win32<p></p></p>
<p class="MsoNormal">Type "help", "copyright", "credits" or "license" for more information.<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">&gt;&gt;&gt; import Cython<p></p></p>
<p class="MsoNormal">&gt;&gt;&gt; Cython.__version__<p></p></p>
<p class="MsoNormal">'0.19.1'<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">Cheers,<p></p></p>
<p class="MsoNormal">Gerard<p></p></p>
</div>
<p>This e-mail and any attachments are confidential, may contain legal, <br>professional or other privileged information, and are intended solely for the <br>addressee.  If you are not the intended recipient, do not use the information <br>in this e-mail in any way, delete this e-mail and notify the sender. -EXCIP</p>
</div>
Stefan Behnel | 3 Jan 19:22 2014
Picon

[Cython] "relaxed_strides" test broken with NumPy 1.8

Hi,

I enabled the NumPy build for our Py3.3 test runs and while I was at it, I
got it to use the latest NumPy release 1.8. This made one of the tests fail:

"""
    Traceback (most recent call last):
      File ".../doctest.py", line 1313, in __run
        compileflags, 1), test.globs)
      File "<doctest relaxed_strides.__test__.test_one_sized (line
29)[3]>", line 1, in <module>
        test_one_sized(a)[0]
      File "relaxed_strides.pyx", line 38, in
relaxed_strides.test_one_sized (relaxed_strides.cpp:1414)
      File "stringsource", line 622, in View.MemoryView.memoryview_cwrapper
(relaxed_strides.cpp:7568)
      File "stringsource", line 327, in
View.MemoryView.memoryview.__cinit__ (relaxed_strides.cpp:3717)

    ValueError: ndarray is not C-contiguous
"""

https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests/1787/ARCH=m64,BACKEND=cpp,PYVERSION=py33m/console

According to the comments in the test file and the corresponding NumPy pull
request, this seems to be somewhat expected.

https://github.com/cython/cython/blob/master/tests/memoryview/relaxed_strides.pyx

https://github.com/numpy/numpy/pull/3162

Does someone know enough about this to figure out what to do?

Stefan

Gmane