Julian Taylor | 13 Jan 19:39 2014

[Cython] remove timestamps from generated source files

Hi,
Cython currently places timestamps into all its generated C source files.
This may be ok for developers who normally only rebuild files they
changed anyway but it is a major annoyance for distribution packagers
who often have to rebuild software from scratch including re-cythonizing.
The reason this is annoying is that the adding of timestamps breaks
caching of compiler results e.g. with ccache. Due to the size of the
generated files recompiling the full sources normally takes a large
fraction of the build time of packages.

I'm not sure what purpose the timestamps really serve, the version
number also included should be sufficient information in order to deduce
its origin.

Would it be possible to change Cython to stop putting timestamps into
the source?
I would also be happy if the granularity is reduced from seconds to
hours/days or an non-default option to disable it (e.g. an environment
variable).

Additionally I think it might be useful to have a vendor id added to the
version number added to the source files. This would allow easier origin
tracking of files created with distribution patched versions of Cython.
E.g. if Debian patches 0.20 of Cython it puts in

 Generated by Cython 0.20 (Debian revision 3)

Where the vendor id is added via a Distribution patch of the package and
none is emitted for upstream builds.
(CC. Debian maintainer of Cython for comments)
(Continue reading)

Julian Taylor | 12 Jan 21:12 2014

[Cython] h5py build broken by 0.20b2

Hi,
a h5py cython file fails to build with cython git head and 0.20b2.
It works with older versions.
This file fails

The offending file seems to be:
https://github.com/h5py/h5py/blob/master/h5py/h5p.pyd

cython h5py/h5p.pyx
Error compiling Cython file:
------------------------------------------------------------
...
from _objects cimport ObjectID

# --- Base classes ---

cdef class PropID(ObjectID):
    """ Base class for all property lists """
   ^
------------------------------------------------------------
h5py/h5p.pxd:17:4: Executable statement not allowed here
...

it was introduced in cython around this commit:
b6b5152f386ddae503674cc26200a547f3b4c8b0
properly handle expressions at the beginning of func/class/etc. blocks

Is this an intentional change? I did not see anything related in the
CHANGES.rst of the 0.20.x branch.
Please advise if this is a bug/regression in cython or
(Continue reading)

Julian Taylor | 12 Jan 20:35 2014

[Cython] libimobiledevice build broken by 0.20b2

Hi,
libimobiledevice cython file fails to build with cython git head and 0.20b2.
It works with older versions, the bad commit in cython is:
commit db5d5a41b852fa32876688198e3d2d46f12de858
Author: Robert Bradshaw <robertwb@...>
Date:   Mon Dec 30 22:24:04 2013 -0800

    Fix bug when base classes were declared out-of-order.

its parent commit succeeds the build.

the file is:
https://github.com/libimobiledevice/libimobiledevice/blob/master/cython/imobiledevice.pyx

build can be run with
./autogen.sh
./configure --disable-openssl
make CYTHON=/path/to/cython

imobiledevice.c:13521:88: error: 'struct
__pyx_obj_13imobiledevice_MobileSyncClient' has no member named '__pyx_vtab'
imobiledevice.c:13644:88: error: 'struct
__pyx_obj_13imobiledevice_MobileSyncClient' has no member named '__pyx_vtab'
imobiledevice.c:13783:88: error: 'struct
__pyx_obj_13imobiledevice_MobileSyncClient' has no member named '__pyx_vtab'
imobiledevice.c:13906:88: error: 'struct
__pyx_obj_13imobiledevice_MobileSyncClient' has no member named '__pyx_vtab'
imobiledevice.c:14029:88: error: 'struct
__pyx_obj_13imobiledevice_MobileSyncClient' has no member named '__pyx_vtab'
imobiledevice.c:14345:88: error: 'struct
(Continue reading)

Simon Jagoe | 8 Jan 21:17 2014

[Cython] cdef class tp_new should call PyBaseObject_Type.tp_new?

Hi all,

I recently posted to the cython-users list about this problem. This
email is to submita potential patch to fix the issue. The issue is as
follows:

At Enthought we are busy porting Traits
(http://github.com/enthought/traits) from C to Cython.

An issue has come up that that prevents classes inheriting from a
cdef-class from playing nicely with ABCs.  It is type's tp_new that
checks for abstractmethods that have not been implemented, so if type's
tp_new is bypassed, the check is never run and you can instantiate an
abstract base class.

It is equivalent to this simple example below (taken from
http://stackoverflow.com/questions/20432335). If you replace the simple
class A with a Cython cdef class, the effect is the same.

import abc

class A(object):
    def __new__(cls):
        # self = object.__new__(cls)
        return 42

class B(A):
    __metaclass__ = abc.ABCMeta

     <at> abc.abstractmethod
(Continue reading)

[Cython] Possible bug in Python string to C++ string conversion

The following extension module crashes when passed a non-string argument rather than throwing a TypeError:

 

************* string_bug.pyx *****************

from libcpp.string cimport string

 

cdef extern from "stdio.h":

   int printf(char *format, ...) nogil

 

def blow_up(string text):

    printf(text.c_str())

 

*****************************************

 

To see this run

 

************* show_bug.py *****************

import string_bug

 

# This is O.K.

string_bug.blow_up("Testing...")

 

# This seg faults for me.

string_bug.blow_up(1)

*****************************************

 

Here is my setup.py for your convenience:

 

************** setup.py *******************

from distutils.core import setup

from distutils.extension import Extension

from Cython.Distutils import build_ext

 

setup(cmdclass={'build_ext': build_ext},

      ext_modules=[Extension("string_bug",

                             sources=["string_bug.pyx"],

                             language="c++")])

******************************************

 

Also, I have:

 

> python

Python 2.7.5 (default, May 15 2013, 22:43:36) [MSC v.1500 32 bit (Intel)] on win32

Type "help", "copyright", "credits" or "license" for more information.

 

>>> import Cython

>>> Cython.__version__

'0.19.1'

 

Cheers,

Gerard

This e-mail and any attachments are confidential, may contain legal,
professional or other privileged information, and are intended solely for the
addressee. If you are not the intended recipient, do not use the information
in this e-mail in any way, delete this e-mail and notify the sender. -EXCIP

<div>
<div class="WordSection1">
<p class="MsoNormal">The following extension module crashes when passed a non-string argument rather than throwing a TypeError:<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">************* string_bug.pyx *****************≤p></p></p>
<p class="MsoNormal">from libcpp.string cimport string<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">cdef extern from "stdio.h":<p></p></p>
<p class="MsoNormal">&nbsp;&nbsp; int printf(char *format, ...) nogil<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">def blow_up(string text):<p></p></p>
<p class="MsoNormal">&nbsp;&nbsp;&nbsp; printf(text.c_str())<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">*****************************************≤p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">To see this run<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">************* show_bug.py *****************≤p></p></p>
<p class="MsoNormal">import string_bug<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal"># This is O.K.<p></p></p>
<p class="MsoNormal">string_bug.blow_up("Testing...")<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal"># This seg faults for me.<p></p></p>
<p class="MsoNormal">string_bug.blow_up(1)<p></p></p>
<p class="MsoNormal">*****************************************≤p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">Here is my setup.py for your convenience:<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">************** setup.py *******************<p></p></p>
<p class="MsoNormal">from distutils.core import setup<p></p></p>
<p class="MsoNormal">from distutils.extension import Extension<p></p></p>
<p class="MsoNormal">from Cython.Distutils import build_ext<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">setup(cmdclass={'build_ext': build_ext},<p></p></p>
<p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ext_modules=[Extension("string_bug",<p></p></p>
<p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; sources=["string_bug.pyx"],<p></p></p>
<p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; language="c++")])<p></p></p>
<p class="MsoNormal">******************************************<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">Also, I have:<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">&gt; python<p></p></p>
<p class="MsoNormal">Python 2.7.5 (default, May 15 2013, 22:43:36) [MSC v.1500 32 bit (Intel)] on win32<p></p></p>
<p class="MsoNormal">Type "help", "copyright", "credits" or "license" for more information.<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">&gt;&gt;&gt; import Cython<p></p></p>
<p class="MsoNormal">&gt;&gt;&gt; Cython.__version__<p></p></p>
<p class="MsoNormal">'0.19.1'<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">Cheers,<p></p></p>
<p class="MsoNormal">Gerard<p></p></p>
</div>
<p>This e-mail and any attachments are confidential, may contain legal, <br>professional or other privileged information, and are intended solely for the <br>addressee.  If you are not the intended recipient, do not use the information <br>in this e-mail in any way, delete this e-mail and notify the sender. -EXCIP</p>
</div>
Stefan Behnel | 3 Jan 19:22 2014
Picon

[Cython] "relaxed_strides" test broken with NumPy 1.8

Hi,

I enabled the NumPy build for our Py3.3 test runs and while I was at it, I
got it to use the latest NumPy release 1.8. This made one of the tests fail:

"""
    Traceback (most recent call last):
      File ".../doctest.py", line 1313, in __run
        compileflags, 1), test.globs)
      File "<doctest relaxed_strides.__test__.test_one_sized (line
29)[3]>", line 1, in <module>
        test_one_sized(a)[0]
      File "relaxed_strides.pyx", line 38, in
relaxed_strides.test_one_sized (relaxed_strides.cpp:1414)
      File "stringsource", line 622, in View.MemoryView.memoryview_cwrapper
(relaxed_strides.cpp:7568)
      File "stringsource", line 327, in
View.MemoryView.memoryview.__cinit__ (relaxed_strides.cpp:3717)

    ValueError: ndarray is not C-contiguous
"""

https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests/1787/ARCH=m64,BACKEND=cpp,PYVERSION=py33m/console

According to the comments in the test file and the corresponding NumPy pull
request, this seems to be somewhat expected.

https://github.com/cython/cython/blob/master/tests/memoryview/relaxed_strides.pyx

https://github.com/numpy/numpy/pull/3162

Does someone know enough about this to figure out what to do?

Stefan
Vinay Sajip | 16 Dec 11:09 2013
Picon

[Cython] Seemingly incorrect specialization of templates

Given this code in Cython/Includes/libcpp/utility.pxd:

cdef extern from "<utility>" namespace "std":
    cdef cppclass pair[T, U]:
        T first
        U second
        # rest omitted

when the following code in Cython/Includes/libcpp/map.pxd is seen:

from utility cimport pair

cdef extern from "<map>" namespace "std":
    cdef cppclass map[T, U]:
        cppclass iterator:
            pair[T, U]& operator*() nogil
            # rest omitted

The line "pair[T, U]& operator*() nogil" causes a specialization
of pair[T, U] to be created, but using placeholders rather than actual
types. This seems wrong - can someone here confirm whether this is
intentional?

This led to a problem where the generated C++ code for a genuine
specialisation instead generated a declaration like "pair<T, U>" instead
of using the actual arguments. Given this code in
Cython/Includes/libcpp/utility.pxd:

cdef extern from "<utility>" namespace "std":
    cdef cppclass pair[T, U]:
        T first
        U second
        # rest omitted

when the following code in Cython/Includes/libcpp/map.pxd is seen:

from utility cimport pair

cdef extern from "<map>" namespace "std":
    cdef cppclass map[T, U]:
        cppclass iterator:
            pair[T, U]& operator*() nogil
            # rest omitted

The line "pair[T, U]& operator*() nogil" causes a specialization
of pair[T, U] to be created, but using placeholders rather than actual
types. This seems wrong - can someone here confirm whether this is
intentional?

This caused a problem where the generated C++ code for a genuine
specialisation instead generated a declaration like "pair<T, U>" instead
of using the actual arguments.Given this code in
Cython/Includes/libcpp/utility.pxd:

cdef extern from "<utility>" namespace "std":
    cdef cppclass pair[T, U]:
        T first
        U second
        # rest omitted

when the following code in Cython/Includes/libcpp/map.pxd is seen:

from utility cimport pair

cdef extern from "<map>" namespace "std":
    cdef cppclass map[T, U]:
        cppclass iterator:
            pair[T, U]& operator*() nogil
            # rest omitted

The line "pair[T, U]& operator*() nogil" causes a specialization
of pair[T, U] to be created, but using placeholders rather than actual
types. This seems wrong - can someone here confirm whether this is
intentional?

This caused a problem where the generated C++ code for a genuine
specialisation instead generated a declaration like "pair<T, U>" instead
of using the actual arguments.

The specialize call occurs in Nodes.py:TemplatedTypeNode.analyse, where the
template_types passed to specialize are actually all instances of
TemplatePlaceholderType.

Regards,

Vinay Sajip
Given this code in Cython/Includes/libcpp/utility.pxd:

cdef extern from "<utility>" namespace "std":
    cdef cppclass pair[T, U]:
        T first
        U second
        # rest omitted

when the following code in Cython/Includes/libcpp/map.pxd is seen:

from utility cimport pair

cdef extern from "<map>" namespace "std":
    cdef cppclass map[T, U]:
        cppclass iterator:
            pair[T, U]& operator*() nogil
            # rest omitted

The line "pair[T, U]& operator*() nogil" causes a specialization
of pair[T, U] to be created, but using placeholders rather than actual
types. This seems wrong - can someone here confirm whether this is
intentional?

This caused a problem where the generated C++ code for a genuine
specialisation instead generated a declaration like "pair<T, U>" instead
of using the actual arguments. This, of course, failed in g++.

The specialize call occurs in Nodes.py:TemplatedTypeNode.analyse, where the
template_types passed to specialize are actually all instances of
TemplatePlaceholderType. It seems like just the base_type could be used.

Regards,

Vinay Sajip

The specialize call occurs in Nodes.py:TemplatedTypeNode.analyse, where the
template_types passed to specialize are actually all instances of
TemplatePlaceholderType.

Regards,

Vinay Sajip

The specialize call occurs in Nodes.py:TemplatedTypeNode.analyse, where the
template_types passed to specialize are actually all instances of
TemplatePlaceholderType.

Regards,

Vinay Sajip

Stefan Behnel | 29 Nov 17:03 2013
Picon

"embedsignature" and "autotestdict" features in Py3.4

Hi,

recent changes in Py3.4 affect the functionality of the two directives
"embedsignature" and "autotestdict".

The so-called "argument clinic" extracts any embedded signatures from
docstrings and moves them into a new property "__text_signature__". This is
done at runtime, i.e. when either "__doc__" or "__text_signature__" are
requested.

http://hg.python.org/cpython/file/6a3e09cd96f3/Objects/methodobject.c#l182

I personally consider this a feature (and it works nicely with the
signatures that Cython embeds), but you may or may not agree. It broke some
of our own doctests, at least, because the "__doc__" value that we tested
for was no longer the same.

Regarding the "autotestdict", CPython also got smarter and now finds
doctests embedded in C implemented functions all by itself. This is clearly
an improvement compared to the previous behaviour. However, this also means
that it now executes both the doctest of the function itself and its copy
in the generated "__test__" dict (stored in the module under that name),
meaning that it executes all function doctests twice. This also broke some
of Cython's own doctests for us, which modified global state and thus
failed the second time.

I'm yet unsure if we should try to do something about this. Currently, the
explicit doctest dict is generated by default, so all code that uses
doctests in function docstrings suffers from this. We might be able to make
Cython a bit smarter so that these docstrings are left out of the test dict
when (C-)compiling in CPython 3.4. Deciding which ones are still needed
might be a bit tricky, though...

Stefan

--

-- 

--- 
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Yury V. Zaytsev | 29 Nov 14:26 2013

[Cython] 'Referenced before assignment' warning triggered due to OpenMP directives?

Hi,

I'm trying to find out whether the number of threads has been actually
set to what I wanted it to be, but the compiler tells me that I'm trying
to reference a variable before assignment. This doesn't happen when I
simply release the GIL, but only within a parallel section.

Is it a problem with Cython or I'm missing something obvious?

Thanks!

$ cat ompt.pyx 
cimport openmp
from cython.parallel import parallel, prange

cdef Py_ssize_t x = 1

with nogil, parallel():
    x = openmp.omp_get_num_threads()

print("Number of OpenMP threads is '{}'!".format(x))

$ cython ompt.pyx 

Error compiling Cython file:
------------------------------------------------------------
...
cdef Py_ssize_t x = 1

with nogil, parallel():
    x = openmp.omp_get_num_threads()

print("Number of OpenMP threads is '{}'!".format(x))
                                                 ^
------------------------------------------------------------

ompt.pyx:9:50: local variable 'x' referenced before assignment

--

-- 
Sincerely yours,
Yury V. Zaytsev

Daniele Nicolodi | 26 Nov 16:18 2013
Picon

[Cython] BUG: assignment to typed memoryview

Hello,

I believe there is a bug in how assignment to typed memoryviews is
handled in the compiler.  I think the following code should be valid code:

  cdef double[::1] a = np.arange(10, dtype=np.double)
  cdef double[::1] b = np.empty(a.size // 2)
  b[:] = a[::2]

However, the Cython compiler complains with:

  Memoryview 'double[:]' not conformable to memoryview 'double[::1]'.

A similar thing happens with memoryview copies:

  cdef double[::1] a = np.arange(10, dtype=np.double)
  cdef double[::1] b
  b = a[::2].copy()

In both cases I would expect Cython to copy the non contiguous elements
of the source array to the destination array (or to a newly created
array in the second case).  The copy() method is defined here
https://github.com/cython/cython/blob/master/Cython/Utility/MemoryView.pyx#L592
and (while I haven't spent much time analyzing it in detail) it seems to
do the right thing.

Indeed, if I short-circuit the src_conforms_to_dst() function
https://github.com/cython/cython/blob/master/Cython/Compiler/MemoryView.py#L141
the code compiles and runs correctly in both cases.

I don't know much about how the Cython compiler works, therefore I don't
know where to dig for the source of this problem.  Can someone point me
in the right direction?

Thanks. Cheers,
Daniele
Vinay Sajip | 25 Nov 11:35 2013
Picon

[Cython] Cython crash when using forward declarations

The following Cython source:

cdef extern from "header":
    cdef cppclass String
    cdef cppclass List[T]
    cdef cppclass StringList(List[String])

    cdef cppclass String:
        String()
        int length()

    cdef cppclass List[T]:
        List()
        int length()

    cdef cppclass StringList(List[String]):
        StringList()
        void sort()

causes a crash:

Error compiling Cython file:
------------------------------------------------------------
...

    cdef cppclass List[T]:
        List()
        int length()

    cdef cppclass StringList(List[String]):
        ^
------------------------------------------------------------

s.pyx:14:9: Cannot inherit from incomplete type

Error compiling Cython file:
------------------------------------------------------------
...
    cdef cppclass List[T]:
        List()
        int length()

    cdef cppclass StringList(List[String]):
        StringList()
       ^
------------------------------------------------------------

s.pyx:15:8: Compiler crash in AnalyseDeclarationsTransform

File 'ModuleNode.py', line 101, in analyse_declarations: ModuleNode(s.pyx:1:0,
    full_module_name = 's')
File 'Nodes.py', line 382, in analyse_declarations: StatListNode(s.pyx:1:5)
File 'Nodes.py', line 438, in analyse_declarations: CDefExternNode(s.pyx:1:5,
    include_file = u'header')
File 'Nodes.py', line 382, in analyse_declarations: StatListNode(s.pyx:2:4)
File 'Nodes.py', line 1313, in analyse_declarations: CppClassNode(s.pyx:14:9,
    base_classes = [...]/1,
    name = u'StringList',
    visibility = u'extern')
File 'Nodes.py', line 1203, in analyse_declarations: CVarDefNode(s.pyx:15:8,
    modifiers = [...]/0,
    visibility = u'extern')

Compiler crash traceback from this point on:
  File ".../Cython/Compiler/Nodes.py", line 1203, in analyse_declarations
    api = self.api, modifiers = self.modifiers)
  File ".../Cython/Compiler/Symtab.py", line 2131, in declare_cfunction
    self.check_base_default_constructor(pos)
  File ".../Cython/Compiler/Symtab.py", line 2108, in \
                                            check_base_default_constructor
    temp_entry = base_class.scope.lookup_here("<init>")
AttributeError: 'NoneType' object has no attribute 'lookup_here'

There seem to be two problems here: the AttributeError crash, but also the
earlier "Cannot inherit from incomplete type" message. None of the types look
incomplete; if you comment out the forward declarations, no error is reported:
so it looks as if the forward declarations aren't being tied up with the actual
declarations. In the actual use case, of course, I need the forward decls.

Regards,

Vinay Sajip


Gmane