William Stein | 26 Sep 03:05 2013
Picon

[Cython] cython wiki

Hello,

I've permanently disabled the Cython wiki, due to all the spam it
contains, and that whenever I try to start it, the
moinmoin/twisted/python processes instantly just burns lots of CPU and
stops responding to requests.   If anybody would like to move the wiki
to github, google code, bitbucket, etc., please send me an email and I
will make the complete tarball of the wiki available to you.
(wstein@...)

    -- William

--

-- 
William Stein
Professor of Mathematics
University of Washington
http://wstein.org
Daniel, Bruno | 18 Sep 15:14 2013

[Cython] Memory leak of memoryview attributes in cdef subclasses

Dear developers,

We encountered memory leaks when using memoryview attributes in cdef subclasses
in Cython code. They can be avoided by adding a dealloc method setting the value
of the memoryview attribute to None. The problem does not occur in topmost
cdef classes.

Here's an example: (See the module docstring on how to compile and run the
example.)

--- File memoryview_leak.pyx --------------------------------------------------
"""
Cython module exposing memory leaks due to memoryview attributes in cdef
classes

Compile::

    python setup.py build_ext --inplace

Run::

    python -c "import memoryview_leak; memoryview_leak.main()"
"""
from __future__ import print_function, division

import unittest
import gc

import numpy as np

(Continue reading)

Wes McKinney | 16 Sep 20:48 2013
Picon

[Cython] Improving compile times for Cython extensions with C sources

I haven't been able to find an easy answer to this question, but here is my problem:

Cython file: extension.pyx
Declarations / C imports: extension.pxd
C dependencies:
a.c
b.c
c.c

These are all compiled together to produce extension.so.

The problem is, when I modify extension.pyx, all 4 C files are recompiled, even if the 3 straight C files are untouched. 

Any way to fix the build system to not do this? Seems that the .o files from the C dependencies could be easily reused.

thanks,
Wes
<div><div dir="ltr">I haven't been able to find an easy answer to this question, but here is my problem:<div><br></div>
<div>Cython file: extension.pyx</div>
<div>Declarations / C imports: extension.pxd</div>
<div>C dependencies:</div>

<div>a.c</div>
<div>b.c</div>
<div>c.c</div>
<div><br></div>
<div>These are all compiled together to produce extension.so.</div>
<div><br></div>
<div>The problem is, when I modify extension.pyx, all 4 C files are recompiled, even if the 3 straight C files are untouched.&nbsp;</div>

<div><br></div>
<div>Any way to fix the build system to not do this? Seems that the .o files from the C dependencies could be easily reused.</div>
<div><br></div>
<div>thanks,</div>
<div>Wes</div>
</div></div>
Emmanuel Gil Peyrot | 13 Sep 00:10 2013
Picon

[Cython] Assign from array to scalar

Hi,

A few weeks ago I started an implementation of that simple usecase:

 cdef long a[2], x, y
 # set a[0] and a[1] to some useful values
 x, y = a

It does works, but I’m not used to the interactions between the
different parts of the code, I mainly hacked away where I could.  I’m
especially dissatisfied with the code duplication I came with (to check
the correct types and size) and the location of the checks.

I added testcases for both the array assignation and the slices of that
same assignation I plan to implement, which seems to integrate
badly with my current implementation, making the code even more
unreadable.

I tried to find a commit implementing a similar feature so I could get
some inspiration but couldn’t find one.  If someone has a hint about
that I’d be glad. :)

Btw, on a Cython.Compiler.ExprNodes.SliceIndexNode object, why does
stop_code() give an int while start_code() gives a string?  It’d be
much better to return an int in both cases, so one could interpolate
without conversion from both python or the generated C.

Preliminary patch joined.

-- 
Emmanuel Gil Peyrot
XMPP: <linkmauve@...>
OpenPGP: 24B1D609
diff --git a/Cython/Compiler/ExprNodes.py b/Cython/Compiler/ExprNodes.py
index 2511eaf..bf04ff1 100755
--- a/Cython/Compiler/ExprNodes.py
+++ b/Cython/Compiler/ExprNodes.py
 <at>  <at>  -5615,6 +5615,11  <at>  <at>  class SequenceNode(ExprNode):
     def generate_assignment_code(self, rhs, code):
         if self.starred_assignment:
             self.generate_starred_assignment_code(rhs, code)
+        elif (isinstance(self, TupleNode) and
+              isinstance(rhs, NameNode) and
+              isinstance(rhs.entry.type, PyrexTypes.CArrayType)):
+            self.generate_parallel_assignment_code2(rhs, code)
+            return
         else:
             self.generate_parallel_assignment_code(rhs, code)

 <at>  <at>  -5627,6 +5632,22  <at>  <at>  class SequenceNode(ExprNode):
             PyrexTypes.CFuncTypeArg("it", PyrexTypes.py_object_type, None),
             ]))

+    def generate_parallel_assignment_code2(self, rhs, code):
+        rhs_type = rhs.entry.type
+        base_type = rhs_type.base_type
+        size = rhs_type.size
+        lhs_args = self.args
+        assert len(lhs_args) == size
+        for arg in lhs_args:
+            if arg.entry.type is not base_type:
+                raise ValueError("Wrong type for parallel assignment of '%s' array: %s" %
+                                 (base_type, arg.entry.type))
+        for i, arg in enumerate(lhs_args):
+            code.putln("%s = %s[%s];" % (
+                            arg.result(),
+                            rhs.result(),
+                            i))
+
     def generate_parallel_assignment_code(self, rhs, code):
         # Need to work around the fact that generate_evaluation_code
         # allocates the temps in a rather hacky way -- the assignment
diff --git a/Cython/Compiler/Nodes.py b/Cython/Compiler/Nodes.py
index fe7adb3..7987cc6 100644
--- a/Cython/Compiler/Nodes.py
+++ b/Cython/Compiler/Nodes.py
 <at>  <at>  -4509,6 +4509,20  <at>  <at>  class SingleAssignmentNode(AssignmentNode):
         self.lhs = self.lhs.analyse_target_types(env)
         self.lhs.gil_assignment_check(env)

+        if (isinstance(self.lhs, ExprNodes.TupleNode) and
+            isinstance(self.rhs, ExprNodes.NameNode)):
+            rhs_type = self.rhs.entry.type
+            if isinstance(rhs_type, PyrexTypes.CArrayType):
+                base_type = rhs_type.base_type
+                size = rhs_type.size
+                lhs_args = self.lhs.args
+                assert len(lhs_args) == size
+                for arg in lhs_args:
+                    if arg.entry.type is not base_type:
+                        break
+                else:
+                    return self
+
         if self.lhs.memslice_broadcast or self.rhs.memslice_broadcast:
             self.lhs.memslice_broadcast = True
             self.rhs.memslice_broadcast = True
diff --git a/tests/run/arrayassign.pyx b/tests/run/arrayassign.pyx
index 29c353e..f76ad2e 100644
--- a/tests/run/arrayassign.pyx
+++ b/tests/run/arrayassign.pyx
 <at>  <at>  -128,6 +128,59  <at>  <at>  def test_ptr_literal_list_slice_end():
     a[:5] = [1,2,3,4,5]
     return (a[0], a[1], a[2], a[3], a[4])

+
+# The opposite, assignation from an array to multiple scalars.
+
+def test_array_to_scalars():
+    """
+    >>> test_array_to_scalars()
+    (1, 2, 3, 4, 5)
+    """
+    cdef int l[5], a, b, c, d, e
+    l[:] = [1,2,3,4,5]
+    a, b, c, d, e = l
+    return (a, b, c, d, e)
+
+def test_slice_all_to_scalars():
+    """
+    >>> test_slice_all_to_scalars()
+    (1, 2, 3, 4, 5)
+    """
+    cdef int l[5], a, b, c, d, e
+    l[:] = [1,2,3,4,5]
+    a, b, c, d, e = l[:]
+    return (a, b, c, d, e)
+
+def test_slice_start_to_scalars():
+    """
+    >>> test_slice_start_to_scalars()
+    (1, 2, 3, 4, 5)
+    """
+    cdef int l[7], a, b, c, d, e
+    l[:] = [6,7,1,2,3,4,5]
+    a, b, c, d, e = l[2:]
+    return (a, b, c, d, e)
+
+def test_slice_end_to_scalars():
+    """
+    >>> test_slice_end_to_scalars()
+    (1, 2, 3, 4, 5)
+    """
+    cdef int l[7], a, b, c, d, e
+    l[:] = [1,2,3,4,5,6,7]
+    a, b, c, d, e = l[:5]
+    return (a, b, c, d, e)
+
+def test_slice_start_end_to_scalars():
+    """
+    >>> test_slice_start_end_to_scalars()
+    (1, 2, 3, 4, 5)
+    """
+    cdef int l[9], a, b, c, d, e
+    l[:] = [6,7,1,2,3,4,5,8,9]
+    a, b, c, d, e = l[2:7]
+    return (a, b, c, d, e)
+
 # tuples aren't supported (yet)
 #
 #def test_literal_tuple():
diff --git a/Cython/Compiler/ExprNodes.py b/Cython/Compiler/ExprNodes.py
index 2511eaf..bf04ff1 100755
--- a/Cython/Compiler/ExprNodes.py
+++ b/Cython/Compiler/ExprNodes.py
 <at>  <at>  -5615,6 +5615,11  <at>  <at>  class SequenceNode(ExprNode):
     def generate_assignment_code(self, rhs, code):
         if self.starred_assignment:
             self.generate_starred_assignment_code(rhs, code)
+        elif (isinstance(self, TupleNode) and
+              isinstance(rhs, NameNode) and
+              isinstance(rhs.entry.type, PyrexTypes.CArrayType)):
+            self.generate_parallel_assignment_code2(rhs, code)
+            return
         else:
             self.generate_parallel_assignment_code(rhs, code)

 <at>  <at>  -5627,6 +5632,22  <at>  <at>  class SequenceNode(ExprNode):
             PyrexTypes.CFuncTypeArg("it", PyrexTypes.py_object_type, None),
             ]))

+    def generate_parallel_assignment_code2(self, rhs, code):
+        rhs_type = rhs.entry.type
+        base_type = rhs_type.base_type
+        size = rhs_type.size
+        lhs_args = self.args
+        assert len(lhs_args) == size
+        for arg in lhs_args:
+            if arg.entry.type is not base_type:
+                raise ValueError("Wrong type for parallel assignment of '%s' array: %s" %
+                                 (base_type, arg.entry.type))
+        for i, arg in enumerate(lhs_args):
+            code.putln("%s = %s[%s];" % (
+                            arg.result(),
+                            rhs.result(),
+                            i))
+
     def generate_parallel_assignment_code(self, rhs, code):
         # Need to work around the fact that generate_evaluation_code
         # allocates the temps in a rather hacky way -- the assignment
diff --git a/Cython/Compiler/Nodes.py b/Cython/Compiler/Nodes.py
index fe7adb3..7987cc6 100644
--- a/Cython/Compiler/Nodes.py
+++ b/Cython/Compiler/Nodes.py
 <at>  <at>  -4509,6 +4509,20  <at>  <at>  class SingleAssignmentNode(AssignmentNode):
         self.lhs = self.lhs.analyse_target_types(env)
         self.lhs.gil_assignment_check(env)

+        if (isinstance(self.lhs, ExprNodes.TupleNode) and
+            isinstance(self.rhs, ExprNodes.NameNode)):
+            rhs_type = self.rhs.entry.type
+            if isinstance(rhs_type, PyrexTypes.CArrayType):
+                base_type = rhs_type.base_type
+                size = rhs_type.size
+                lhs_args = self.lhs.args
+                assert len(lhs_args) == size
+                for arg in lhs_args:
+                    if arg.entry.type is not base_type:
+                        break
+                else:
+                    return self
+
         if self.lhs.memslice_broadcast or self.rhs.memslice_broadcast:
             self.lhs.memslice_broadcast = True
             self.rhs.memslice_broadcast = True
diff --git a/tests/run/arrayassign.pyx b/tests/run/arrayassign.pyx
index 29c353e..f76ad2e 100644
--- a/tests/run/arrayassign.pyx
+++ b/tests/run/arrayassign.pyx
 <at>  <at>  -128,6 +128,59  <at>  <at>  def test_ptr_literal_list_slice_end():
     a[:5] = [1,2,3,4,5]
     return (a[0], a[1], a[2], a[3], a[4])

+
+# The opposite, assignation from an array to multiple scalars.
+
+def test_array_to_scalars():
+    """
+    >>> test_array_to_scalars()
+    (1, 2, 3, 4, 5)
+    """
+    cdef int l[5], a, b, c, d, e
+    l[:] = [1,2,3,4,5]
+    a, b, c, d, e = l
+    return (a, b, c, d, e)
+
+def test_slice_all_to_scalars():
+    """
+    >>> test_slice_all_to_scalars()
+    (1, 2, 3, 4, 5)
+    """
+    cdef int l[5], a, b, c, d, e
+    l[:] = [1,2,3,4,5]
+    a, b, c, d, e = l[:]
+    return (a, b, c, d, e)
+
+def test_slice_start_to_scalars():
+    """
+    >>> test_slice_start_to_scalars()
+    (1, 2, 3, 4, 5)
+    """
+    cdef int l[7], a, b, c, d, e
+    l[:] = [6,7,1,2,3,4,5]
+    a, b, c, d, e = l[2:]
+    return (a, b, c, d, e)
+
+def test_slice_end_to_scalars():
+    """
+    >>> test_slice_end_to_scalars()
+    (1, 2, 3, 4, 5)
+    """
+    cdef int l[7], a, b, c, d, e
+    l[:] = [1,2,3,4,5,6,7]
+    a, b, c, d, e = l[:5]
+    return (a, b, c, d, e)
+
+def test_slice_start_end_to_scalars():
+    """
+    >>> test_slice_start_end_to_scalars()
+    (1, 2, 3, 4, 5)
+    """
+    cdef int l[9], a, b, c, d, e
+    l[:] = [6,7,1,2,3,4,5,8,9]
+    a, b, c, d, e = l[2:7]
+    return (a, b, c, d, e)
+
 # tuples aren't supported (yet)
 #
 #def test_literal_tuple():
Joonas Paalasmaa | 12 Sep 07:46 2013
Picon

[Cython] Appending a dict to a struct vector creates invalid code

Hello,

I am appending Python dict objects to a C++ vector of structs. If it
is done implicitly (calling v.push_back(dict_object)), Cython creates
C++ code that does not compile. If dict_object has been cdef'd to be
of the struct type, everything works. I am using version cf75e9eb7e
from the master branch.

Here is the code for reproducing the problem

# cythonbug.pyx
from libcpp.vector cimport vector

ctypedef struct test_struct:
    int a
    int b

def main():
    cdef vector[test_struct] test_vector

#    cdef test_struct dict_object # Uncommenting this avoids the bug

    dict_object = {"a": 0, "b": 1}
    test_vector.push_back(dict_object)

# end cythonbug.pyx

The C++ output from "cython --cplus cythonbug.pyx" contains the
statement "__pyx_t_9cythonbug_test_struct;" which GCC complains about:
"error: declaration does not declare anything [-fpermissive]".

Best wishes,
Joonas Paalasmaa
Dénes Vadász | 11 Sep 00:26 2013
Picon

[Cython] Metaclass bug corrupts the heap

Hello,

according to valgrind the following Cython fragment causes a heap corruption (with Python 2.7.5 and Cython 0.19.1 on Ubuntu):

cdef class MetaClass(type):
    cdef int i

class MyClass(object):
    __metaclass__ = MetaClass


Please find below the result of a many hour investigation (originally triggered by random crashes and heap corruptions of a bigger Cython program that uses metaclasses).

MetaClass is compiled to a C structure that extends PyTypeObject (a cc. 200 byte structure on a 32-bit architecture):

struct __pyx_obj_4meta_MetaClass {
  PyTypeObject __pyx_base;
  int i;
};

Instances of PyTypeObject are supposed to be allocated statically when initializing extension modules, because the structure does not support (among others) garbage collection. However, when MyClass is created, an instance of struct __pyx_obj_4meta_MetaClass (cc. 200 + 4 bytes) is dynamically allocated by the Python memory management machinery. The machinery then tries to initialize the allocated memory. The problem is that it expects that the first member of struct __pyx_obj_4meta_MetaClass is of type PyHeapTypeObject (a cc. 500 byte structure) and after a type cast it writes to the tail members of the assumed PyHeapTypeObject, i.e. way beyond the allocated cc. 200 + 4 bytes, corrupting the object heap. This corruption is nicely reported by valgrind.

The proposed fix is to make __pyx_obj_4meta_MetaClass extend PyHeapTypeObject:
struct __pyx_obj_4meta_MetaClass {
  PyHeapTypeObject __pyx_base;
  int i;
};

This can be achieved by adding the below 2 lines to Compiler/Builtin.py:
382a383,384
>         elif name == 'type':
>             objstruct_cname = 'PyHeapTypeObject'

So that the init_builtin_types function becomes:

def init_builtin_types():
    global builtin_types
    for name, cname, methods in builtin_types_table:
        utility = builtin_utility_code.get(name)
        if name == 'frozenset':
            objstruct_cname = 'PySetObject'
        elif name == 'bool':
            objstruct_cname = None
       elif name == 'type':
           objstruct_cname = 'PyHeapTypeObject'
        else:
            objstruct_cname = 'Py%sObject' % name.capitalize()
        the_type = builtin_scope.declare_builtin_type(name, cname, utility, objstruct_cname)
        builtin_types[name] = the_type
        for method in methods:
            method.declare_in_type(the_type)

After patching my Cython installation with the above, valgrind stopped to complain and there were no more crashes.

Please consider adding this fix in the next release of Cython.

Regards

Dénes Vadász
<div>
    Hello,<br><br>
    according to valgrind the following Cython fragment causes a heap
    corruption (with Python 2.7.5 and Cython 0.19.1 on Ubuntu):<br><br><blockquote>cdef class MetaClass(type):<br>&nbsp;&nbsp;&nbsp; cdef int i<br><br>class MyClass(object): <br>&nbsp;&nbsp;&nbsp; __metaclass__ = MetaClass<br>
</blockquote>
    <br><br>
    Please find below the result of a many hour investigation
    (originally triggered by random crashes and heap corruptions of a
    bigger Cython program that uses metaclasses).<br><br>
    MetaClass is compiled to a C structure that extends PyTypeObject (a
    cc. 200 byte structure on a 32-bit architecture):<br><br><blockquote>struct
          __pyx_obj_4meta_MetaClass {<br>&nbsp; PyTypeObject __pyx_base;<br>&nbsp; int i;<br>};<br>
</blockquote>
    <br>
    Instances of PyTypeObject are supposed to be allocated statically
    when initializing extension modules, because the structure does not
    support (among others) garbage collection. However, when MyClass is
    created, an instance of struct __pyx_obj_4meta_MetaClass (cc. 200 +
    4 bytes) is dynamically allocated by the Python memory management
    machinery. The machinery then tries to initialize the allocated
    memory. The problem is that it expects that the first member of
    struct __pyx_obj_4meta_MetaClass is of type PyHeapTypeObject (a cc.
    500 byte structure) and after a type cast it writes to the tail
    members of the assumed PyHeapTypeObject, i.e. way beyond the
    allocated cc. 200 + 4 bytes, corrupting the object heap. This
    corruption is nicely reported by valgrind.<br><br>
    The proposed fix is to make __pyx_obj_4meta_MetaClass extend
    PyHeapTypeObject:<br><blockquote>struct
          __pyx_obj_4meta_MetaClass {<br>
        &nbsp; PyHeapTypeObject __pyx_base;<br>
        &nbsp; int i;<br>
        };<br>
</blockquote>

      <br>
    This can be achieved by adding the below 2 lines to
    Compiler/Builtin.py:<br>
    <blockquote>382a383,384<br>&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; elif name == 'type':<br>&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; objstruct_cname = 'PyHeapTypeObject'<br>
</blockquote>
    <br>
    So that the init_builtin_types function becomes:<br><br>
    <blockquote>def init_builtin_types():<br>&nbsp;&nbsp;&nbsp; global builtin_types<br>&nbsp;&nbsp;&nbsp; for name, cname, methods in builtin_types_table:<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; utility = builtin_utility_code.get(name)<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; if name == 'frozenset':<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; objstruct_cname = 'PySetObject'<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; elif name == 'bool':<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; objstruct_cname = None<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; elif name == 'type':<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; objstruct_cname = 'PyHeapTypeObject'<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; else:<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; objstruct_cname = 'Py%sObject' %
          name.capitalize()<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; the_type = builtin_scope.declare_builtin_type(name,
          cname, utility, objstruct_cname)<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; builtin_types[name] = the_type<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; for method in methods:<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; method.declare_in_type(the_type)<br>
</blockquote>
    <br>
    After patching my Cython installation with the above, valgrind
    stopped to complain and there were no more crashes.<br><br>
    Please consider adding this fix in the next release of Cython.<br><br>
    Regards<br><br>
    D&eacute;nes Vad&aacute;sz<br>
</div>
Robert Bradshaw | 10 Sep 02:14 2013
Picon

[Cython] memoryviews bloat?

>> Wow. The first version of this PR used Cython memoryviews, which added a whopping 14 kLOC of generated C to
the repo. Switched back to bare pointers to keep the compile times within bounds.
> It would be interesting to hear the Cython team's point of view on that.  <at> sturlamolden  <at> robertwb
 <at> markflorisson88  <at> dagss

The priority has always been to produce the most optimized runtime
code, with compile time being way down on the list of priorities, if
even thought about at all. This means that there's likely a lot of
low-hanging fruit for those who find the compile time too slow (do you
have some specific stats?). Also, while Cython generates very verbose
C code, much of it is optimized away. While the lines-of-code and
compile times are related, they should not be conflated.

The recently added common_utility_include_dir option to cythonize(...)
might help as well--it allows utility code to be shared across
multiple generated files.

Perhaps a faster -c dbg option for development would be handy.

- Robert
Stefan Behnel | 1 Sep 12:17 2013
Picon

[Cython] redesigning extension modules, avoiding static global C state

Hi,

I recently started a discussion on python-dev to move this topic forward.

http://thread.gmane.org/gmane.comp.python.devel/141262

Basically, the idea is to allow extension modules to implement their own
module type, with all benefits like module level properties, proper garbage
collection, sub-interpreter support, module reloading, etc.

One outcome of that discussion (for me) was that a) there is an interest
over there in doing that and b) it makes sense to start implementing this
in Cython as far as we can already, meaning that we'd move all module
globals into an extension type, turn global functions into methods of that
type, move extension types to the heap and then hide that module type
behind the existing module dict by copying over the references to its types
and bound methods at module init time. PEP 3119 provides an infrastructure
that we can use for now to get at least some of the benefits.

Besides being some work to implement, I suspect that this also has a
visible user impact, sadly. There might be a performance regression when we
move from static globals to pointer indirections, often even multiple
indirections. What would be more annoying is if we had to loosen some
assumptions about globals that we currently make, e.g. regarding nogil
safety or temping, as that could prevent users from doing stuff they
currently do in their code. I don't remember if there were any major
changes here for closures (which are similar), so not sure yet what the
exact impact will be for globals.

I think we should try to keep this out of the compiler itself as much as
possible by using C macros for accessing globals. We'd still need to
calculate the access path to the module state instance for each scope, but
if we can move the rest to C compilation time, we may even be able to keep
up a static-globals mode that modules could use if they value access time
higher than more dynamic module support (e.g. because they cannot be
reloaded or reinstantiated anyways due to external C dependencies not
playing nicely).

I'm not sure yet how much time I can invest into this during the next
months, so if anyone else wants to join in for this, that'd be helpful.

Stefan
Stefan Behnel | 31 Aug 15:28 2013
Picon

[Cython] Should we monkey patch inspect.isfunction() ?

Hi,

currently, this fails in Cython compiled code:

    def cyfunc(): pass

    import inspect
    assert inspect.isfunction(cyfunc)

There isn't really much of a reason why this *should* fail, except for the
implementation of inspect.isfunction(), which tests for

    isinstance(obj, types.FunctionType)

where types.FunctionType is type(some_python_function).

In the benchmarks that we run on Jenkins, I'm patching into this as follows:

"""
def patch_inspect_isfunction():
    import inspect
    orig_isfunction = inspect.isfunction
    def isfunction(obj):
        return (orig_isfunction(obj)
                or type(obj).__name__ == 'cython_function_or_method')
    isfunction._orig_isfunction = orig_isfunction
    inspect.isfunction = isfunction

patch_inspect_isfunction()
"""

For example, Django uses inspect.isfunction() in some dynamic code and
requires this fix to run properly when compiled.

Now that we have a function type that is shared between different Cython
modules, I think it's worth considering the addition of a proper type test
in the inspect module as a general 'fix'. Meaning that we could patch the
inspect module automatically whenever we insert the Cython function type
into the shared Cython types module.

I'm aware that this is a bit evil and hackish, but it usually works quite
nicely.

I'm also aware that the proper way to do this would eventually be the ABC
mechanism as defined by PEP 3119, but Python's function type doesn't
currently support the ABC protocol. We could additionally lobby for getting
that changed.

http://www.python.org/dev/peps/pep-3119/#overloading-isinstance-and-issubclass

Opinions?

Stefan
Stefan Behnel | 29 Aug 07:03 2013
Picon

[Cython] Problem with final cdef methods

Hi,

I noticed two problems with final cdef methods. When overriding a normal
cdef method with a final cdef method, you currently get a C compiler
warning about a call with the wrong 'self' type, and when you call them
before you declare them in your code, you get an error because the method
is not forward declared.

http://trac.cython.org/cython_trac/ticket/819

I think this suggests that we should try to come up with a cleaner way to
integrate this feature into the function overloading architecture rather
than just remembering their cname and call that.

Stefan
Felix Salfelder | 25 Aug 16:52 2013

[Cython] bug in cythonize dependency tracking?

Hi there.

Lets say, i have foo.pyx. and use cythonize() to generate foo.c.
Calling cythonize() a second time will correctly do nothing, because
foo.c has been modified after foo.pyx.

If I now add foo.pxd, cythonize will correctly update foo.c. In general
foo.c now looks different. But: Deleting foo.pxd and running cythonize
another time will fail (=do nothing) and leave foo.c in the wrong state.

This might be a bug, depending on whether cythonize is meant to refresh
output files consistently.

regards
felix

Gmane