Michael Enßlin | 22 Jun 04:04 2015

[Cython] Feature Request: Variadic Templates

Hi,

I'd like to see support for variadic template arguments, such as this:

test.h:

template<typename RetT, typename ... VARARGS>
RetT foo(VARARGS ...);

test.pyx:

cdef extern from "test.h":
    cdef RetT foo[RetT, ... VARARGS](... VARARGS)

def test():
    cdef int i = foo[int, float, int](1, 2.5, 3)

This would allow Cython's libcpp to easily support many of C++11's new
types, including std::tuple and std::function.

Support for the latter in particular would prove quite useful in passing
around function pointers between Cython and C++.

I have tried to implement this feature myself, but I'm entirely
unfamiliar with Cython's codebase, and all my attempts ended in
unsatisfactorily hacky, unstable code.
I believe that any experienced Cython developer on this list would be
able to properly implement this in a matter of a few hours, as it seems
like a rather minor feature.

(Continue reading)

Stephen LARROQUE | 15 Jun 12:34 2015
Picon

[Cython] Bug: Extension Type inheriting from int cause a MemoryError

Hello,

I am trying to make an extension type inheriting from int or cython.int (to do arithmetic operations in Galois Fields). However, this causes a MemoryError because it seems such extension type is not freed correctly. Other than that, it works perfectly well.

Here is a test case to reproduce the bug on Python 2.7.9 Win32 (I'm running Windows 7, but it may happen on other platform, I did not test) with Cython v0.22 on Anaconda:

########################
# mode: run

import sys

cdef class ExtendedInt(int): pass

_ERRORS = u"""
MemoryError
"""

# Simple test case to raise a MemoryError by generating lots of ExtendedInt
total_it = 1000
for i in xrange(total_it):
    for j in xrange(10000000):
        ExtendedInt(j)
    sys.stdout.write("\rGenerating lists of ExtendedInt : %i/%i" % (i, total_it))
########################

My current workaround as advised in the cython-users list:

is to inherit from standard (object) and then reimplement all int magic methods, which is cumbersome and less optimized, but at least there's no MemoryError.

<div><div dir="ltr">Hello,<div><br></div>
<div>
<span>I am trying to make an extension type inheriting from int or&nbsp;</span><a href="http://cython.int/" target="_blank" rel="nofollow">cython.int</a>&nbsp;(to do arithmetic operations in Galois Fields). However, this causes a MemoryError because it seems such extension type is not freed correctly. Other than that, it works perfectly well.<br>
</div>
<div><br></div>
<div>
<span>Here is a test case to reproduce the bug on Python 2.7.9 Win32 (I'm running Windows 7, but it may happen on other platform, I did not test) with Cython v0.22 on Anaconda:</span><br>
</div>
<div><span><br></span></div>
<div>
<span>########################</span><br>
</div>
<div>
<div># mode: run</div>
<div><br></div>
<div>import sys</div>
<div><br></div>
<div>cdef class ExtendedInt(int): pass</div>
<div><br></div>
<div>_ERRORS = u"""</div>
<div>MemoryError</div>
<div>"""</div>
<div><br></div>
<div># Simple test case to raise a MemoryError by generating lots of ExtendedInt</div>
<div>total_it = 1000</div>
<div>for i in xrange(total_it):</div>
<div>&nbsp; &nbsp; for j in xrange(10000000):</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; ExtendedInt(j)</div>
<div>&nbsp; &nbsp; sys.stdout.write("\rGenerating lists of ExtendedInt : %i/%i" % (i, total_it))</div>
<div>########################</div>
<div><br></div>
<div>My current workaround as advised in the cython-users list:</div>
<div>
<a href="https://groups.google.com/forum/#!topic/cython-users/WJ5BJ5moAh4">https://groups.google.com/forum/#!topic/cython-users/WJ5BJ5moAh4</a><br>
</div>
<div><br></div>
<div>is to inherit from standard (object) and then reimplement all int magic methods, which is cumbersome and less optimized, but at least there's no MemoryError.</div>
<div><br></div>
</div>
</div></div>
Andrew Svetlov | 13 Jun 08:30 2015
Picon

[Cython] Bug: comprehensions clear current exception

I have an issue in aiohttp library:
https://github.com/KeepSafe/aiohttp/issues/410

The source of problem is: when I execute the following code

body = ', '.join("'{}': {!r}".format(k, v) for k, v in self.items())

in except block Cython clears exception.

Changing from comprehension to regular iteration works well:

lst = []
for k, v in self._items:
    lst.append("'{}': {!r}".format(k, v))
body = ', '.join(lst)

In pyre Python both versions don't clear current exception.

I think it's a Cython bug.

--

-- 
Thanks,
Andrew Svetlov
Michael Enßlin | 12 Jun 23:24 2015

[Cython] Feature request: Flag to enable warnings when 'extern' functions are not declared 'except +'

Hi everybody,

as you surely have guessed from the amount of Bug reports I have
recently submitted to this mailing list, I'm currently working on a
rather large Cython project.
More precisely: I'm using Cython as the glue layer between the Python
and C++ components of openage; an overview and source code may be found
here:
https://github.com/mic-e/openage/blob/pyinterface/doc/implementation/pyinterface.md
https://github.com/mic-e/openage/tree/pyinterface/cpp/pyinterface
https://github.com/mic-e/openage/tree/pyinterface/openage/cppinterface

In openage, almost all C++ functions that are exposed to Cython SHOULD
be declared as 'except +'.
Forgetting to do so can cause hard-to-debug issues, including useless
Exception messages, corruption of CPython and right-out program
termination, if those methods should throw a C++ exception.

As a solution, I'd like to see Cython warnings if an extern cimport has
no 'except' declaration.

To intentionally omit the "except +", I suggest a new except
declaration, "except -", or, even better, the C++11 keyword "noexcept"
(since precisely those methods that are declared 'noexcept' should be
cimported without 'except+').

Of course, those warnings (or errors?) should be disabled by default,
and enabled by a special command-line flag.

Any thoughts on this?

	~ mic_e
Michael Enßlin | 12 Jun 13:01 2015

[Cython] Can't call functions with multi-token template arguments such as 'char *'

Hi,

it seems to be impossible to use anything but a single word as a
template type for functions.

Classes don't suffer from this issue, as seen below.

As a workaround, complex types can be ctypedef-d to a single word (also
seen below).

$ cat t10.pyx

cdef extern from "nope.h":
    cdef cppclass bar[T]:
        void func(T arg)

    void foo[T](T arg)

ctypedef char * charptr

def test():
    # works
    cdef bar[char *] barobj
    barobj.func(NULL)

    # works
    foo[int](5)

    # works
    foo[charptr](NULL)

    # fails
    foo[char *](NULL)

$ cython --cplus t10.pyx

Error compiling Cython file:
------------------------------------------------------------
...

    # works
    foo[charptr](NULL)

    # fails
    foo[char *](NULL)
             ^
------------------------------------------------------------

t10.pyx:27:14: Expected an identifier or literal

Happy debugging,
	~ mic_e

Hi,

it seems to be impossible to use anything but a single word as a
template type for functions.

Classes don't suffer from this issue, as seen below.

As a workaround, complex types can be ctypedef-d to a single word (also
seen below).

$ cat t10.pyx

cdef extern from "nope.h":
    cdef cppclass bar[T]:
        void func(T arg)

    void foo[T](T arg)

ctypedef char * charptr

def test():
    # works
    cdef bar[char *] barobj
    barobj.func(NULL)

    # works
    foo[int](5)

    # works
    foo[charptr](NULL)

    # fails
    foo[char *](NULL)

$ cython --cplus t10.pyx

Error compiling Cython file:
------------------------------------------------------------
...

    # works
    foo[charptr](NULL)

    # fails
    foo[char *](NULL)
             ^
------------------------------------------------------------

t10.pyx:27:14: Expected an identifier or literal

Happy debugging,
	~ mic_e

Michael Enßlin | 12 Jun 12:51 2015

[Cython] Compiler crash in AnalyseExpressionsTransform

Hi,

another bug report:

mic <at> mic /tmp $ cat t11.pyx
cdef cppclass foo:
    pass

def test():
    foo()

mic <at> mic /tmp $ cython --cplus t11.pyx

Error compiling Cython file:
------------------------------------------------------------
...
cdef cppclass foo:
    pass

def test():
    foo()
      ^
------------------------------------------------------------

t11.pyx:5:7: Compiler crash in AnalyseExpressionsTransform

ModuleNode.body = StatListNode(t11.pyx:1:0)
StatListNode.stats[1] = DefNode(t11.pyx:4:0,
    modifiers = [...]/0,
    name = u'test',
    py_wrapper_required = True,
    reqd_kw_flags_cname = '0',
    used = True)
File 'Nodes.py', line 421, in analyse_expressions: StatListNode(t11.pyx:5:7)
File 'Nodes.py', line 4652, in analyse_expressions:
ExprStatNode(t11.pyx:5:7)
File 'ExprNodes.py', line 434, in analyse_expressions:
SimpleCallNode(t11.pyx:5:7,
    use_managed_ref = True)
File 'ExprNodes.py', line 4495, in analyse_types:
SimpleCallNode(t11.pyx:5:7,
    use_managed_ref = True)
File 'ExprNodes.py', line 4429, in analyse_as_type_constructor:
SimpleCallNode(t11.pyx:5:7,
    use_managed_ref = True)

Compiler crash traceback from this point on:
  File "/usr/lib/python2.7/dist-packages/Cython/Compiler/ExprNodes.py",
line 4429, in analyse_as_type_constructor
    self.function = RawCNameExprNode(self.function.pos, constructor.type)
AttributeError: 'NoneType' object has no attribute 'type'

The code  is obviously nonsensical, but Cython should produce a proper
error message instead of crashing.

Hi,

another bug report:

mic <at> mic /tmp $ cat t11.pyx
cdef cppclass foo:
    pass

def test():
    foo()

mic <at> mic /tmp $ cython --cplus t11.pyx

Error compiling Cython file:
------------------------------------------------------------
...
cdef cppclass foo:
    pass

def test():
    foo()
      ^
------------------------------------------------------------

t11.pyx:5:7: Compiler crash in AnalyseExpressionsTransform

ModuleNode.body = StatListNode(t11.pyx:1:0)
StatListNode.stats[1] = DefNode(t11.pyx:4:0,
    modifiers = [...]/0,
    name = u'test',
    py_wrapper_required = True,
    reqd_kw_flags_cname = '0',
    used = True)
File 'Nodes.py', line 421, in analyse_expressions: StatListNode(t11.pyx:5:7)
File 'Nodes.py', line 4652, in analyse_expressions:
ExprStatNode(t11.pyx:5:7)
File 'ExprNodes.py', line 434, in analyse_expressions:
SimpleCallNode(t11.pyx:5:7,
    use_managed_ref = True)
File 'ExprNodes.py', line 4495, in analyse_types:
SimpleCallNode(t11.pyx:5:7,
    use_managed_ref = True)
File 'ExprNodes.py', line 4429, in analyse_as_type_constructor:
SimpleCallNode(t11.pyx:5:7,
    use_managed_ref = True)

Compiler crash traceback from this point on:
  File "/usr/lib/python2.7/dist-packages/Cython/Compiler/ExprNodes.py",
line 4429, in analyse_as_type_constructor
    self.function = RawCNameExprNode(self.function.pos, constructor.type)
AttributeError: 'NoneType' object has no attribute 'type'

The code  is obviously nonsensical, but Cython should produce a proper
error message instead of crashing.

Michael Enßlin | 11 Jun 18:03 2015

[Cython] Cython produces invalid code for operator()

Hi guys,

have a look at this:

$ cat bar.pyx
cdef extern from "foo.h":
    cdef cppclass Foo:
        int operator() (int arg)
        int do_call (int arg)

cdef int bar(int arg):
    cdef Foo foo
    foo.do_call(arg)
    return foo(arg)

$ cython3 --cplus bar.pyx
$ cat bar.cpp

(...)

static int __pyx_f_3bar_bar(int __pyx_v_arg) {
  Foo __pyx_v_foo;
  int __pyx_r;
  __Pyx_RefNannyDeclarations
  __Pyx_RefNannySetupContext("bar", 0);

  /* "bar.pyx":9
 * cdef int bar(int arg):
 *     cdef Foo foo
 *     foo.do_call(arg)             # <<<<<<<<<<<<<<
 *     return foo(arg)
 */
  __pyx_v_foo.do_call(__pyx_v_arg);

  /* "bar.pyx":10
 *     cdef Foo foo
 *     foo.do_call(arg)
 *     return foo(arg)             # <<<<<<<<<<<<<<
 */
  __pyx_r = operator()(__pyx_v_arg);
  goto __pyx_L0;

  /* "bar.pyx":7
 *
 *
 * cdef int bar(int arg):             # <<<<<<<<<<<<<<
 *     cdef Foo foo
 *     foo.do_call(arg)
 */

  /* function exit code */
  __pyx_L0:;
  __Pyx_RefNannyFinishContext();
  return __pyx_r;
}

(...)

Note how the function invocation for "do_call" is generated correctly,
but the invocation of operator() is nonsensical.

The correct line would be this:

  __pyx_r = __pyx_v_foo(__pyx_v_arg);

Happy debugging :D

	~ mic_e

Hi guys,

have a look at this:

$ cat bar.pyx
cdef extern from "foo.h":
    cdef cppclass Foo:
        int operator() (int arg)
        int do_call (int arg)

cdef int bar(int arg):
    cdef Foo foo
    foo.do_call(arg)
    return foo(arg)

$ cython3 --cplus bar.pyx
$ cat bar.cpp

(...)

static int __pyx_f_3bar_bar(int __pyx_v_arg) {
  Foo __pyx_v_foo;
  int __pyx_r;
  __Pyx_RefNannyDeclarations
  __Pyx_RefNannySetupContext("bar", 0);

  /* "bar.pyx":9
 * cdef int bar(int arg):
 *     cdef Foo foo
 *     foo.do_call(arg)             # <<<<<<<<<<<<<<
 *     return foo(arg)
 */
  __pyx_v_foo.do_call(__pyx_v_arg);

  /* "bar.pyx":10
 *     cdef Foo foo
 *     foo.do_call(arg)
 *     return foo(arg)             # <<<<<<<<<<<<<<
 */
  __pyx_r = operator()(__pyx_v_arg);
  goto __pyx_L0;

  /* "bar.pyx":7
 *
 *
 * cdef int bar(int arg):             # <<<<<<<<<<<<<<
 *     cdef Foo foo
 *     foo.do_call(arg)
 */

  /* function exit code */
  __pyx_L0:;
  __Pyx_RefNannyFinishContext();
  return __pyx_r;
}

(...)

Note how the function invocation for "do_call" is generated correctly,
but the invocation of operator() is nonsensical.

The correct line would be this:

  __pyx_r = __pyx_v_foo(__pyx_v_arg);

Happy debugging :D

	~ mic_e

Michael Enßlin | 8 Jun 16:50 2015

[Cython] Feature request: Expose methods to fake stack traces

Hi everybody,

my C++ exceptions contain some stack trace information (starting with
__FILE__ and __LINE__), and I'd like to preserve that information in my
custom 'except+' translator.

Cython seems to have some internal methods to fake Python stack trace
objects, but research (and an unanswered question on the Users list
[https://groups.google.com/forum/#!topic/cython-users/9coFCVwigpE])
seems to imply that this functionality is not currently exposed.

I'd like to see that internal functionality exposed for usage from .pyx
files.

 ~ mic_e

Hi everybody,

my C++ exceptions contain some stack trace information (starting with
__FILE__ and __LINE__), and I'd like to preserve that information in my
custom 'except+' translator.

Cython seems to have some internal methods to fake Python stack trace
objects, but research (and an unanswered question on the Users list
[https://groups.google.com/forum/#!topic/cython-users/9coFCVwigpE])
seems to imply that this functionality is not currently exposed.

I'd like to see that internal functionality exposed for usage from .pyx
files.

 ~ mic_e

Neal Becker | 6 Jun 20:25 2015
Picon

[Cython] cython-0.23 monkey-patches stdlib?

According to
https://github.com/cython/cython/blob/master/CHANGES.rst

the default build would attempt to modify python stdlib files?

How would this work on distributions?  I am maintainer for Cython on Fedora.  
So installing Cython would try to overwrite stdlib files?  That sounds 
difficult.  For one thing, it means more than 1 package would own the same 
files.  And I don't know if the general user community would like this 
approach.

Picon

[Cython] Exporting inline functions from a pyx.

Dear Cython developers,
thanks for the awesome compiler!, we recently faced an issue exporting inline functions defined in a .pyx. According to this: http://docs.cython.org/src/tutorial/pxd_files.html
the full inline definition (not just the declaration) should be in the pxd. However, if we put the definition in the .pyx and just its declaration header in the .pxd,  Cython declares a pointer to the inline function similar to:

static CYTHON_INLINE double (*__function_name)(...); /*proto*/

but this causes a compilation error in some platforms (but successfully compiles in others) because variables cannot be declare as inline. You can find a log with the compilation errors here:

I guess it should be possible to detect when a function is inline and in that case remove the CYTHON_INLINE from the pointer declaration, or at least consistently fail in all platforms. Is this a known issue? 

Thank you very much!
With warm regards,
-Omar.
--
"Cada quien es dueño de lo que calla y esclavo de lo que dice"
-Proverbio chino.
"We all are owners of what we keep silent and slaves of what we say"
-Chinese proverb.

http://www.cimat.mx/~omar
<div><div dir="ltr">Dear Cython developers,<div>thanks for the awesome compiler!, we recently faced an issue exporting inline functions defined in a .pyx. According to this:&nbsp;<a href="http://docs.cython.org/src/tutorial/pxd_files.html">http://docs.cython.org/src/tutorial/pxd_files.html</a>
</div>
<div>the full inline definition (not just the declaration) should be in the pxd. However, if we put the definition in the .pyx and just its declaration header in the .pxd, &nbsp;Cython declares a pointer to the inline function similar to:</div>
<div><br></div>
<div>static CYTHON_INLINE double (*__function_name)(...); /*proto*/</div>
<div><br></div>
<div>but this causes a compilation error in some platforms (but successfully compiles in others) because variables cannot be declare as inline. You can find a log with the compilation errors here:</div>
<div>
<a href="http://nipy.bic.berkeley.edu/builders/dipy-py2.7-osx-10.8/builds/362/steps/shell_5/logs/stdio">http://nipy.bic.berkeley.edu/builders/dipy-py2.7-osx-10.8/builds/362/steps/shell_5/logs/stdio</a><br>
</div>
<div><br></div>
<div>I guess it should be possible to detect when a function is inline and in that case remove the CYTHON_INLINE from the pointer declaration, or at least consistently fail in all platforms. Is this a known issue?&nbsp;</div>
<div><br></div>
<div>Thank you very much!</div>
<div>With warm regards,</div>
<div>-Omar.<br>-- <br><div class="gmail_signature">"Cada quien es due&ntilde;o de lo que calla y esclavo de lo que dice"<br>-Proverbio chino.<br>"We all are owners of what we keep silent and slaves of what we say"<br>-Chinese proverb.<br><br><a href="http://www.cimat.mx/~omar" target="_blank">http://www.cimat.mx/~omar</a><br>
</div>
</div>
</div></div>
Jeroen Demeyer | 23 May 10:47 2015
Picon

[Cython] Pull request #374 (api mangling prefix) merged in Sage

This has been merged in Sage's version of Cython:
https://github.com/cython/cython/pull/374

The change looks quite innocent, so I so little reason to not merge it 
in Cython master.

Gmane