Ian Flitman | 20 Sep 00:39 2014
Picon

Compiler crash in AnalyseDeclarationsTransform - too many open files and/or RuntimeError: maximum recursion depth exceeded in cmp

Hello Cythoners,

I am having trouble compiling a cython project which will be, I hope, a Python binding for the C-based multimedia framework GPAC.

When I try to compile the .pyx file to .c using

cython gpacfile.pyx

The complier crashes. The trace is rather long and it looks like a recursive loop of some sort:

Error compiling Cython file:
------------------------------------------------------------
...

cimport isomedia
       ^
------------------------------------------------------------

gpacfile.pyx:2:8: Compiler crash in AnalyseDeclarationsTransform

File 'ModuleNode.py', line 101, in analyse_declarations: ModuleNode(gpacfile.pyx:2:0,
    full_module_name = 'MP4Box.gpac.gpacfile')
File 'Nodes.py', line 383, in analyse_declarations: StatListNode(gpacfile.pyx:2:0)
File 'Nodes.py', line 383, in analyse_declarations: StatListNode(gpacfile.pyx:2:8)
File 'Nodes.py', line 6785, in analyse_declarations: CImportStatNode(gpacfile.pyx:2:8,
    module_name = u'isomedia')

Compiler crash traceback from this point on:
  File "/usr/lib/python2.7/dist-packages/Cython/Compiler/Nodes.py", line 6785, in analyse_declarations
    module_scope = env.find_module(self.module_name, self.pos)
  File "/usr/lib/python2.7/dist-packages/Cython/Compiler/Symtab.py", line 1089, in find_module
    module_name, relative_to = self.parent_module, pos = pos)
  File "/usr/lib/python2.7/dist-packages/Cython/Compiler/Main.py", line 167, in find_module
    err, result = self.process_pxd(source_desc, scope, module_name)
  File "/usr/lib/python2.7/dist-packages/Cython/Compiler/Main.py", line 99, in process_pxd
    result = Pipeline.run_pipeline(pipeline, source_desc)
  File "/usr/lib/python2.7/dist-packages/Cython/Compiler/Pipeline.py", line 328, in run_pipeline
    data = phase(data)
  File "/usr/lib/python2.7/dist-packages/Cython/Compiler/Pipeline.py", line 44, in parse
    full_module_name=module_name)
  File "/usr/lib/python2.7/dist-packages/Cython/Compiler/Main.py", line 300, in parse
    tree = Parsing.p_module(s, pxd, full_module_name)
  File "Parsing.py", line 3171, in Cython.Compiler.Parsing.p_module (Cython/Compiler/Parsing.c:49563)
  File "Parsing.py", line 3186, in Cython.Compiler.Parsing.p_module (Cython/Compiler/Parsing.c:49310)
  File "Parsing.py", line 1920, in Cython.Compiler.Parsing.p_statement_list (Cython/Compiler/Parsing.c:30876)
  File "Parsing.py", line 1890, in Cython.Compiler.Parsing.p_statement (Cython/Compiler/Parsing.c:30447)
  File "Parsing.py", line 1652, in Cython.Compiler.Parsing.p_include_statement (Cython/Compiler/Parsing.c:27163)
  File "Parsing.py", line 1920, in Cython.Compiler.Parsing.p_statement_list (Cython/Compiler/Parsing.c:30876)
  File "Parsing.py", line 1890, in Cython.Compiler.Parsing.p_statement (Cython/Compiler/Parsing.c:30447)
  File "Parsing.py", line 1652, in Cython.Compiler.Parsing.p_include_statement (Cython/Compiler/Parsing.c:27163)
  File "Parsing.py", line 1920, in Cython.Compiler.Parsing.p_statement_list (Cython/Compiler/Parsing.c:30876)
  File "Parsing.py", line 1890, in Cython.Compiler.Parsing.p_statement (Cython/Compiler/Parsing.c:30447)
  File "Parsing.py", line 1652, in Cython.Compiler.Parsing.p_include_statement (Cython/Compiler/Parsing.c:27163)
  File "Parsing.py", line 1920, in Cython.Compiler.Parsing.p_statement_list (Cython/Compiler/Parsing.c:30876)
  File "Parsing.py", line 1890, in Cython.Compiler.Parsing.p_statement (Cython/Compiler/Parsing.c:30447)
  File "Parsing.py", line 1652, in Cython.Compiler.Parsing.p_include_statement (Cython/Compiler/Parsing.c:27163)
  File "Parsing.py", line 1920, in Cython.Compiler.Parsing.p_statement_list (Cython/Compiler/Parsing.c:30876)

etc until...

File "/usr/lib/python2.7/dist-packages/Cython/Utils.py", line 285, in open_source_file
     f = open_source_file(source_filename, encoding="UTF-8", mode=mode, error_handling='ignore')
  File "/usr/lib/python2.7/dist-packages/Cython/Utils.py", line 309, in open_source_file
    encoding=encoding, errors=error_handling)


The gpactypes.pxd is the typedef the GPAC library uses, and when I remove it as an include in an attempt to remove any circularity, I get a intelliJ errors instead.

When I use the following setup.py instead to compile to .c and get the .so:

from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize

setup(
    ext_modules = cythonize([
        Extension("gpacfile", ["gpacfile.pyx"],
                     libraries=["libgpac"])
    ])
)

through the following command line:

ian <at> ian-UX303LN:~/workspace/CythonTest/GPAC/MP4Box/gpac$ CFLAGS="-I/home/ian/projects/tools/gpac/include/gpac"  LDFLAGS="-L/home/ian/projects/tools/gpac/bin/gcc"  python setup.py build_ext -i

I get a long trace again:

Traceback (most recent call last):
  File "setup.py", line 10, in <module>
    libraries=["libgpac"])
  File "/usr/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 707, in cythonize
    aliases=aliases)
  File "/usr/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 633, in create_extension_list
    kwds = deps.distutils_info(file, aliases, base).values
  File "/usr/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 543, in distutils_info
    return (self.transitive_merge(filename, self.distutils_info0, DistutilsInfo.merge)
  File "/usr/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 553, in transitive_merge
    node, extract, merge, seen, {}, self.cimported_files)[0]
  File "/usr/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 565, in transitive_merge_helper
    sub_deps, sub_loop = self.transitive_merge_helper(next, extract, merge, seen, stack, outgoing)
  File "/usr/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 558, in transitive_merge_helper
    deps = extract(node)
  File "/usr/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 534, in distutils_info0
    externs = self.cimports_and_externs(filename)[1]
  File "/usr/lib/python2.7/dist-packages/Cython/Utils.py", line 29, in wrapper
    res = cache[args] = f(self, *args)
  File "/usr/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 444, in cimports_and_externs
    for include in self.included_files(filename):
  File "/usr/lib/python2.7/dist-packages/Cython/Utils.py", line 29, in wrapper
    res = cache[args] = f(self, *args)
  File "/usr/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 432, in included_files
    all.update(self.included_files(include_path))
  File "/usr/lib/python2.7/dist-packages/Cython/Utils.py", line 29, in wrapper
    res = cache[args] = f(self, *args)
  File "/usr/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 432, in included_files
    all.update(self.included_files(include_path))
  File "/usr/lib/python2.7/dist-packages/Cython/Utils.py", line 29, in wrapper
    res = cache[args] = f(self, *args)
  File "/usr/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 432, in included_files
    all.update(self.included_files(include_path))
  File "/usr/lib/python2.7/dist-packages/Cython/Utils.py", line 29, in wrapper
    res = cache[args] = f(self, *args)
  File "/usr/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 432, in included_files
    all.update(self.included_files(include_path))
  File "/usr/lib/python2.7/dist-packages/Cython/Utils.py", line 29, in wrapper
    res = cache[args] = f(self, *args)
  File "/usr/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 432, in included_files
    all.update(self.included_files(include_path))
  File "/usr/lib/python2.7/dist-packages/Cython/Utils.py", line 29, in wrapper
    res = cache[args] = f(self, *args)
  File "/usr/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 432, in included_files
    all.update(self.included_files(include_path))
  File "/usr/lib/python2.7/dist-packages/Cython/Uti

etc until

 File "/usr/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 432, in included_files
    all.update(self.included_files(include_path))
  File "/usr/lib/python2.7/dist-packages/Cython/Utils.py", line 29, in wrapper
    res = cache[args] = f(self, *args)
  File "/usr/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 424, in included_files
    for include in self.parse_dependencies(filename)[1]:
  File "/usr/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 417, in parse_dependencies
    return parse_dependencies(source_filename)
  File "/usr/lib/python2.7/dist-packages/Cython/Utils.py", line 14, in wrapper
    res = cache.get(args, uncomputed)
RuntimeError: maximum recursion depth exceeded in cmp

Again this looks like an error based on an infinite recursive loop. However, I repeat the code looks ok to my Cython-savy IDE.
Is there a way to include a pxd definition just once to prevent this, or is it something else?

I am pretty new to Cython so any pointers or advice would be much appreciated. My current failing source code is viewable at:

https://github.com/ianflitman/CyGPAC

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
20100.delecroix | 19 Sep 23:18 2014
Picon

error propagation (set from the C API)

Hello,

I am a Sage developer and that is my first time on this list ;-) I am digging into Cython and the C-API in order to speedup an Extension type for words in Sage (i.e. a sequence of symbols). At first glance, the extension type mostly looks like a list. A critical operation is the __getitem__ one. I tested several options but the only fast way seems to copy/paste what is done in the __getitem__ of lists, namely:

def __getitem__(self, key):
    cdef Py_ssize_t i, start, stop, step, slicelength
    cdef int res
    if PySlice_Check(<PyObject *>key):
        res = PySlice_GetIndicesEx(<PySliceObject *>key,
                    self._length,
                    &start, &stop, &step,
                    &slicelength)
        if res < 0:
            print "HAAAAAAAAAAA... there will soon be a crash"
        # ...

    elif PyIndex_Check(<PyObject *>key):
        i = PyNumber_AsSsize_t(<PyObject *>key, PyExc_IndexError)
        if i == -1 and PyErr_Occurred():
            print "HAAAAAAAAAAA... there will soon be a crash"
        # ...

As you see, I did not found any way to propagate the exception that might be set by PySlice_GetIndicesEx or PyNumber_AsSsize_t. Is there a Cython solution to that ?

Best
Vincent

PS: the timings for my __getitem__ are as close as possible as the one for lists.... very cool!

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Nick Repole | 18 Sep 23:56 2014
Picon

Help with wrapping a struct (specifically containing 1D and 2D arrays)

Say I have example.h:

struct SimpleStruct {
    int simple_array[5];
    int two_d_array[5][5];
}

Then I have example.pyx:

cdef extern from "example.h":
    cdef struct CppSimpleStruct "SimpleStruct":
        int simple_array[5]
        int two_d_array[5][5]

How do I go about wrapping this struct so that I can get and set both members? I attempted the following:

cdef class SimpleStruct:
    cdef CppSimpleStruct* thisptr

    def __cinit__(self):
       self.thisptr = new SimpleStruct()

    def __dealloc__(self):
        if self.thisptr is not NULL:
            del self.thisptr

This works if I use cdef class instead of cdef struct in the initial extern declaration. Obviously no setting or getting of members yet. If I use cdef struct it fails due to the new keyword being C++ only and the struct being considered a C object. I attempted to remedy this using malloc instead, but got an error that mentions temporary objects.

As for actually accessing the members, if I use the cdef class hack mentioned above, I still have no idea how to deal with the simple_array member or the two dimensional member. How do I go about converting that int[5] to a python array?

I spent a good deal of time going through the documentation, trying to find sample code here or on github, but I'm struggling to accomplish something that I believe is meant to be relatively simple? Hopefully I'm just missing something easy.

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
1989lzhh | 18 Sep 16:16 2014
Picon

is cython possible to support special method like __getitem__ defined as cpdef

hi eveyone,
i have aware cython for a long time and impressed by her elegant usage and speed.
now I want to write an extension class to wrap allocated memory. I want the index part to be as fast as it can.
First, I try with special method __getitem__, it worked as expected but the index speed is about 6 times
slower than calling "cdefed " equivalent function ( in cython side). 
Since the definition of special methods like __getitem__  now in cython extension class are limited by
using “def” only. I am wondring if it is possible allowing “cpdef”. 
Thanks

Yours,
Liu zhenhai

--

-- 

--- 
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Jerome Kieffer | 17 Sep 12:19 2014

Module name issue with OpenMP

Hi all,

While porting my work to Apple computer which currently lack OpenMP
support (and my code cimport openmp), I faced an issue.

I wrote two version of the modules:
module_omp.pyx
module_nomp.pyx

They are selected by the setup.py and compiled to the module.so

When loading into python the module, it compalains:
"dynamic module does not define init function."

Is there a way to tell cython the name of the module or is my only option to copy
module_[n]omp.pyx -> module.pyx
before compilation ?

Thanks for the help.
-- 
Jerome Kieffer <google <at> terre-adelie.org>

--

-- 

--- 
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Array of maps

Hi all

I am wanting to use a C array of maps, something like this:

from libcpp.map cimport map

cdef
enum:
    MAX_BOARD_WIDTH
= 19
    POS_INT
= MAX_BOARD_WIDTH * MAX_BOARD_WIDTH
    PRIORITIES
= 6

ctypedef
unsigned short POS_T
ctypedef
char CAND_T

ctypedef map
[POS_T, CAND_T] cbp_t

So there is the map typedef, but how do I typedef an array of them?

ctypedef cbp_t[POS_T, CAND_T] cbpp_t[PRIORITIES]

    gives:
pentai/ai/properties.pxd:20:78: Template parameter not a type

ctypedef cbp_t cbpp_t[PRIORITIES]

    also gives:
pentai/ai/properties.pxd:21:53: Template parameter not a type

However, an array of ints can be typedefed with the same syntax AFAICT:
ctypedef int foo_arr[27]

I am still fairly new to cython. Any suggestions would be appreciated.

Thanks,
Bruce

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Nick Repole | 14 Sep 04:30 2014
Picon

C++ wrapping, maintaining namespaces, and submodules?

Is it possible to have Cython submodules? Lets say I have a module mymodule and classes/functions for that module are in mymodule.cpp/mymodule.h. Some utility functions are in utils.cpp/utils.h and are used by mymodule. Can I create a Cython module with the structure:

mymodule
  MyClass
  mymodule_function
  utils
    utils.utils_function

Where utils is a submodule of mymodule? If I were to compile a separate utils.pyx, this seems doable, but the result of that is double compiling (since mymodule requires utils). From a C++ perspective, I use namespaces such that mymodule::MyClass and mymodule::utils::utils_function are how such items are accessed, and I'd ideally like to be able to replicate that structure with Cython.

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Mikhail Korobov | 14 Sep 01:20 2014
Picon

c_string_type, std::string and explicit bytes

Hi,

I'm using Cython 0.21 with the following options:

# cython: c_string_type=str, c_string_encoding=ascii

My goal is to convert std::string to Python bytes. This code:

cdef string my_str
# ...
... <bytes>my_str

generates C++ code that uses __pyx_convert_string_to_py_ (the same as without "<bytes>"); it fails for non-ascii strings in Python 3.x.

<bytes>my_str.c_str()

works as expected - it uses __Pyx_PyBytes_FromString and works for non-ascii strings.

Is it a bug, or am I missing something?

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Skip Montanaro | 12 Sep 22:04 2014
Picon

Compiling with py+pxd file

I asked a question on the code-quality list about a PyLint type thing
for Cython code. Mike Mueller referred me to the pxd file feature. I
have a couple questions about that.

* I'm using Cython 0.17.1. Does that support this feature or do I need
a more recent version?

* Consider this simple function to compute an exponentially weighted
moving average:

def _ewma(alpha, old, new):
    return (1 - alpha) * old + alpha * new

And this Cythonified version:

cdef _ewma(float alpha, float old, float new):
    return (1 - alpha) * old + alpha * new

Do I just duplicate the code, or declare it in the pxd file as something like:

cdef _ewma(float alpha, float old, float new):
    pass

What if the only change to a function is the declaration of the types
of the parameters and a small number of local variables? How much goes
in the pxd file? The examples here:

http://docs.cython.org/src/tutorial/pure.html

give no examples of defining local variables in the pxd file.

Thanks,

Skip

--

-- 

--- 
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Antoine Martin | 12 Sep 16:06 2014
Picon

Cython 0.21 and ImportError raised in module causes segfault eventually

Hi,

There seems to be a bug in Cython 0.21 (or at least an undocumented
change in behaviour): I have some modules that perform some sanity
checks and raise an ImportError on some platforms to prevent the module
from loading.
In previous versions (up to and including 0.20.2), I could just catch
the ImportError and ignore the module, but with Cython 0.21 the
exception is not raised to the location that imports the module (I use
__import__ if that's relevant) and leaves the module in a
half-initialized state which crashes the Python interpreter as soon as
you try to use the module.

Thanks
Antoine

--

-- 

--- 
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Alyson Deines | 11 Sep 18:50 2014
Picon

cimport sage modules

Hi!

I have a (really) basic question.  I want to be able to cimport Sage classes in a project setup like this:
Project/
setup.py
text.pyx

When I use Cython in the Sage notebook, I can cimport via:
 
%cython
from sage.rings.integer cimport Integer

and this auto generates come code. 

Right now my setup.py file is:

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

setup(   
    cmdclass = {'build_ext': build_ext},
    ext_modules = [Extension('test',['test.pyx'])]
)
and my text.pyx file is:

from sage.rings.integer cimport Integer

When I try to compile I get the following error:

~$ sage -python setup.py build_ext --inplace
running build_ext
cythoning test.pyx to test.c
 
Error compiling Cython file:
------------------------------------------------------------
...
from sage.rings.integer cimport Integer
^
------------------------------------------------------------
 
test.pyx:1:0: 'sage.rings.integer.pxd' not found
 
Error compiling Cython file:
------------------------------------------------------------
...
from sage.rings.integer cimport Integer
^
------------------------------------------------------------
 
test.pyx:1:0: 'Integer.pxd' not found
 
Error compiling Cython file:
------------------------------------------------------------
...
from sage.rings.integer cimport Integer
                               ^
------------------------------------------------------------
 
test.pyx:1:32: Name 'Integer' not declared in module 'sage.rings.integer'
building 'test' extension
gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/local/sage/sage-6.3.beta6/local/include/python2.7 -c test.c -o build/tem
p.linux-x86_64-2.7/test.o
test.c:1:2: error: #error Do not use this file, it is the result of a failed Cython compilation.
 #error Do not use this file, it is the result of a failed Cython compilation.
  ^
error: command 'gcc' failed with exit status 1

I've tried changing my setup.py file to:

import os
import sys

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

if not 'SAGE_ROOT' in os.environ:
    print " ERROR: The environment variable SAGE_ROOT must be defined."
    sys.exit(1)
else:
    SAGE_ROOT = os.environ['SAGE_ROOT']   
setup(   
    cmdclass = {'build_ext': build_ext},
    ext_modules = [Extension('test',
                    ['test.pyx'],
                    include_dirs = [
                    os.path.join(SAGE_ROOT, 'local/include'),
                    os.path.join(SAGE_ROOT, 'local/include/csage'),
                    os.path.join(SAGE_ROOT, 'devel/sage/sage/ext'),
                    os.path.join(SAGE_ROOT, 'src/sage/ext'),
                    os.path.join(SAGE_ROOT, 'devel/sage'),
                    os.path.join(SAGE_ROOT, 'src')],
                    library_dirs = [os.path.join(SAGE_ROOT, 'local/lib')],)]
)


but I still get the following error:

~$ sage -python setup.py build_ext --inplace
running build_ext
cythoning test.pyx to test.c
 
Error compiling Cython file:
------------------------------------------------------------
...
from sage.rings.integer cimport Integer
^
------------------------------------------------------------
 
test.pyx:1:0: 'sage.rings.integer.pxd' not found
 
Error compiling Cython file:
------------------------------------------------------------
...
from sage.rings.integer cimport Integer
^
------------------------------------------------------------
 
test.pyx:1:0: 'Integer.pxd' not found

Thanks!

Aly

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Gmane