Elmar | 21 Jan 21:44 2015

Best Way of Passing array.array('L') to Cython

Hi,

I am new to Cython and have very simple problem for which I am not sure what the best solution is. I want to pass several lists (array.array('L') to be precise) from Python to Cython to perform some computations in a tight loop.
Currently I have the following Cython function:

from cpython cimport array
from array import array

def compute(list_a, list_b):
    cdef array.array[unsigned long] a = array.array('L', list_a)
    cdef array.array[unsigned long] b = array.array('L', list_b)
    cdef int loop_index = 0
    result = []
    .... compute and construct result (add element from a or b to result list - pure Python code)...
    return result


Is this the preferred way of passing and accessing the data (I only need to read from it)? Does array.array in this case actually copy the data or is it a "memory view"?

I am experiencing a massive increase in memory usage (GBs) after calling the function thousands of times. This is definately related to the Cython function - replacing it with a pure Python solution shows no such behavior. Am I missing some memory management here (i.e. if the data is copied it might never be freed)? The result is a simple list of elements that come from one of the lists (a or b).

Thanks for any hints!

Elmar

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Justin Israel | 21 Jan 23:52 2015
Picon

cdef <at> staticmethod support and pxd files

Hi,

I’m now trying out the new 0.21.x support for <at> staticmethod decorators on cdef methods, allowing me to move some of my init functions into methods, but I am a bit confused about how to properly reference them in a pxd file. However it works fine when not trying to reference it in a pxd file...

# foo.pyx cdef class Thing: <at> staticmethod cdef Thing init(ThingPtr* ptr): ... # foo.pxd cdef class Thing: <at> staticmethod cdef Thing init(ThingPtr* ptr)

And I end up with a compile error like:

Self argument (ThingPtr *) of C method 'init' does not match parent type (Thing)

I am not sure if I am doing this wrong, or its a bug, or even just not supported in pxd files?

Justin

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Andreas van Cranenburgh | 20 Jan 16:10 2015
Picon

bug: unsigned loop variable and reversed range

The following code loops indefinitely:

cdef unsigned i

for i in reversed(range(10)):
    print(i)

There should either be an error that unsigned loop variables are not supported, or the generated code should be a while loop that avoids the underflow issue.

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Fabien Georget | 19 Jan 22:51 2015
Picon

Python2/3 : Signature not compatible with previous declaration

p, li { white-space: pre-wrap; }

Hello,


I have a problem with the new version of cython (0.22.2)[1]


When compiling with the "-3" flag the following code :


----------------------------------------------------------


database.pxd :

--------------


from memory cimport shared_ptr


cdef extern from "database/data_container.hpp" namespace "specmicp::database":

cdef cppclass DataContainer:

# content of the struct DataContainer


cdef extern from "database/database.hpp" namespace "specmicp::database":

cdef cppclass Database:

# definition of the useful methods of Database

# this class is used to manage a DataContainer instance


cdef class DatabaseManager:

# attributs, must be declared here

cdef Database *database

cdef shared_ptr[DataContainer] container

# methods

cdef int truc(self)

cdef shared_ptr[DataContainer] get_raw_db(self)

cdef DataContainer* _get(self)



database.pyx :

--------------


from memory cimport shared_ptr

from database cimport Database, DataContainer


cdef class DatabaseManager:

def __cinit__(self, bytes filepath):

"""filepath is the path to the database file"""

self.database = new Database(filepath)

self.container = self.database.get_database()

cdef int truc(self):

pass

cdef shared_ptr[DataContainer] get_raw_db(self):

return self.container

cdef DataContainer* _get(self):

return self.container.get()

# a bunch of python method


-------------------------------------------------------------------


I have the following error message :


-------------------------------------------------------------------

Error compiling Cython file:

------------------------------------------------------------

...

"""filepath is the path to the database file"""

self.database = new Database(filepath)

self.container = self.database.get_database()

cdef int truc(self):

pass

cdef shared_ptr[DataContainer] get_raw_db(self):

^

------------------------------------------------------------


/home/manawy/princeton/progra/specmicp/cython/specmicp/database.pyx:25:9: Signature not compatible with previous declaration


Error compiling Cython file:

------------------------------------------------------------

...

# attributs, must be declared here

cdef Database *database

cdef shared_ptr[DataContainer] container

# methods

cdef int truc(self)

cdef shared_ptr[DataContainer] get_raw_db(self)

^

------------------------------------------------------------


/home/manawy/princeton/progra/specmicp/cython/specmicp/database.pxd:64:45: Previous declaration is here


Error compiling Cython file:

------------------------------------------------------------

...

self.container = self.database.get_database()

cdef int truc(self):

pass

cdef shared_ptr[DataContainer] get_raw_db(self):

return self.container

cdef DataContainer* _get(self):

^

------------------------------------------------------------


/home/manawy/princeton/progra/specmicp/cython/specmicp/database.pyx:27:9: Signature not compatible with previous declaration


Error compiling Cython file:

------------------------------------------------------------

...

cdef Database *database

cdef shared_ptr[DataContainer] container

# methods

cdef int truc(self)

cdef shared_ptr[DataContainer] get_raw_db(self)

cdef DataContainer* _get(self)

^

------------------------------------------------------------


/home/manawy/princeton/progra/specmicp/cython/specmicp/database.pxd:65:22: Previous declaration is here



-------------------------------------------------------------------



It should be noted that the method "cdef int truc(self) " do not raise any error. The error is also not raised when compiling with the option "-2".[2]

I don't think the cython syntax should be different if the target is python 3 rather than python 2, so it looks like a bug. Unless I missed something ?


Thanks

fabien


[1] I didn't have this behavior before. Unfortunately I may have missed a few version, I don't know when the problem was introduced

[2] Note : the compilation is done using a cmake script not distutils because it is part of a larger c++ project, this is how I compile a cython file :


set(CYTHON_FLAGS -3 --cplus -Wextra -I ${PY_INCLUDES_DIR} )

# cython compilation macro
# ------------------------
# USAGE : cython_compilation(dir base_name dependancies)
macro(cython_compilation dir base_name)
add_custom_command(
    OUTPUT ${CMAKE_CURRENT_BINARY_DIR}${CMAKE_FILES_DIRECTORY}/${base_name}.cpp
    COMMAND cython ${CYTHON_FLAGS} -I ${dir} -o ${CMAKE_CURRENT_BINARY_DIR}${CMAKE_FILES_DIRECTORY}/${base_name}.cpp ${dir}/${base_name}.pyx
    DEPENDS ${ARGN})
# ${ARGN} is the list of arguments passed after the last expected argument
# i.e. ${ARGN} = dependancies
endmacro(cython_compilation)

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Mike Pennington | 19 Jan 20:55 2015
Picon

Metaclass .pxd: How should I implement __eq__()?

I am trying to augment an existing python source with a cython .pxd; however, I keep hitting a wall with __eq__() in the .pxd.  I know I need to use __richcmp__(), but the details of doing this are not obvious...

 - If I use "cdef inline __richcmp__(Japan_Car_ABC self, Japan_Car_ABC other):", cython complains that "Special methods must be declared with 'def', not 'cdef'"
 - If I use "def __richcmp__(Japan_Car_ABC self, Japan_Car_ABC other):", cython complains that "function definition in pxd file must be declared 'cdef inline'"

Details and code are shown in this question....

http://stackoverflow.com/questions/28023151/cython-metaclass-pxd-how-should-i-implement-eq

Is there a way to fix this without rewriting everything into a .pyx file?

Apologies for cross-posting from Stack Overflow, but I have a feeling that this might be a problem for the list...

Thoughts?
\mike

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
kasty.jose | 19 Jan 02:06 2015
Picon

weird behaviour with pyobjc

The following example runs fine on OSX 10.9 with Python 2.7.9. It just listens for click events and prints a message:


import sys
import Foundation
import AppKit
import Cocoa
import Quartz.CoreGraphics
from PyObjCTools import AppHelper
frozen = getattr(sys, 'frozen', '')
if frozen:
    foundation = Foundation.Foundation
    appkit = AppKit.AppKit
else:
    foundation = Foundation
    appkit = AppKit
cocoa = Cocoa
quartz = Quartz

class CocoaMonitor:
    def createAppDelegate(self):
        sc = self
        print "createAppDelegate"
        class AppDelegate(foundation.NSObject):
            def applicationDidFinishLaunching_(self, notification):
                print "applicationDidFinishLaunching"
                mask = (appkit.NSLeftMouseDownMask)
                appkit.NSEvent.addGlobalMonitorForEventsMatchingMask_handler_(mask, sc.handler)
        return AppDelegate

    def run(self):
        appkit.NSApplication.sharedApplication()
        delegate = self.createAppDelegate().alloc().init()
        appkit.NSApp().setDelegate_(delegate)
        appkit.NSApp().setActivationPolicy_(appkit.NSApplicationActivationPolicyProhibited)
        self.workspace = appkit.NSWorkspace.sharedWorkspace()
        AppHelper.runEventLoop()

    def cancel(self):
        AppHelper.stopEventLoop()

    def handler(self, event):
        print "mouse click event"

def main():
    mon = CocoaMonitor()
    mon.run()

if __name__ == '__main__':
    main()

Output:

createAppDelegate
applicationDidFinishLaunching
mouse click event
mouse click event
...


However when I compile it with cython 0.21.1, I don't see any events. This is how I do it:

cython test.py --embed
/usr/bin/clang -fno-strict-aliasing -fno-common -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk -arch x86_64 -g -O2 -DNDEBUG -g -O3 `python-config --ldflags` `python-config --libs` `python-config --includes` test.c -o test
./test

Output

createAppDelegate



I used dtruss to inspect system calls but I didn't see anything wrong. Am I missing anything?


--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Colin | 13 Jan 21:47 2015
Picon

Possible Bug: Cythonmagic include directories requires explicit ending /

I noticed that for the IPython Notebook, cythonmagic requires an explicit "/" at the end of the directory path to work properly.

Not-working:
%%cythonmagic -I /path/to/directory

Working:
%%cythonmagic -I /path/to/directory/

I was following the numpy.get_include function to define my own custom function for generating the include path. On my system numpy.get_include does not follow the path with a ending "/".

Is the requirement of the ending "/" a bug?

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Skip Montanaro | 13 Jan 18:56 2015
Picon

float v. double

I'm still using 0.17.1 (not much I can really do about it in the short
term). I just discovered a bug in some Cython code. I had naively
declared a couple local variables as float:

        cdef float t = self._time
        cdef float v

thinking that "float" was the same as Python's float (e.g., a C
double). That proved not to be the case. When I changed the
declarations to specify "double", things started working.

Looking in the Language Basics documentation:

http://docs.cython.org/src/userguide/language_basics.html

I saw no mention of the distinction between Python float and Cython
float. It seems like it should be mentioned somewhere. If so, the
searching I did turned nothing up.

Skip Montanaro

--

-- 

--- 
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Björn Dahlgren | 13 Jan 16:49 2015
Picon

Passing Python callbacks to an external C library.

Hi all!

I am writing a partial wrapper around sundials (and ODE solver written in C) for a project of mine and
I wanted to be able to pass python callbacks to the C library using Cython (even though
I mainly pass C pointers to native code).

It was not immediately apparent how I should go about achieving this but I ended up using
a global variable in the Cython module to point to the user provided Python callback.

I implemented a minimal working example around a fictitious "library" which has two functions
for computing the derivative and gradient of functions using a simple forward difference formula.

If my implementation is sound (criticism is very welcome) maybe it could serve as a self contained example
(I could rewrite it for inclusion in the documentation if there is any interest)?

The code is available at:
https://github.com/bjodah/expose_pycallback_to_a_clib


--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Max Klein | 12 Jan 04:49 2015
Picon

preserving complex namespaces of C++ libraries in Cython wrapper

I'm in the process of writing a Cython wrapper for a complex C++ library. I
think I've mostly figured out how to write the necessary .pxd and .pyx files.
My problem now is that although my C++ program has about 100 separate 
namespaces, the namespace of the Cython-compiled python library is totally 
flat. 

For example, if I have this in my .pxd file:

cdef extern from "lm/io/hdf5/SimulationFile.h" namespace "lm::io::hdf5":
    cdef cppclass CppHdf5File "lm::io::hdf5::Hdf5File":
        ...

and this in my .pyx file:

cdef class Hdf5File:
    cdef CppHdf5File* thisptr 
    ...

then the Cython-compiled Python library contains a class named Hdf5File. 
Ideally, I'd like the Python to contain a lm.io.hdf5.Hdf5File class (i.e. 
a Hdf5File class in a lm.io.hdf5 module). In other words, I'd like it if 
there was a way to translate the C++ :: scoping operator into the 
Python . dot operator.

Is there a way to get Cython to play nicely with my existing C++ namespaces?

--

-- 

--- 
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Matthew Honnibal | 11 Jan 21:28 2015
Picon

A workaround for specifying compile-time dependencies in setuptools

Python packaging is quite a mess.  The standard library module, distutils, lacks a lot of features, but the replacement, setuptools, brought a lot of its own issues. For a long time setuptools seemed to go out of its way to break Cython compilation, by silently renaming .pyx files to .c files if Pyrex was not installed. This has since been fixed, but the word "Cython" still occurs exactly 0 times in the setuptools documentation.

A key feature that distutils lacks is dependency resolution: if someone wants to install your package, you want specified dependencies to be downloaded and installed automatically. This is unsupported by distutils. There's a "require" keyword argument to distutils.core.setup, but it does absolutely nothing.

The setuptools.setup function supports two keyword arguments for dependencies: setup_requires and install_requires. The packages you specify in install_requires will be installed, but the setup_requires packages will only be downloaded and inserted into sys.path, so that they can be imported during your setup.py script. They will not be available at runtime.

Frustratingly, if you specify the same package in setup_requires and install_requires, it will not be installed:

https://bitbucket.org/pypa/setuptools/issue/209
https://bitbucket.org/pypa/setuptools/issue/80/packages-listed-in-setup_requires-are

I'm shipping a package written in Cython, and it requires certain C header files to be present at compile time. I'd like my users to be able to write "pip install mylibrary". I don't want them to have to write "pip install numpy && pip install mylibrary". If the dependency resolution system is broken, it's broken all the way down the tree. Any library that imports my library will also suffer from this problem.

In the absence of a good solution, here's *a* solution:

https://github.com/syllog1sm/headers_workaround

I uploaded a separate package to PyPi, which distributes the headers I require. I then specify this package as the setup_requires dependency, and add some code to my setup.py file telling the headers_workaround module which headers to install.

Obviously, this is unideal, but it's been working for me. If anyone has better ideas, I'd love to hear them!

Best,
Matt.

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Gmane