headers
Waqas Muhi | 1 May 2012 03:00
Picon

Re: Python to Cython conversion

Hello,

I installed MS Visual Studio 2008 Express Edition from this link: http://www.microsoft.com/en-us/download/details.aspx?id=14597 and I'm no longer seeing the problem I was seeing before. Thanks Chris for pointing that out.

Now, finally getting to my actual Python-to-Cython conversion :), below is the Cython .pyx file I've started writing. Since this file involves usage of numpy, I'm using this link as a guide: http://docs.cython.org/src/tutorial/numpy.html

import numpy as np
# "cimport" is used to import special compile-time information
# about the numpy module (this is stored in a file numpy.pxd which is
# currently part of the Cython distribution).
cimport numpy as np
# We now need to fix a datatype for our arrays. I've used the variable
# DTYPE for this, which is assigned to the usual NumPy runtime
# type info object.
DTYPE = np.float
# "ctypedef" assigns a corresponding compile-time type to DTYPE_t. For
# every type in the numpy module there's a corresponding compile-time
# type with a _t-suffix.
ctypedef np.float_t DTYPE_t

def calculate_position(np.ndarray[DTYPE_t, ndim=1] u_A, np.ndarray[DTYPE_t, ndim=1] v_A, float t, float t_A):
   
    assert u_A.dtype == DTYPE and v_A.dtype == DTYPE
    cdef float told = t - t_A
    return (u_A + v_A)*told.tolist()

However, when I run my top-level .py module, I see that the above .pyx file is compiled ok but I get the following error output:

C:\Source\src_Python_sim\PNT_Cython_001>python run_pnt.py
_tethered_particle_system.c
C:\Users\Waqas/.pyxbld\temp.win32-2.6\Release\pyrex\_tethered_particle_system.c(
237) : fatal error C1083: Cannot open include file: 'numpy/arrayobject.h': No su
ch file or directory
Traceback (most recent call last):
  File "run_pnt.py", line 3, in <module>
    from presynaptic_nerve_terminal_DEVS import presynaptic_nerve_terminal_DEVS
  File "C:\Source\src_Python_sim\PNT_Cython_001\presynaptic_nerve_terminal_DEVS.
py", line 3, in <module>
    from tethered_particle_system_DEVS import tethered_particle_system
  File "C:\Source\src_Python\tethered_particle_system_DEVS.py", line 4, in <modu
le>
    from tethered_particle_system import tethered_particle_system
  File "C:\Source\src_Python\tethered_particle_system.py", line 5, in <module>
    from _tethered_particle_system import calculate_position
  File "C:\Python26\lib\site-packages\pyximport\pyximport.py", line 335, in load
_module
    self.pyxbuild_dir)
  File "C:\Python26\lib\site-packages\pyximport\pyximport.py", line 183, in load
_module
    so_path = build_module(name, pyxfilename, pyxbuild_dir)
  File "C:\Python26\lib\site-packages\pyximport\pyximport.py", line 167, in buil
d_module
    reload_support=pyxargs.reload_support)
  File "C:\Python26\lib\site-packages\pyximport\pyxbuild.py", line 85, in pyx_to
_dll
    dist.run_commands()
  File "C:\Python26\lib\distutils\dist.py", line 975, in run_commands
    self.run_command(cmd)
  File "C:\Python26\lib\distutils\dist.py", line 995, in run_command
    cmd_obj.run()
  File "C:\Python26\lib\site-packages\Cython\Distutils\build_ext.py", line 135,
in run
    _build_ext.build_ext.run(self)
  File "C:\Python26\lib\distutils\command\build_ext.py", line 340, in run
    self.build_extensions()
  File "C:\Python26\lib\site-packages\Cython\Distutils\build_ext.py", line 143,
in build_extensions
    self.build_extension(ext)
  File "C:\Python26\lib\distutils\command\build_ext.py", line 499, in build_exte
nsion
    depends=ext.depends)
  File "C:\Python26\lib\distutils\msvc9compiler.py", line 533, in compile
    raise CompileError(msg)
ImportError: Building module failed: ['CompileError: command \'"C:\\Program File
s (x86)\\Microsoft Visual Studio 9.0\\VC\\BIN\\cl.exe"\' failed with exit status
 2\n']

If I remove the numpy stuff from my file, then I don't see the above error output. So it is clearly something to do with how I'm using numpy. Can anyone please guide me where I'm going wrong?

Thanks,
Waqas

On Mon, Apr 9, 2012 at 12:15 PM, Chris Barker <chris.barker <at> noaa.gov> wrote:
On Sun, Apr 8, 2012 at 7:02 AM, Waqas Muhi
> There's a particular file I seem to be missing. I tried both the "import
> pyximport" as well as the "Building a Cython module using the DistUtils"
> approach mentioned here: http://docs.cython.org/src/quickstart/build.html

> In both cases, I get an error message about vcvarsall.bat:

vcvarsall.bat comes with Microsoft Visual Studio (Visual C, whatever
the heck else they call it). It is used to ser of the compiler
environment when invoked.

Cython, of course, requires a compiler. distutils (and I presume
pyximport) is set up (by default) to build extensions with the same
complier and settings that Python was build with (that's one of its
core nifty features).

The Python 2.6 and 2.7 binaries distributed by python.org are built
with VS2008 (I'm pretty sure). The easiest way to get set up for that
is to install the Microsoft Visual Studio 2008 express addition (It's
free on the MS web site -- it's a bit hard to find - MS tries to get
you to use the 2010 version, but that may not work with the python
builds). I've found installing VS 2008 express then "just works" with
distutils (and thus Cython).

If you want 64 bit -- it theoretically can be done, but its takes some
hacking, and I haven't gotten it to work yet (I've given up twice
after messing around or an hour or two). But maybe I just messed
something up -- there is a page in the Cython wiki about how to do it.

-Chris





--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker <at> noaa.gov

Permalink | Reply | 14 comments |
headers
Ian Bell | 1 May 2012 03:46
Picon

Re: Python to Cython conversion



On Tue, May 1, 2012 at 1:00 PM, Waqas Muhi <waqas.muhi <at> gmail.com> wrote:
Hello,

I installed MS Visual Studio 2008 Express Edition from this link: http://www.microsoft.com/en-us/download/details.aspx?id=14597 and I'm no longer seeing the problem I was seeing before. Thanks Chris for pointing that out.

Now, finally getting to my actual Python-to-Cython conversion :), below is the Cython .pyx file I've started writing. Since this file involves usage of numpy, I'm using this link as a guide: http://docs.cython.org/src/tutorial/numpy.html

import numpy as np
# "cimport" is used to import special compile-time information
# about the numpy module (this is stored in a file numpy.pxd which is
# currently part of the Cython distribution).
cimport numpy as np
# We now need to fix a datatype for our arrays. I've used the variable
# DTYPE for this, which is assigned to the usual NumPy runtime
# type info object.
DTYPE = np.float
# "ctypedef" assigns a corresponding compile-time type to DTYPE_t. For
# every type in the numpy module there's a corresponding compile-time
# type with a _t-suffix.
ctypedef np.float_t DTYPE_t

def calculate_position(np.ndarray[DTYPE_t, ndim=1] u_A, np.ndarray[DTYPE_t, ndim=1] v_A, float t, float t_A):
   
    assert u_A.dtype == DTYPE and v_A.dtype == DTYPE
    cdef float told = t - t_A
    return (u_A + v_A)*told.tolist()

However, when I run my top-level .py module, I see that the above .pyx file is compiled ok but I get the following error output:

C:\Source\src_Python_sim\PNT_Cython_001>python run_pnt.py
_tethered_particle_system.c
C:\Users\Waqas/.pyxbld\temp.win32-2.6\Release\pyrex\_tethered_particle_system.c(
237) : fatal error C1083: Cannot open include file: 'numpy/arrayobject.h': No su
ch file or directory
Traceback (most recent call last):
  File "run_pnt.py", line 3, in <module>
    from presynaptic_nerve_terminal_DEVS import presynaptic_nerve_terminal_DEVS
  File "C:\Source\src_Python_sim\PNT_Cython_001\presynaptic_nerve_terminal_DEVS.
py", line 3, in <module>
    from tethered_particle_system_DEVS import tethered_particle_system
  File "C:\Source\src_Python\tethered_particle_system_DEVS.py", line 4, in <modu
le>
    from tethered_particle_system import tethered_particle_system
  File "C:\Source\src_Python\tethered_particle_system.py", line 5, in <module>
    from _tethered_particle_system import calculate_position
  File "C:\Python26\lib\site-packages\pyximport\pyximport.py", line 335, in load
_module
    self.pyxbuild_dir)
  File "C:\Python26\lib\site-packages\pyximport\pyximport.py", line 183, in load
_module
    so_path = build_module(name, pyxfilename, pyxbuild_dir)
  File "C:\Python26\lib\site-packages\pyximport\pyximport.py", line 167, in buil
d_module
    reload_support=pyxargs.reload_support)
  File "C:\Python26\lib\site-packages\pyximport\pyxbuild.py", line 85, in pyx_to
_dll
    dist.run_commands()
  File "C:\Python26\lib\distutils\dist.py", line 975, in run_commands
    self.run_command(cmd)
  File "C:\Python26\lib\distutils\dist.py", line 995, in run_command
    cmd_obj.run()
  File "C:\Python26\lib\site-packages\Cython\Distutils\build_ext.py", line 135,
in run
    _build_ext.build_ext.run(self)
  File "C:\Python26\lib\distutils\command\build_ext.py", line 340, in run
    self.build_extensions()
  File "C:\Python26\lib\site-packages\Cython\Distutils\build_ext.py", line 143,
in build_extensions
    self.build_extension(ext)
  File "C:\Python26\lib\distutils\command\build_ext.py", line 499, in build_exte
nsion
    depends=ext.depends)
  File "C:\Python26\lib\distutils\msvc9compiler.py", line 533, in compile
    raise CompileError(msg)
ImportError: Building module failed: ['CompileError: command \'"C:\\Program File
s (x86)\\Microsoft Visual Studio 9.0\\VC\\BIN\\cl.exe"\' failed with exit status
 2\n']

If I remove the numpy stuff from my file, then I don't see the above error output. So it is clearly something to do with how I'm using numpy. Can anyone please guide me where I'm going wrong?

Thanks,
Waqas

On Mon, Apr 9, 2012 at 12:15 PM, Chris Barker <chris.barker <at> noaa.gov> wrote:
On Sun, Apr 8, 2012 at 7:02 AM, Waqas Muhi
> There's a particular file I seem to be missing. I tried both the "import
> pyximport" as well as the "Building a Cython module using the DistUtils"
> approach mentioned here: http://docs.cython.org/src/quickstart/build.html

> In both cases, I get an error message about vcvarsall.bat:

vcvarsall.bat comes with Microsoft Visual Studio (Visual C, whatever
the heck else they call it). It is used to ser of the compiler
environment when invoked.

Cython, of course, requires a compiler. distutils (and I presume
pyximport) is set up (by default) to build extensions with the same
complier and settings that Python was build with (that's one of its
core nifty features).

The Python 2.6 and 2.7 binaries distributed by python.org are built
with VS2008 (I'm pretty sure). The easiest way to get set up for that
is to install the Microsoft Visual Studio 2008 express addition (It's
free on the MS web site -- it's a bit hard to find - MS tries to get
you to use the 2010 version, but that may not work with the python
builds). I've found installing VS 2008 express then "just works" with
distutils (and thus Cython).

If you want 64 bit -- it theoretically can be done, but its takes some
hacking, and I haven't gotten it to work yet (I've given up twice
after messing around or an hour or two). But maybe I just messed
something up -- there is a page in the Cython wiki about how to do it.

-Chris





--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker <at> noaa.gov

Waqas,

Cython can't find your numpy headers.  You need to tell Cython where to look.  In your distutils setup.py file, you need to

a) import numpy
b) into the setup() function, you need to add the numpy include folder, something like

setup(
  name = 'MyProject',
  cmdclass={'build_ext': build_ext},
  ext_modules = [My_module],
  include_dirs = [numpy.get_include()]
)

Hope this helps.

Ian
Permalink | Reply | 13 comments |
headers
becker.nils | 1 May 2012 10:55
Picon

Re: segfault in PyArray_FromAny from a memoryviewslice

hi again,

i have now isolated what could have been the cause of the problems before. check this out:

--- test.pyx

import cython
cimport cython
import numpy as np
cimport numpy as np


def test1():
    print np.random.rand(3).dtype
    cdef np.float64_t[::1] aslice = np.random.rand(2)
    cdef float[::1] bslice = np.random.rand(2)   

def test2(nn):
    cdef double[::1] aslice = np.random.rand(2)
    cdef double[::1] bslice
    cdef int ii
    print np.asarray(aslice)
    for ii in range(nn):
        bslice = aslice.copy()
    print 'success.'


now in a console:

>from test import test1, test2

>test2(10)

runs through but

>test2(1000)

segfaults. this is on python 2.7.2, cython 0.16.


the other test function may or may not be related. when i call test1(), i get

ValueError: Buffer dtype mismatch, expected 'float' but got 'double'

when assigning to bslice. (aslice works.) i found that surprising since i though float and np.float are compatible.

n.


Permalink | Reply | 2 comments |
headers
mark florisson | 1 May 2012 15:10
Picon
Gravatar

Re: segfault in PyArray_FromAny from a memoryviewslice

On 1 May 2012 09:55, becker.nils <becker.nils <at> gmx.net> wrote:
> hi again,
>
> i have now isolated what could have been the cause of the problems before.
> check this out:
>
> --- test.pyx
>
> import cython
> cimport cython
> import numpy as np
> cimport numpy as np
>
>
> def test1():
>     print np.random.rand(3).dtype
>     cdef np.float64_t[::1] aslice = np.random.rand(2)
>     cdef float[::1] bslice = np.random.rand(2)
>
> def test2(nn):
>     cdef double[::1] aslice = np.random.rand(2)
>     cdef double[::1] bslice
>     cdef int ii
>     print np.asarray(aslice)
>     for ii in range(nn):
>         bslice = aslice.copy()
>     print 'success.'
>
>
> now in a console:
>
>>from test import test1, test2
>
>>test2(10)
>
> runs through but
>
>>test2(1000)
>
> segfaults. this is on python 2.7.2, cython 0.16.
>
>
> the other test function may or may not be related. when i call test1(), i
> get
>
> ValueError: Buffer dtype mismatch, expected 'float' but got 'double'
>
> when assigning to bslice. (aslice works.) i found that surprising since i
> though float and np.float are compatible.

Yes, np.float is actually a np.float64, just like np.double :)

> n.
>
>

Thanks for the report, that's is very useful. It seems the initial
memoryview implementation had a reference count error. Can you try
again with the Cython from github?
https://github.com/cython/cython/commit/3d5bc03f1e7f4e3f34cec3131fe5cb0d0a177a18
Permalink | Reply | 1 comment |
headers
becker.nils | 1 May 2012 16:08
Picon

Re: segfault in PyArray_FromAny from a memoryviewslice


Thanks for the report, that's is very useful. It seems the initial
memoryview implementation had a reference count error. Can you try
again with the Cython from github?
https://github.com/cython/cython/commit/3d5bc03f1e7f4e3f34cec3131fe5cb0d0a177a18

i did. everything seems to be working now. the test function as well as my original code that segfaulted before (posted (partially) at the beginning of the thread)

thanks!

n.
Permalink | Reply | 0 comments |
headers
becker.nils | 1 May 2012 16:56
Picon

pointer to np.array.data vs. memoryview

hi,

i was wondering what the relative benefits are of using pointers to the data attribute of ndarrays vs. using memoryviews.
i have an implementation of a function looping over the entries of an array; no fancy slicing is needed. i had that implemented
by passing a pointer to the .data to a cdef function.

cdef f(float * data)

f(<float *>myarray.data)

i tried changing it into

cdef f(float[::1] data)

cdef float[::1] myview = myarray

f(myview)


both versions work, except that the pointer one is faster by a factor of 2 or so. my question is: is there something that's better about the memview version?

n.

Permalink | Reply | 2 comments |
headers
Wenxiang Chen | 2 May 2012 00:50
Picon
Gravatar

Allocate an array of vectors in Cython?

Hi All,

I am new to Cython. I have tried a couple of different approaches, yet I still failed to allocate an array of vectors.

The first method I have tried is 
"cdef vector[int] stack[10]"
The cython compiler showed:
    #Error compiling Cython file:
    #------------------------------------------------------------
    #...
    #    void* malloc(size_t size)
    #    void* realloc(void* ptr, size_t size)
    #
    #def vecTest(self):
    #    cdef int n = 10
    #    cdef vector[int] stack[10]
    #                          ^
    #------------------------------------------------------------
    # arrOfVec.pyx:10:27: Template parameter not a type.


The second method which I also tried is:
    cdef vector[int]* arr
    arr = <vector[int] *> malloc(sizeof(vector[int])*n)
It still cannot pass the compilation. The error message is shown below:
    #Error compiling Cython file:
    #------------------------------------------------------------
    #    arr = <vector[int] *> malloc(sizeof(vector[int])*n)
    #                      ^
    #------------------------------------------------------------
    #
    #arrOfVec.pyx:27:23: Unknown type

The minimal code reproducing the error is attached as "arrOfVec.pyx", the setup file is attached as well.

Could you guys show me some hints on how to fix this issue? Many thanks in advance.

- Wenxiang
Attachment (arrOfVec.pyx): application/octet-stream, 1069 bytes
Attachment (testSetup.py): text/x-python, 311 bytes
Permalink | Reply | 2 comments |
headers
Ian Bell | 2 May 2012 04:31
Picon

Problem with pure Python mode class constructors type definitions

Unless I am missing something, it does not seem to be possible to apply type definitions for a class constructor in a .pxd file.  In my pure Python file I can do something like

import cython

class A:

    <at> cython.locals(x=cython.double)
    def __init__(self,x):
        pass

but if I try to make a type definition in the PXD file like

cdef class A:

    def __init__(self, double x)

and compile I get an error:

Error compiling Cython file:
------------------------------------------------------------
...

cdef class A:

    def __init__(self, double x)                               ^
------------------------------------------------------------

A.pxd:4:32: Expected ':', found 'NEWLINE'

Is there some special syntax that I am missing?  I can get this to work for the "normal" functions without a problem.

Regards,
Ian

Permalink | Reply | 3 comments |
headers
Robert Bradshaw | 2 May 2012 08:15
Picon

Re: Problem with pure Python mode class constructors type definitions

On Tue, May 1, 2012 at 7:31 PM, Ian Bell <ian.h.bell <at> gmail.com> wrote:
> Unless I am missing something, it does not seem to be possible to apply type
> definitions for a class constructor in a .pxd file.  In my pure Python file
> I can do something like
>
> import cython
>
> class A:
>
>      <at> cython.locals(x=cython.double)
>     def __init__(self,x):
>         pass
>
> but if I try to make a type definition in the PXD file like
>
> cdef class A:
>
>     def __init__(self, double x)
>
> and compile I get an error:
>
> Error compiling Cython file:
> ------------------------------------------------------------
> ...
>
> cdef class A:
>
>     def __init__(self, double
x)                               ^
> ------------------------------------------------------------
>
> A.pxd:4:32: Expected ':', found 'NEWLINE'
>
> Is there some special syntax that I am missing?  I can get this to work for
> the "normal" functions without a problem.
>
> Regards,
> Ian

__init__ is a standard python method, and should be simply left out of
the declaration in your .pxd file. Only c(p)def functions belong
there.

- Robert
Permalink | Reply | 2 comments |
headers
Robert Bradshaw | 2 May 2012 08:33
Picon

Re: Allocate an array of vectors in Cython?

On Tue, May 1, 2012 at 3:50 PM, Wenxiang Chen <chenwx.ustc <at> gmail.com> wrote:
> Hi All,
>
> I am new to Cython. I have tried a couple of different approaches, yet I
> still failed to allocate an array of vectors.
>
> The first method I have tried is
> "cdef vector[int] stack[10]"
> The cython compiler showed:
>     #Error compiling Cython file:
>     #------------------------------------------------------------
>     #...
>     #    void* malloc(size_t size)
>     #    void* realloc(void* ptr, size_t size)
>     #
>     #def vecTest(self):
>     #    cdef int n = 10
>     #    cdef vector[int] stack[10]
>     #                          ^
>     #------------------------------------------------------------
>     # arrOfVec.pyx:10:27: Template parameter not a type.
>
>
> The second method which I also tried is:
>     cdef vector[int]* arr
>     arr = <vector[int] *> malloc(sizeof(vector[int])*n)
> It still cannot pass the compilation. The error message is shown below:
>     #Error compiling Cython file:
>     #------------------------------------------------------------
>     #    arr = <vector[int] *> malloc(sizeof(vector[int])*n)
>     #                      ^
>     #------------------------------------------------------------
>     #
>     #arrOfVec.pyx:27:23: Unknown type
>
> The minimal code reproducing the error is attached as "arrOfVec.pyx", the
> setup file is attached as well.
>
> Could you guys show me some hints on how to fix this issue? Many thanks in
> advance.
>
> - Wenxiang

It looks like there's a bug in trying to allocate an array of
templated types, but the following works for me (Cython 0.16).

from libcpp.vector cimport vector
from libc.stdlib cimport malloc

def test(int n):
    cdef vector[int] *stack
    stack = <vector[int] *> malloc(sizeof(vector[int])*n)

- Robert
Permalink | Reply | 1 comment |

Gmane