THOTH | 2 Sep 2010 04:28
Picon

Re: I/O C library wrapping

Hello, Everyone.

Let me repeat the question in a shorter form.

How I can access the following functions of my shared library:

1. Pointer based array get/put data:
int mylib_read(
      BIN_TAG bin_tag,
      void *buf,
      int count)
// bin_tag, count - are input integer parameters, buf - is
the array where data will be stored after processing

2. int cdds_write(
      BIN_TAG bin_tag,
      const void *buf,
      int count)
// now they all input parameters, but I still don't know how to pass
void buff from python to my functions and vise versa.

I use the following code (it's not copy paste so typos can appear -
sorry) with no success:

import cython
cdef extern from "cdds.h":
       int mylib_read(int bin_tag, void *buf, int count)

def plib_read(int bin_tag, int count):
   cdef int t
(Continue reading)

Robert Bradshaw | 2 Sep 2010 06:07
Favicon

Re: Re: I/O C library wrapping

On Wed, Sep 1, 2010 at 7:28 PM, THOTH <anaryusifov <at> gmail.com> wrote:
> Hello, Everyone.
>
> Let me repeat the question in a shorter form.
>
> How I can access the following functions of my shared library:
>
> 1. Pointer based array get/put data:
> int mylib_read(
>      BIN_TAG bin_tag,
>      void *buf,
>      int count)
> // bin_tag, count - are input integer parameters, buf - is
> the array where data will be stored after processing
>
> 2. int cdds_write(
>      BIN_TAG bin_tag,
>      const void *buf,
>      int count)
> // now they all input parameters, but I still don't know how to pass
> void buff from python to my functions and vise versa.
>
> I use the following code (it's not copy paste so typos can appear -
> sorry) with no success:
>
> import cython
> cdef extern from "cdds.h":
>       int mylib_read(int bin_tag, void *buf, int count)
>
> def plib_read(int bin_tag, int count):
(Continue reading)

Anar Yusifov | 2 Sep 2010 06:54
Picon

Re: Re: I/O C library wrapping

Hi Robert,

I'm sorry for unclear question. bin_tag - is file descriptor to the data matrix. count - specifies number of rows to be read at once. x - is buffer which contains read data.   

Here is an Example (could be typos)

#include <cdds.h>
int ier;
float s;
float *buf = (float*)0;
float *wbuf;
buf = (float*)malloc(4); //assume that we have 4 values in each row
wbuf = (float*)malloc(4);
... //using other routines I get bin_tag
ier = mylib_read(bin_tag, buf, 1); // here I get 'ier' and 'buf'
s = buf[0] + buf[1] + buf[2] +buf[3];
//printf('%f',s);
wbuf[0] = s;
wbuf[1]=s;
wbuf[2] =s;
wbuf[3] = s;
ier = mylib_write(outbin_tag,wbuf,1);

For simplicity you can assume that you are wrapping libpng library which reads RGB values from the picture and you plan process that data using your python tools.

So in Python I want to do:

import mylib
... #other calls
buf = range(4)
ier = mylib.read(bin_tag,buf,1)
if (ier >= 0):
   buf[0] = 3.14*buf[0]
ier = mylib.write(out_tag,buf,1)

I hope that now it's clear.

Thanks in Advance,
THOTH  
On Thu, Sep 2, 2010 at 9:07 AM, Robert Bradshaw <robertwb <at> math.washington.edu> wrote:
On Wed, Sep 1, 2010 at 7:28 PM, THOTH <anaryusifov <at> gmail.com> wrote:
> Hello, Everyone.
>
> Let me repeat the question in a shorter form.
>
> How I can access the following functions of my shared library:
>
> 1. Pointer based array get/put data:
> int mylib_read(
>      BIN_TAG bin_tag,
>      void *buf,
>      int count)
> // bin_tag, count - are input integer parameters, buf - is
> the array where data will be stored after processing
>
> 2. int cdds_write(
>      BIN_TAG bin_tag,
>      const void *buf,
>      int count)
> // now they all input parameters, but I still don't know how to pass
> void buff from python to my functions and vise versa.
>
> I use the following code (it's not copy paste so typos can appear -
> sorry) with no success:
>
> import cython
> cdef extern from "cdds.h":
>       int mylib_read(int bin_tag, void *buf, int count)
>
> def plib_read(int bin_tag, int count):
>   cdef int t
>   cython.declare(x = cython.float, x_ptr=cython.p_void)
>   # 't' will be <0 if there were an error and >= 0 if everything is
> OK
>   t = mylib_read(bin_tag, x, count)
>   # Basically I need only list of values in array 'x',
>   # so this return is to check success of read call
>   # In the future I plan to make it tuple(t,x) or something like
> that
>   return t
>
> Could there please anybody help me with this?

It's still really unclear what your question is. Are you wanting to do

 t = mylib_read(bin_tag, <void*>&x, count)

? Perhaps if you gave an example of using mylib_read in C that would help.

- Robert

Robert Bradshaw | 2 Sep 2010 08:12
Favicon

Re: Re: I/O C library wrapping

On Wed, Sep 1, 2010 at 9:54 PM, Anar Yusifov <anaryusifov <at> gmail.com> wrote:
> Hi Robert,
> I'm sorry for unclear question. bin_tag - is file descriptor to the data
> matrix. count - specifies number of rows to be read at once. x - is buffer
> which contains read data.
> Here is an Example (could be typos)
> #include <cdds.h>
> int ier;
> float s;
> float *buf = (float*)0;
> float *wbuf;
> buf = (float*)malloc(4); //assume that we have 4 values in each row
> wbuf = (float*)malloc(4);
> ... //using other routines I get bin_tag
> ier = mylib_read(bin_tag, buf, 1); // here I get 'ier' and 'buf'
> s = buf[0] + buf[1] + buf[2] +buf[3];
> //printf('%f',s);
> wbuf[0] = s;
> wbuf[1]=s;
> wbuf[2] =s;
> wbuf[3] = s;
> ier = mylib_write(outbin_tag,wbuf,1);
> For simplicity you can assume that you are wrapping libpng library which
> reads RGB values from the picture and you plan process that data using your
> python tools.
> So in Python I want to do:
> import mylib
> ... #other calls
> buf = range(4)
> ier = mylib.read(bin_tag,buf,1)
> if (ier >= 0):
>    buf[0] = 3.14*buf[0]
> ier = mylib.write(out_tag,buf,1)
> I hope that now it's clear.

To do that in Cython you're still going to have to malloc a float*
(and note that malloc(4) != malloc(4 * sizeof(float))). Thus you would
want to do something like

def read(int bin_tag, int count):
    cdef float *buf = (float*)malloc(count * sizeof(float)
    mylib_read(bin_tag, buf, count)
    py_buf = [buf[i] for i in range(count)]
    free(buf)
    return py_buf

Similarly for writing.

- Robert

Anar Yusifov | 2 Sep 2010 17:08
Picon

Re: Re: I/O C library wrapping

Thank you, Robert.


It looks reasonable, but don't work for some reason.
Here is the reason from cython:

cdef float *buf = (float*)malloc(nsamp*count*sizeof(float))
                            ^
------------------------------------------------------------

.pyx:107:29: Expected an identifier or literal

Any suggestions about?

Thanks in Advance,
THOTH.

On Thu, Sep 2, 2010 at 11:12 AM, Robert Bradshaw <robertwb <at> math.washington.edu> wrote:
On Wed, Sep 1, 2010 at 9:54 PM, Anar Yusifov <anaryusifov <at> gmail.com> wrote:
> Hi Robert,
> I'm sorry for unclear question. bin_tag - is file descriptor to the data
> matrix. count - specifies number of rows to be read at once. x - is buffer
> which contains read data.
> Here is an Example (could be typos)
> #include <cdds.h>
> int ier;
> float s;
> float *buf = (float*)0;
> float *wbuf;
> buf = (float*)malloc(4); //assume that we have 4 values in each row
> wbuf = (float*)malloc(4);
> ... //using other routines I get bin_tag
> ier = mylib_read(bin_tag, buf, 1); // here I get 'ier' and 'buf'
> s = buf[0] + buf[1] + buf[2] +buf[3];
> //printf('%f',s);
> wbuf[0] = s;
> wbuf[1]=s;
> wbuf[2] =s;
> wbuf[3] = s;
> ier = mylib_write(outbin_tag,wbuf,1);
> For simplicity you can assume that you are wrapping libpng library which
> reads RGB values from the picture and you plan process that data using your
> python tools.
> So in Python I want to do:
> import mylib
> ... #other calls
> buf = range(4)
> ier = mylib.read(bin_tag,buf,1)
> if (ier >= 0):
>    buf[0] = 3.14*buf[0]
> ier = mylib.write(out_tag,buf,1)
> I hope that now it's clear.

To do that in Cython you're still going to have to malloc a float*
(and note that malloc(4) != malloc(4 * sizeof(float))). Thus you would
want to do something like

def read(int bin_tag, int count):
   cdef float *buf = (float*)malloc(count * sizeof(float)
   mylib_read(bin_tag, buf, count)
   py_buf = [buf[i] for i in range(count)]
   free(buf)
   return py_buf

Similarly for writing.

- Robert

Stefan Behnel | 2 Sep 2010 18:26
Picon
Favicon

Re: Re: I/O C library wrapping

Anar Yusifov, 02.09.2010 17:08:
> cdef float *buf = (float*)malloc(nsamp*count*sizeof(float))

That was a typo. Try

     <float*>malloc ...

Stefan

Anar Yusifov | 2 Sep 2010 21:17
Picon

Re: Re: I/O C library wrapping

Thanks!

I tried it. It looks like OK. I also added free function like it done in here http://wiki.cython.org/DynamicMemoryAllocation.

I'll try it later today - let see if it works correctly.

There is also one question of mine:

How to wrap this function:

int mylib_scanf(
     const char *alias,
     const char *fmt,
     ...)

So, there is a multiple arguments can be passed at the end, with different types and ect.

Thanks in Advance,
THOTH.

On Thu, Sep 2, 2010 at 9:26 PM, Stefan Behnel <stefan_ml <at> behnel.de> wrote:
Anar Yusifov, 02.09.2010 17:08:

cdef float *buf = (float*)malloc(nsamp*count*sizeof(float))

That was a typo. Try

   <float*>malloc ...

Stefan

Craig Macomber | 4 Sep 2010 00:15
Picon

"unresolved external symbol" On Windows trying to call Panda3D's C++ API

I believe I might be seeing this issue, but I'm not sure:
http://trac.cython.org/cython_trac/ticket/106

I'm trying to compile a trivial Cython script that makes a call into
Panda3D's C++ API (Panda has a great python API, but it's easy to mix
with the C++ API, which seems like a nice job for Cython). It works
great for me on my Macs. On Windows, I have to use MSVC compiler (Form
Visual Studio 2008 Express) to work with Panda3D, and I get this error
when compiling:

   Creating library build\temp.win32-2.6\Release\testx.lib and object
build\temp
.win32-2.6\Release\testx.exp
testc.obj : error LNK2019: unresolved external symbol
"__declspec(dllimport) pub
lic: void __thiscall Geom::clear_cache(void)" (__imp_?
clear_cache <at> Geom <at>  <at> QAEXXZ)
referenced in function "struct _object * __cdecl
__pyx_pf_5testx_test(struct _ob
ject *,struct _object *)" (?
__pyx_pf_5testx_test <at>  <at> YAPAU_object <at>  <at> PAU1 <at> 0 <at> Z)
P:\CythonCXXTest\testc\testx.pyd : fatal error LNK1120: 1 unresolved
externals
error: command '"C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC
\BIN\link.
exe"' failed with exit status 1120

(Full output here: http://dpaste.com/238519/ )

My Cython source is:

cdef extern from "geom.h":
    cdef cppclass Geom:
        void clear_cache()

from panda3d.core import GeomVertexFormat,GeomVertexData
from panda3d.core import Geom as pGeom

print "Lets roll with C++"

format=GeomVertexFormat.getV3n3c4t2()
vdata=GeomVertexData('vdata', format, pGeom.UHStatic)
geom=pGeom(vdata)
v=geom.this

print "getting ptr"

cdef long dataPtr=v
cdef Geom* g=<Geom*>dataPtr

print "calling C++ method"

g.clear_cache()

print "done"

My setup.py:

import sys
import os

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

# make setup do what we want,
# build extension modules in place
sys.argv.append('build_ext')
sys.argv.append('--inplace')

if os.name=='nt':
    include_dirs=[r'C:\Panda3D-1.7.0\include']
elif os.name=='posix':
    include_dirs=['/Developer/Panda3D/include/']
else:
    # Specify Panda3D include directory here for other systems
    include_dirs=['']

setup(
    name = 'Demos',
    ext_modules=[
    Extension("testx",
              sources=["testc.pyx"],
              language="c++",
              include_dirs=include_dirs,
              ),
    ],
    cmdclass = {'build_ext': build_ext},
)
print "Importing module"
from panda3d.core import Geom
import testx

I have tried the the Rectangles C++ sample with this setup on Windows,
and it did work fine, and if I remove the Call to panda's C++ API, my
code works fine, so I'm pretty sure the issue is linked to the C++
call.

If possible, I'd like to actually make this work on Windows. Any
suggestions for what to try would be greatly appreciated as I'm
completely out of ideas. I'd be happy to do further testing if it
would help resolve the issue, but I don't know what to try.

Thanks!

Lisandro Dalcin | 4 Sep 2010 20:11
Picon
Gravatar

Re: "unresolved external symbol" On Windows trying to call Panda3D's C++ API

On 3 September 2010 19:15, Craig Macomber <fishyfish55 <at> gmail.com> wrote:
> I believe I might be seeing this issue, but I'm not sure:
> http://trac.cython.org/cython_trac/ticket/106
>

No, your issue is unrelated.

> I'm trying to compile a trivial Cython script that makes a call into
> Panda3D's C++ API (Panda has a great python API, but it's easy to mix
> with the C++ API, which seems like a nice job for Cython). It works
> great for me on my Macs. On Windows, I have to use MSVC compiler (Form
> Visual Studio 2008 Express) to work with Panda3D, and I get this error
> when compiling:
>
>   Creating library build\temp.win32-2.6\Release\testx.lib and object
> build\temp
> .win32-2.6\Release\testx.exp
> testc.obj : error LNK2019: unresolved external symbol
> "__declspec(dllimport) pub
> lic: void __thiscall Geom::clear_cache(void)" (__imp_?
> clear_cache <at> Geom <at>  <at> QAEXXZ)
> referenced in function "struct _object * __cdecl
> __pyx_pf_5testx_test(struct _ob
> ject *,struct _object *)" (?
> __pyx_pf_5testx_test <at>  <at> YAPAU_object <at>  <at> PAU1 <at> 0 <at> Z)
> P:\CythonCXXTest\testc\testx.pyd : fatal error LNK1120: 1 unresolved
> externals
> error: command '"C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC
> \BIN\link.
> exe"' failed with exit status 1120
>
> (Full output here: http://dpaste.com/238519/ )
>
> My Cython source is:
>
> cdef extern from "geom.h":
>    cdef cppclass Geom:
>        void clear_cache()
>

Is geom.h your own code or it is a panda3d header?

> from panda3d.core import GeomVertexFormat,GeomVertexData
> from panda3d.core import Geom as pGeom
>
> print "Lets roll with C++"
>
> format=GeomVertexFormat.getV3n3c4t2()
> vdata=GeomVertexData('vdata', format, pGeom.UHStatic)
> geom=pGeom(vdata)
> v=geom.this
>
> print "getting ptr"
>
> cdef long dataPtr=v
> cdef Geom* g=<Geom*>dataPtr
>
> print "calling C++ method"
>
> g.clear_cache()
>
> print "done"
>
>
>
> My setup.py:
>
> import sys
> import os
>
> from distutils.core import setup
> from distutils.extension import Extension
> from Cython.Distutils import build_ext
>
> # make setup do what we want,
> # build extension modules in place
> sys.argv.append('build_ext')
> sys.argv.append('--inplace')
>
> if os.name=='nt':
>    include_dirs=[r'C:\Panda3D-1.7.0\include']
> elif os.name=='posix':
>    include_dirs=['/Developer/Panda3D/include/']
> else:
>    # Specify Panda3D include directory here for other systems
>    include_dirs=['']
>
> setup(
>    name = 'Demos',
>    ext_modules=[
>    Extension("testx",
>              sources=["testc.pyx"],
>              language="c++",
>              include_dirs=include_dirs,
>              ),
>    ],
>    cmdclass = {'build_ext': build_ext},
> )
> print "Importing module"
> from panda3d.core import Geom
> import testx
>

Any chance you are missing to pass 'libraries' arg to Extension
constructor? panda3d should have a library, right?

>
> I have tried the the Rectangles C++ sample with this setup on Windows,
> and it did work fine, and if I remove the Call to panda's C++ API, my
> code works fine, so I'm pretty sure the issue is linked to the C++
> call.
>

Indeed, this smells to you not passing libraries and libray_dirs for
the linker to grab a panda3d library.

> If possible, I'd like to actually make this work on Windows.

...

> Any
> suggestions for what to try would be greatly appreciated as I'm
> completely out of ideas. I'd be happy to do further testing if it
> would help resolve the issue, but I don't know what to try.
>

Could you point a link to some panda3d docs were the way to build apps
calling panda3d API is explained?

--

-- 
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

Craig Macomber | 5 Sep 2010 04:43
Picon

Re: "unresolved external symbol" On Windows trying to call Panda3D's C++ API


> Is geom.h your own code or it is a panda3d header?

It's from Panda3d. Info here along with a link to the actual header: http://www.panda3d.org/reference/cxx/class!_geom

>
> Any chance you are missing to pass 'libraries' arg to Extension
> constructor? panda3d should have a library, right?

It was unneeded on Mac, but this did fix that issue on Windows. Thanks!
Now I have a pyd file that throws a "SystemError: dynamic module not  
initialized properly" when I import it.
Any suggestions for fixing this?

Some of the warnings are worrisome, but I don't understand what they  
mean.
The full output from the build is here:

P:\CythonCXXTest\testc>ppython setup.py
running build_ext
cythoning testc.pyx to testc.cpp
building 'testc' extension
creating build
creating build\temp.win32-2.6
creating build\temp.win32-2.6\Release
C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\cl.exe /c / 
nologo /Ox
/MD /W3 /GS- /DNDEBUG -IC:\Panda3D-1.7.0\include -IC: 
\Panda3D-1.7.0\python\inclu
de -IC:\Panda3D-1.7.0\python\PC /Tptestc.cpp /Fobuild 
\temp.win32-2.6\Release\tes
tc.obj
testc.cpp
C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\INCLUDE 
\xlocale(342) : war
ning C4530: C++ exception handler used, but unwind semantics are not  
enabled. Sp
ecify /EHsc
C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\INCLUDE 
\vector(439) : warn
ing C4275: non dll-interface class 'std::_Container_base_aux' used as  
base for d
ll-interface class 'std::_Container_base_aux_alloc_real<_Alloc>'
         with
         [
             _Alloc=pallocator_array<TypedWritable *>
         ]
         C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\INCLUDE 
\xutility(3
77) : see declaration of 'std::_Container_base_aux'
         c:\panda3d-1.7.0\include\vector_src.h(64) : see reference to  
class templ
ate instantiation 'std::vector<_Ty,_Ax>' being compiled
         with
         [
             _Ty=TypedWritable *,
             _Ax=pallocator_array<TypedWritable *>
         ]
C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\INCLUDE 
\vector(439) : warn
ing C4275: non dll-interface class 'std::_Container_base_aux' used as  
base for d
ll-interface class 'std::_Container_base_aux_alloc_real<_Alloc>'
         with
         [
             _Alloc=pallocator_array<std::string>
         ]
         C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\INCLUDE 
\xutility(3
77) : see declaration of 'std::_Container_base_aux'
         c:\panda3d-1.7.0\include\vector_src.h(64) : see reference to  
class templ
ate instantiation 'std::vector<_Ty,_Ax>' being compiled
         with
         [
             _Ty=std::string,
             _Ax=pallocator_array<std::string>
         ]
C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\INCLUDE 
\vector(439) : warn
ing C4275: non dll-interface class 'std::_Container_base_aux' used as  
base for d
ll-interface class 'std::_Container_base_aux_alloc_real<_Alloc>'
         with
         [
             _Alloc=pallocator_array<unsigned char>
         ]
         C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\INCLUDE 
\xutility(3
77) : see declaration of 'std::_Container_base_aux'
         c:\panda3d-1.7.0\include\vector_src.h(64) : see reference to  
class templ
ate instantiation 'std::vector<_Ty,_Ax>' being compiled
         with
         [
             _Ty=unsigned char,
             _Ax=pallocator_array<unsigned char>
         ]
C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\INCLUDE 
\vector(439) : warn
ing C4275: non dll-interface class 'std::_Container_base_aux' used as  
base for d
ll-interface class 'std::_Container_base_aux_alloc_real<_Alloc>'
         with
         [
             _Alloc=pallocator_array<int>
         ]
         C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\INCLUDE 
\xutility(3
77) : see declaration of 'std::_Container_base_aux'
         c:\panda3d-1.7.0\include\vector_src.h(64) : see reference to  
class templ
ate instantiation 'std::vector<_Ty,_Ax>' being compiled
         with
         [
             _Ty=int,
             _Ax=pallocator_array<int>
         ]
C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\link.exe / 
DLL /nologo
/INCREMENTAL:NO /LIBPATH:C:\Panda3D-1.7.0\python\libs /LIBPATH:C: 
\Panda3D-1.7.0\
python\PCbuild C:\Panda3D-1.7.0\lib\libp3framework.lib C: 
\Panda3D-1.7.0\lib\libp
anda.lib C:\Panda3D-1.7.0\lib\libpandafx.lib C:\Panda3D-1.7.0\lib 
\libpandaexpres
s.lib C:\Panda3D-1.7.0\lib\libp3dtool.lib C:\Panda3D-1.7.0\lib 
\libp3dtoolconfig.
lib C:\Panda3D-1.7.0\lib\libp3pystub.lib C:\Panda3D-1.7.0\lib 
\libp3direct.lib /E
XPORT:inittestc build\temp.win32-2.6\Release\testc.obj /OUT:P: 
\CythonCXXTest\tes
tc\testc.pyd /IMPLIB:build\temp.win32-2.6\Release\testc.lib / 
MANIFESTFILE:build\
temp.win32-2.6\Release\testc.pyd.manifest
    Creating library build\temp.win32-2.6\Release\testc.lib and object  
build\temp
.win32-2.6\Release\testc.exp
C:\Program Files\Microsoft SDKs\Windows\v6.0A\bin\mt.exe -nologo - 
manifest build
\temp.win32-2.6\Release\testc.pyd.manifest -outputresource:P: 
\CythonCXXTest\test
c\testc.pyd;2
Importing module
Traceback (most recent call last):
   File "setup.py", line 44, in <module>
     import testc
SystemError: dynamic module not initialized properly

P:\CythonCXXTest\testc>

>
> Could you point a link to some panda3d docs were the way to build apps
> calling panda3d API is explained?
>

This page lists the libraries I needed, as some other things:
http://www.panda3d.org/manual/index.php/How_to_build_a_CXX_Panda3D_game_using_Microsoft_Visual_Studio_2008

>
>
> -- 
> Lisandro Dalcin
> ---------------
> CIMEC (INTEC/CONICET-UNL)
> Predio CONICET-Santa Fe
> Colectora RN 168 Km 472, Paraje El Pozo
> Tel: +54-342-4511594 (ext 1011)
> Tel/Fax: +54-342-4511169


Gmane