Re: Cython error :: UnicodeDecodeError

Re: Cython error :: UnicodeDecodeError

typedef problem

Re: typedef problem

On 1 July 2010 14:01, Rafael Cardoso Dias Costa <rfcard <at> gmail.com> wrote:
> Hi, guys!
>
> How I translate in Cython the follow header statements?
>
> /**
>  * The EMSO reporting call back function prototype.
>  * This function receives a string  <at> a str and its length  <at> a len.
>  * The expected behavior is to print the given string somewhere and
>  * return the number of characters actually printed.
>  *
>  * A very simple implementation which prints into the stdout could be:
>  *  <at> code
>  * int_t simple_report(const char *str, int_t len){
>  *     char buff[256];
>  *     int_t ret = snprintf(buff, len, str);
>  *     printf(buff);
>  *     return ret;
>  * }
>  *  <at> endcode
>  */
> typedef int_t(EMSO_REPORT)(const char *str, int_t len);
>
>

cdef extern from *: #
    ctypedef int int_t
    ctypedef char const_char "const_char"
    ctypedef int_t EMSO_REPORT(const_char *str, int_t len)
    EMSO_REPORT simple_report

--

-- 
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

Re: typedef problem

Expose the c function to the Python environment

Re: Expose the c function to the Python environment

On 1 July 2010 16:23, Rafael Cardoso Dias Costa <rfcard <at> gmail.com> wrote:
> Guys,
>
> I wish to expose the c function to the Python environment. How do I?
> I'm trying as follow, but is giving error.
>
> cdef extern from "emso_dll.h":
>     ctypedef long int_t "int_t"
>     ctypedef char const_char "const_char"
>     ctypedef int_t EMSO_REPORT(const_char *str, int_t len)
>     void emso_init(EMSO_REPORT *out, EMSO_REPORT *err, const_char *path, \
>         const_char *path_dll, int_t *ret_val)
>
> cpdef py_emso_init(EMSO_REPORT *out, EMSO_REPORT *err, const_char *path, \
>         const_char *path_dll, int_t *ret_val):
>     emso_init(out, err, path, path_dll, ret_val)
>

This is a pretty standard C-wrapping task 

You cannot pass pointers from Python code, then you have to accept
Python callables, use a couple of global vars, and a wrapper C
function: more or less like below.

def object out_callable = None
def object err_callable = None

cdef int_t out_callback(const_char *s, int_t n):
    if out_callable is None: return 0
    ret = out_callable(s)
    if ret is None: return 0
    return ret

cdef int_t err_callback(const_char *s, int_t n):
    if err_callable is None: return 0
    ret = err_callable(s)
    if ret is None: return 0
    return ret

def init(path, path_dll, out=None, err=None):
    global out_callable, err_callable
    out_callable = out
    err_callable = err
    cdef int_t ret_val = 0
    emso_init(out_callback, err_callback, path, path_dll, &ret_val)
    return ret_val

Disclaimer: This was written in the browser, so is untested, I leave
the details to you.

--

-- 
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

Re: Expose the c function to the Python environment

Re: Expose the c function to the Python environment

On 2 July 2010 10:14, Rafael Cardoso Dias Costa <rfcard <at> gmail.com> wrote:
> Thank you, Lisandro!
>
> I'm testing the code that you wrote.
> I do not understand why Cython not doing the automatic conversion of types.
> He gives the following error message:
>

Try to add explicit casts to <char*>, like this out_callable(<char*>s)

>
>
> Error converting Pyrex file to C:
> ------------------------------------------------------------
> ...
> cdef object out_callable = None
> cdef object err_callable = None
>
> cdef int_t out_callback(const_char *s, int_t n):
>    if out_callable is None: return 0
>    ret = out_callable(s)
>                       ^
> ------------------------------------------------------------
>
> /home/rfcard/emso-python/CythonEMSO/emso.pyx:14:23: Cannot convert
> 'emso.const_char *' to Python object
>
>
>
>
> I changed "def" with "cdef" in indicated lines.
>
>
> cdef extern from "emso_dll.h":
>     ctypedef long int_t "int_t"
>     ctypedef char const_char "const_char"
>     ctypedef int_t EMSO_REPORT(const_char *str, int_t len)
>     void emso_init(EMSO_REPORT *out, EMSO_REPORT *err, const_char *path, \
>         const_char *path_dll, int_t *ret_val)
>
>
> cdef object out_callable = None  <==========================
> cdef object err_callable = None   <==========================
>
> cdef int_t out_callback(const_char *s, int_t n):
>    if out_callable is None: return 0
>    ret = out_callable(s)
>    if ret is None: return 0
>    return ret
>
> cdef int_t err_callback(const_char *s, int_t n):
>    if err_callable is None: return 0
>    ret = err_callable(s)
>    if ret is None: return 0
>    return ret
>
> def init(path, path_dll, out=None, err=None):
>    global out_callable, err_callable
>    out_callable = out
>    err_callable = err
>    cdef int_t ret_val = 0
>    emso_init(out_callback, err_callback, path, path_dll, &ret_val)
>    return ret_val
>
>
>
> I await answers.
> Thank you!
>
>
> --
> Rafael Cardoso Dias Costa
> Control and Automation Engineer - UFMG
> Chemical Engineering Master Student - Process Control - UFRGS
>
>
>
>
>
>
>
> On Thu, Jul 1, 2010 at 4:36 PM, Lisandro Dalcin <dalcinl <at> gmail.com> wrote:
>>
>> On 1 July 2010 16:23, Rafael Cardoso Dias Costa <rfcard <at> gmail.com> wrote:
>> > Guys,
>> >
>> > I wish to expose the c function to the Python environment. How do I?
>> > I'm trying as follow, but is giving error.
>> >
>> > cdef extern from "emso_dll.h":
>> >     ctypedef long int_t "int_t"
>> >     ctypedef char const_char "const_char"
>> >     ctypedef int_t EMSO_REPORT(const_char *str, int_t len)
>> >     void emso_init(EMSO_REPORT *out, EMSO_REPORT *err, const_char *path,
>> > \
>> >         const_char *path_dll, int_t *ret_val)
>> >
>> > cpdef py_emso_init(EMSO_REPORT *out, EMSO_REPORT *err, const_char *path,
>> > \
>> >         const_char *path_dll, int_t *ret_val):
>> >     emso_init(out, err, path, path_dll, ret_val)
>> >
>>
>> This is a pretty standard C-wrapping task 
>>
>> You cannot pass pointers from Python code, then you have to accept
>> Python callables, use a couple of global vars, and a wrapper C
>> function: more or less like below.
>>
>> def object out_callable = None
>> def object err_callable = None
>>
>> cdef int_t out_callback(const_char *s, int_t n):
>>    if out_callable is None: return 0
>>    ret = out_callable(s)
>>    if ret is None: return 0
>>    return ret
>>
>> cdef int_t err_callback(const_char *s, int_t n):
>>    if err_callable is None: return 0
>>    ret = err_callable(s)
>>    if ret is None: return 0
>>    return ret
>>
>> def init(path, path_dll, out=None, err=None):
>>    global out_callable, err_callable
>>    out_callable = out
>>    err_callable = err
>>    cdef int_t ret_val = 0
>>    emso_init(out_callback, err_callback, path, path_dll, &ret_val)
>>    return ret_val
>>
>> Disclaimer: This was written in the browser, so is untested, I leave
>> the details to you.
>>
>> --
>> Lisandro Dalcin
>> ---------------
>> CIMEC (INTEC/CONICET-UNL)
>> Predio CONICET-Santa Fe
>> Colectora RN 168 Km 472, Paraje El Pozo
>> Tel: +54-342-4511594 (ext 1011)
>> Tel/Fax: +54-342-4511169
>
>
>

--

-- 
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

Fast parsing

Hi,

I have a bunch of programs for which the slowest part is reading text-
based data into an internal datastructure. A simple situation might
look something like this:

for l in input:
    line = l.rstrip().split()
    foo = int(line[1]
    bar = float(line[2])
    data[foo] = bar

Where data might be a dict, a numpy array or even a pytables object
(hdf5). The processing can be more complex, but the files are almost
always in simple 'table' like formats.
Even very simple examples are often CPU and not disk-limited in
standard python. Does anyone here have experience with speeding such
loops up? Just dropping the whole thing into cython does not seem to
do much (probably because of all the python calls and conversions).
Does using C string routines help? Are there already fast parsers for
this kind of simple situation around? Just looking for some experience
from others before reinventing the wheel.

Thanks
  Felix