General discussion about the Cython source code generator for Python extensions

Ondrej Certik | 3 Mar 2010 21:22

[Cython] using the C API Declarations in several C++ files

Hi,

I am using the C API Declarations[0], which generate the file
_hermes_common_api.h, that I include in my C++ files. I am having
problems with initializing the module properly.

Here is an example of my C++ function, that internally calls scipy's
sparse solver to solve the system:

void solve_linear_system_scipy_cg(CooMatrix *mat, double *res)
{
    if (import__hermes_common())
        throw std::runtime_error("hermes_common failed to import.");
    CSRMatrix M(mat);
    insert_object("m", c2py_CSRMatrix(&M));
    insert_object("rhs", c2numpy_double_inplace(res, mat->get_size()));
    cmd("A = m.to_scipy_csr()");
    cmd("from scipy.sparse.linalg import cg");
    cmd("x, res = cg(A, rhs)");
    double *x;
    int n;
    numpy2c_double_inplace(get_object("x"), &x, &n);
    memcpy(res, x, n*sizeof(double));
}

the functions insert_object(), cmd(), c2numpy_double_inplace() etc.
are declared in my .pyx file to ease C++ and Python integration (the
whole code is at [1]). Everything works, as long as I call the
"import__hermes_common()" function. Since I call Python things all
over my C++ code, it's kind of annoying to always call this by hand.

(Continue reading)

headers

Lisandro Dalcin | 3 Mar 2010 22:55

Re: [Cython] using the C API Declarations in several C++ files

On 3 March 2010 17:22, Ondrej Certik <ondrej <at> certik.cz> wrote:
> Hi,
>
> I am using the C API Declarations[0], which generate the file
> _hermes_common_api.h, that I include in my C++ files. I am having
> problems with initializing the module properly.
>
> Here is an example of my C++ function, that internally calls scipy's
> sparse solver to solve the system:
>
> void solve_linear_system_scipy_cg(CooMatrix *mat, double *res)
> {
>    if (import__hermes_common())
>        throw std::runtime_error("hermes_common failed to import.");
>    CSRMatrix M(mat);
>    insert_object("m", c2py_CSRMatrix(&M));
>    insert_object("rhs", c2numpy_double_inplace(res, mat->get_size()));
>    cmd("A = m.to_scipy_csr()");
>    cmd("from scipy.sparse.linalg import cg");
>    cmd("x, res = cg(A, rhs)");
>    double *x;
>    int n;
>    numpy2c_double_inplace(get_object("x"), &x, &n);
>    memcpy(res, x, n*sizeof(double));
> }
>
> the functions insert_object(), cmd(), c2numpy_double_inplace() etc.
> are declared in my .pyx file to ease C++ and Python integration (the
> whole code is at [1]). Everything works, as long as I call the
> "import__hermes_common()" function. Since I call Python things all

(Continue reading)

headers

Ondrej Certik | 4 Mar 2010 00:28

Re: [Cython] using the C API Declarations in several C++ files

On Wed, Mar 3, 2010 at 1:55 PM, Lisandro Dalcin <dalcinl <at> gmail.com> wrote:
> On 3 March 2010 17:22, Ondrej Certik <ondrej <at> certik.cz> wrote:
>> Hi,
>>
>> I am using the C API Declarations[0], which generate the file
>> _hermes_common_api.h, that I include in my C++ files. I am having
>> problems with initializing the module properly.
>>
>> Here is an example of my C++ function, that internally calls scipy's
>> sparse solver to solve the system:
>>
>> void solve_linear_system_scipy_cg(CooMatrix *mat, double *res)
>> {
>>    if (import__hermes_common())
>>        throw std::runtime_error("hermes_common failed to import.");
>>    CSRMatrix M(mat);
>>    insert_object("m", c2py_CSRMatrix(&M));
>>    insert_object("rhs", c2numpy_double_inplace(res, mat->get_size()));
>>    cmd("A = m.to_scipy_csr()");
>>    cmd("from scipy.sparse.linalg import cg");
>>    cmd("x, res = cg(A, rhs)");
>>    double *x;
>>    int n;
>>    numpy2c_double_inplace(get_object("x"), &x, &n);
>>    memcpy(res, x, n*sizeof(double));
>> }
>>
>> the functions insert_object(), cmd(), c2numpy_double_inplace() etc.
>> are declared in my .pyx file to ease C++ and Python integration (the
>> whole code is at [1]). Everything works, as long as I call the

(Continue reading)

headers

Lisandro Dalcin | 4 Mar 2010 00:54

Re: [Cython] using the C API Declarations in several C++ files

On 3 March 2010 20:28, Ondrej Certik <ondrej <at> certik.cz> wrote:
>>
>> Cython could grow support for your use case... If you really feel you
>> will need this (as opposed to implementing the "Python" class),
>> perhaps I could take a look at it and implement something...
>
> Do you have some ideas how Cython could be improved to make my use case easier?
>

The easier way to get something working would be to use a preprocessor
macro to control the storage (i.e. static vs. extern) of the symbols
you get in the generated C header... Then you just have to #include
the module API header like this:

#define CYTHON_USE_GLOBAL_API
#include <mymodule_api.h>

You will still need to call import_mymodule(), but you just need to do
it earlier in your code, likely a few lines below Py_Initialize()

Does this look good/easy enough for you?

> I can try to write something too --- I am still in the stage of
> figuring things out, not sure yet what I really want in details. I
> only know my general goal --- it should be easy for C++ users to use
> it, should be robust (no segfaults) and Python internals should not be
> visible at all. I am half way there already.
>

I have to insist: The best (though admittedly complex) way to do this

(Continue reading)

headers

Ondrej Certik | 4 Mar 2010 02:01

Re: [Cython] using the C API Declarations in several C++ files

On Wed, Mar 3, 2010 at 3:54 PM, Lisandro Dalcin <dalcinl@...> wrote:
> On 3 March 2010 20:28, Ondrej Certik <ondrej@...> wrote:
>>>
>>> Cython could grow support for your use case... If you really feel you
>>> will need this (as opposed to implementing the "Python" class),
>>> perhaps I could take a look at it and implement something...
>>
>> Do you have some ideas how Cython could be improved to make my use case easier?
>>
>
> The easier way to get something working would be to use a preprocessor
> macro to control the storage (i.e. static vs. extern) of the symbols
> you get in the generated C header... Then you just have to #include
> the module API header like this:
>
> #define CYTHON_USE_GLOBAL_API
> #include <mymodule_api.h>
>
> You will still need to call import_mymodule(), but you just need to do
> it earlier in your code, likely a few lines below Py_Initialize()
>
> Does this look good/easy enough for you?

That looks cool, I'll give it a shot.

>
>> I can try to write something too --- I am still in the stage of
>> figuring things out, not sure yet what I really want in details. I
>> only know my general goal --- it should be easy for C++ users to use
>> it, should be robust (no segfaults) and Python internals should not be

(Continue reading)

headers

Lisandro Dalcin | 4 Mar 2010 02:50

Re: [Cython] using the C API Declarations in several C++ files

On 3 March 2010 22:01, Ondrej Certik <ondrej <at> certik.cz> wrote:
> On Wed, Mar 3, 2010 at 3:54 PM, Lisandro Dalcin <dalcinl <at> gmail.com> wrote:
>> On 3 March 2010 20:28, Ondrej Certik <ondrej <at> certik.cz> wrote:
>>>>
>>>> Cython could grow support for your use case... If you really feel you
>>>> will need this (as opposed to implementing the "Python" class),
>>>> perhaps I could take a look at it and implement something...
>>>
>>> Do you have some ideas how Cython could be improved to make my use case easier?
>>>
>>
>> The easier way to get something working would be to use a preprocessor
>> macro to control the storage (i.e. static vs. extern) of the symbols
>> you get in the generated C header... Then you just have to #include
>> the module API header like this:
>>
>> #define CYTHON_USE_GLOBAL_API
>> #include <mymodule_api.h>
>>
>> You will still need to call import_mymodule(), but you just need to do
>> it earlier in your code, likely a few lines below Py_Initialize()
>>
>> Does this look good/easy enough for you?
>
> That looks cool, I'll give it a shot.
>

OK, but you will need an additional step... You have to create an
additional C source, #include the generated API header, and make it
'define' all these pointers... So perhaps you will need an additional

(Continue reading)

headers

Ondrej Certik | 4 Mar 2010 03:08

Re: [Cython] using the C API Declarations in several C++ files

On Wed, Mar 3, 2010 at 5:50 PM, Lisandro Dalcin <dalcinl@...> wrote:
> On 3 March 2010 22:01, Ondrej Certik <ondrej@...> wrote:
>> On Wed, Mar 3, 2010 at 3:54 PM, Lisandro Dalcin <dalcinl@...> wrote:
>>> On 3 March 2010 20:28, Ondrej Certik <ondrej@...> wrote:
>>>>>
>>>>> Cython could grow support for your use case... If you really feel you
>>>>> will need this (as opposed to implementing the "Python" class),
>>>>> perhaps I could take a look at it and implement something...
>>>>
>>>> Do you have some ideas how Cython could be improved to make my use case easier?
>>>>
>>>
>>> The easier way to get something working would be to use a preprocessor
>>> macro to control the storage (i.e. static vs. extern) of the symbols
>>> you get in the generated C header... Then you just have to #include
>>> the module API header like this:
>>>
>>> #define CYTHON_USE_GLOBAL_API
>>> #include <mymodule_api.h>
>>>
>>> You will still need to call import_mymodule(), but you just need to do
>>> it earlier in your code, likely a few lines below Py_Initialize()
>>>
>>> Does this look good/easy enough for you?
>>
>> That looks cool, I'll give it a shot.
>>
>
> OK, but you will need an additional step... You have to create an
> additional C source, #include the generated API header, and make it

(Continue reading)

headers

Lisandro Dalcin | 4 Mar 2010 03:23

Re: [Cython] using the C API Declarations in several C++ files

On 3 March 2010 23:08, Ondrej Certik <ondrej <at> certik.cz> wrote:
>>
>> Am I being clear enough?
>
> Yes, that is clear to me, thanks.
>

OK. Once you have something working, let's polish it and ask other for
opinions, then push to cython-devel.

>
> That is already implemented, here is my exec function:
>
> cdef api object run_cmd(const_char_p text, object namespace):
>    try:
>        verbose = namespace.get("verbose")
>        if verbose:
>            print "got a text:", text
>        if verbose:
>            print "evaluting in the namespace:"
>            print namespace
>        code = compile(text, "", "exec")
>        eval(code, {}, namespace)
>        if verbose:
>            print "new namespace:"
>            print namespace
>        return namespace
>    except SystemExit, e:
>        try:
>            exit_code = int(e)

(Continue reading)

headers

Greg Ewing | 4 Mar 2010 04:06

Re: [Cython] using the C API Declarations in several C++ files

Ondrej Certik wrote:

> and add() is implemented in C++, and internally it calls some stuff in
> Python (like scipy). In principle it could be recursively nested
> couple times. Can the Python interpreter be confused by this
> "nesting"?

No, there shouldn't be any problem calling back and
forth between Python and extension code as deep as
you want (subject to the Python recursion depth
limitation).

--

-- 
Greg

headers

Toni Moll | 4 Mar 2010 04:22

[Cython] pxd pyx parser bug for mismatch in function argument with default value

Hello,

this is a bug report for cython 0.12.1. The bug occurs when a function argument with default value is named differently in the .pyx and the .pxd file.
A minimal example is:

foo.pyx contains:
===========================
cpdef test(test=5):
pass
===========================

foo.pxd contains:
===========================
cpdef test(FJKHFKLJDHF=*)
===========================

the traceback is:

/tmp/Cython-0.12.1$ python2.5 cython.py foo.pyx
Traceback (most recent call last):
File "cython.py", line 8, in <module>
main(command_line = 1)
File "/tmp/Cython-0.12.1/Cython/

Compiler/Main.py", line 743, in main
    result = compile(sources, options)
File "/tmp/Cython-0.12.1/Cython/Compiler/Main.py", line 718, in compile
    return compile_multiple(source, options)
File "/tmp/Cython-0.12.1/Cython/Compiler/Main.py", line 690, in compile_multiple
    result = run_pipeline(source, options)
File "/tmp/Cython-0.12.1/Cython/Compiler/Main.py", line 561, in run_pipeline
    err, enddata = context.run_pipeline(pipeline, source)
File "/tmp/Cython-0.12.1/Cython/Compiler/Main.py", line 221, in run_pipeline
    data = phase(data)
File "/tmp/Cython-0.12.1/Cython/Compiler/Main.py", line 154, in generate_pyx_code
    module_node.process_implementation(options, result)
File "/tmp/Cython-0.12.1/Cython/Compiler/ModuleNode.py", line 72, in process_implementation
    self.generate_c_code(env, options, result)
File "/tmp/Cython-0.12.1/Cython/Compiler/ModuleNode.py", line 274, in generate_c_code
    self.body.generate_function_definitions(env, code)
File "/tmp/Cython-0.12.1/Cython/Compiler/Nodes.py", line 341, in generate_function_definitions
    stat.generate_function_definitions(env, code)
File "/tmp/Cython-0.12.1/Cython/Compiler/Nodes.py", line 1106, in generate_function_definitions
    self.generate_argument_parsing_code(env, code)
File "/tmp/Cython-0.12.1/Cython/Compiler/Nodes.py", line 1493, in generate_argument_parsing_code
    code.putln('%s = %s->%s;' % (arg.cname, Naming.optional_args_cname, self.type.opt_arg_cname(declarator.name)))
File "/tmp/Cython-0.12.1/Cython/Compiler/PyrexTypes.py", line 1571, in opt_arg_cname
    return self.op_arg_struct.base_type.scope.lookup(arg_name).cname
AttributeError: 'NoneType' object has no attribute 'cname'

best regards and thanks a lot for cython, I like it a lot!
Toni

<div>
<p>Hello,<br><br>this is a bug report for cython 0.12.1. The bug occurs
when a function argument with default value is named differently in
the .pyx and the .pxd file.<br>A minimal example is:<br><br>foo.pyx contains:<br>===========================<br>
cpdef test(test=5):<br>&nbsp;&nbsp;&nbsp; pass<br>===========================<br><br>foo.pxd contains:<br>===========================<br>cpdef test(FJKHFKLJDHF=*)<br>===========================<br><br>the traceback is:<br><br>/tmp/Cython-0.12.1$ python2.5 cython.py foo.pyx <br>

Traceback (most recent call last):<br>&nbsp; File "cython.py", line 8, in &lt;module&gt;<br>&nbsp;&nbsp;&nbsp; main(command_line = 1)<br>&nbsp; File "/tmp/Cython-0.12.1/Cython/≤/p>
<div class="ii gt">Compiler/Main.py", line 743, in main<br>
&nbsp;&nbsp;&nbsp; result = compile(sources, options)<br>
&nbsp; File "/tmp/Cython-0.12.1/Cython/Compiler/Main.py", line 718, in compile<br>&nbsp;&nbsp;&nbsp; return compile_multiple(source, options)<br>&nbsp; File "/tmp/Cython-0.12.1/Cython/Compiler/Main.py", line 690, in compile_multiple<br>

&nbsp;&nbsp;&nbsp; result = run_pipeline(source, options)<br>&nbsp; File "/tmp/Cython-0.12.1/Cython/Compiler/Main.py", line 561, in run_pipeline<br>&nbsp;&nbsp;&nbsp; err, enddata = context.run_pipeline(pipeline, source)<br>&nbsp; File "/tmp/Cython-0.12.1/Cython/Compiler/Main.py", line 221, in run_pipeline<br>

&nbsp;&nbsp;&nbsp; data = phase(data)<br>&nbsp; File "/tmp/Cython-0.12.1/Cython/Compiler/Main.py", line 154, in generate_pyx_code<br>&nbsp;&nbsp;&nbsp; module_node.process_implementation(options, result)<br>&nbsp; File "/tmp/Cython-0.12.1/Cython/Compiler/ModuleNode.py", line 72, in process_implementation<br>

&nbsp;&nbsp;&nbsp; self.generate_c_code(env, options, result)<br>&nbsp; File "/tmp/Cython-0.12.1/Cython/Compiler/ModuleNode.py", line 274, in generate_c_code<br>&nbsp;&nbsp;&nbsp; self.body.generate_function_definitions(env, code)<br>&nbsp; File "/tmp/Cython-0.12.1/Cython/Compiler/Nodes.py", line 341, in generate_function_definitions<br>

&nbsp;&nbsp;&nbsp; stat.generate_function_definitions(env, code)<br>&nbsp; File "/tmp/Cython-0.12.1/Cython/Compiler/Nodes.py", line 1106, in generate_function_definitions<br>&nbsp;&nbsp;&nbsp; self.generate_argument_parsing_code(env, code)<br>&nbsp; File "/tmp/Cython-0.12.1/Cython/Compiler/Nodes.py", line 1493, in generate_argument_parsing_code<br>

&nbsp;&nbsp;&nbsp; code.putln('%s = %s-&gt;%s;' % (arg.cname, Naming.optional_args_cname, self.type.opt_arg_cname(<a href="http://declarator.name/" target="_blank">declarator.name</a>)))<br>&nbsp; File "/tmp/Cython-0.12.1/Cython/Compiler/PyrexTypes.py", line 1571, in opt_arg_cname<br>

&nbsp;&nbsp;&nbsp; return self.op_arg_struct.base_type.scope.lookup(arg_name).cname<br>AttributeError: 'NoneType' object has no attribute 'cname'<br><br>best regards and thanks a lot for cython, I like it a lot!<br>Toni<br>
</div>
</div>

Mon	Tue	Wed	Thu	Fri	Sat	Sun

1	2	3	4	5	6	7
8	9	10	11	12	13	14
15	16	17	18	19	20	21
22	23	24	25	26	27	28
29	30	31

24	June 2013
54	May 2013
129	April 2013
87	March 2013
75	February 2013
31	January 2013
30	December 2012
33	November 2012
45	October 2012
56	September 2012
154	August 2012
128	July 2012
124	June 2012
279	May 2012
235	April 2012
158	March 2012
146	February 2012
125	January 2012
20	December 2011
43	November 2011
196	October 2011
116	September 2011
184	August 2011
245	July 2011
72	June 2011
220	May 2011
294	April 2011
217	March 2011
105	February 2011
200	January 2011
237	December 2010
377	November 2010
62	October 2010
124	September 2010
232	August 2010
288	July 2010
88	June 2010
135	May 2010
230	April 2010
222	March 2010
181	February 2010
111	January 2010
253	December 2009
245	November 2009
607	October 2009
360	September 2009
219	August 2009
206	July 2009
357	June 2009
639	May 2009
527	April 2009
678	March 2009
406	February 2009
181	January 2009
274	December 2008
291	November 2008
418	October 2008
256	September 2008
320	August 2008
215	July 2008
284	June 2008
705	May 2008
398	April 2008
223	March 2008
65	February 2008
133	January 2008
137	December 2007
82	November 2007
49	October 2007
151	September 2007
32	August 2007