Michael Kamensky | 5 Aug 2010 05:26
Picon

[Cython] A couple bug reports + possibly suboptimal solutions for them

Hello,

I'd like to report a bug with Cython 0.12.1 which is also present in the latest SVN version:
embedding an interpreter with the --embed option (to create the .exe file) fails on MS Windows
with the error "undefined reference to WinMain <at> 16". I was able to overcome that error by
changing this line in the output .c file:

int wmain(int argc, wchar_t **argv) {

into this:

int main(int argc, wchar_t **argv) {


The second problem (also on MS Windows) is that sys.argv do not get parsed correctly due to
the fact that it seems like the command line is parsed as "char**", but Python expects "wchar_t**".
Here's what I did to overcome the issue:

I changed this:

  PySys_SetArgv(argc, argv);    __pyx_module_is_main_test1 = 1;

Into this:

  //PySys_SetArgv(argc, argv);    __pyx_module_is_main_test1 = 1;
  PyObject *py_argv= PyList_New(0);
  int i;
      for (i=0; i<argc; i++) {
          PyObject *item= PyUnicode_Decode(argv[i], strlen(argv[i]), Py_FileSystemDefaultEncoding, NULL);
      if(item==NULL) { // should never happen
          PyErr_Print();
          PyErr_Clear();
      }
      else {
          PyList_Append(py_argv, item);
          Py_DECREF(item);
      }
      }
      PySys_SetObject("argv", py_argv);
      Py_DECREF(py_argv);


I'm not sure if it's optimal, I do get warnings during compilation, but at least it works (it might not be cross-platform
or whatever, so it's just a proposed bug report with a demo as to how I solved the issue). Hope it helps. :)

I love your project, thank you so much for Cython! :D

Best regards,
Michael

<div><p>Hello,
<br><br>I'd like to report a bug with Cython 0.12.1 which is also present in the 
latest SVN version:
<br>embedding an interpreter with the --embed option (to create the .exe 
file) fails on MS Windows
<br>with the error "undefined reference to WinMain <at> 16". I was able to 
overcome that error by
<br>changing this line in the output .c file:
<br><br>int wmain(int argc, wchar_t **argv) {
<br><br>into this:
<br><br>int main(int argc, wchar_t **argv) {
<br><br><br>The second problem (also on MS Windows) is that sys.argv do not get 
parsed correctly due to
<br>the fact that it seems like the command line is parsed as "char**", but 
Python expects "wchar_t**".
<br>Here's what I did to overcome the issue:
<br><br>I changed this:
<br><br>&nbsp; PySys_SetArgv(argc, argv);&nbsp;&nbsp;&nbsp; __pyx_module_is_main_test1 = 1;
<br><br>Into this:
<br><br>&nbsp; //PySys_SetArgv(argc, argv);&nbsp;&nbsp;&nbsp; __pyx_module_is_main_test1 = 1;
<br>&nbsp; PyObject *py_argv= PyList_New(0);
<br>&nbsp; int i;
<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; for (i=0; i&lt;argc; i++) {
<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; PyObject *item= PyUnicode_Decode(argv[i], strlen(argv[i]), 
Py_FileSystemDefaultEncoding, NULL);
<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; if(item==NULL) { // should never happen
<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; PyErr_Print();
<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; PyErr_Clear();
<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; }
<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; else {
<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; PyList_Append(py_argv, item);
<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Py_DECREF(item);
<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; }
<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; }
<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; PySys_SetObject("argv", py_argv);
<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Py_DECREF(py_argv);
<br><br><br>I'm not sure if it's optimal, I do get warnings during compilation, but 
at least it works (it might not be cross-platform
<br>or whatever, so it's just a proposed bug report with a demo as to how I 
solved the issue). Hope it helps. <span class="moz-smiley-s1"><span> :) </span></span>
<br><br>I love your project, thank you so much for Cython! :D
<br><br>Best regards,
<br>Michael
<br><br></p></div>
Carl Witty | 5 Aug 2010 06:01
Picon

[Cython] working on Cython for IronPython

I'm working with Enthought on a branch of Cython to target .NET for
IronPython.  I'm maintaining a repository at
http://bitbucket.org/cwitty/cython-for-ironpython (I just started
writing code, so there's not much interesting there yet).

I'm hopeful that eventually this code will be merged back into Cython;
I don't want to fork Cython.

I'm starting by essentially duplicating all of the generate_* methods
into generate_dotnet_* methods, and modifying the duplicated methods
to generate C++/CLI code.  My hope is that once I have a working
compiler, I can refactor to merge most of the methods back together
(eliminating the duplicate code) while still not being very intrusive
to the original codebase.

Carl
Dag Sverre Seljebotn | 5 Aug 2010 09:08
Picon
Picon

Re: [Cython] working on Cython for IronPython

On 08/05/2010 06:01 AM, Carl Witty wrote:
> I'm working with Enthought on a branch of Cython to target .NET for
> IronPython.  I'm maintaining a repository at
> http://bitbucket.org/cwitty/cython-for-ironpython (I just started
> writing code, so there's not much interesting there yet).
>    
Great, I think this is fantastic news.
> I'm hopeful that eventually this code will be merged back into Cython;
> I don't want to fork Cython.
>    
I hope this stays in Cython as well.
> I'm starting by essentially duplicating all of the generate_* methods
> into generate_dotnet_* methods, and modifying the duplicated methods
> to generate C++/CLI code.  My hope is that once I have a working
>    
Ah, this is very interesting -- when I've spoken with Enthought on the 
matter we've largely ignored C++ for this and were talking about a C# 
backend.
> compiler, I can refactor to merge most of the methods back together
> (eliminating the duplicate code) while still not being very intrusive
> to the original codebase.
>    

Have you considered writing a transform instead of adding more methods 
to the nodes? E.g. something like Cython/CodeWriter.py for the .NET 
backend? That would be even less intrusive...ideally, one would just 
configure the pipeline differently depending on what the backend is.

(I once suggested that the CPython generation was also moved into a 
seperate generator class like this as well; of course, that's a lot of 
unnecessary work)

I'm just not too sure about having "if dotnet:" all over the place 
(eventually). Of course, you should just do whatever gives results 
quickest at this point (and I very much trust your judgement), I'm just 
making sure the option is mentioned.

Dag Sverre
Robert Bradshaw | 5 Aug 2010 10:27
Favicon

Re: [Cython] working on Cython for IronPython

On Thu, Aug 5, 2010 at 12:08 AM, Dag Sverre Seljebotn
<dagss@...> wrote:
> On 08/05/2010 06:01 AM, Carl Witty wrote:
>> I'm working with Enthought on a branch of Cython to target .NET for
>> IronPython.  I'm maintaining a repository at
>> http://bitbucket.org/cwitty/cython-for-ironpython (I just started
>> writing code, so there's not much interesting there yet).
>>
> Great, I think this is fantastic news.

I'm excited about this too.

>> I'm hopeful that eventually this code will be merged back into Cython;
>> I don't want to fork Cython.
>>
> I hope this stays in Cython as well.
>> I'm starting by essentially duplicating all of the generate_* methods
>> into generate_dotnet_* methods, and modifying the duplicated methods
>> to generate C++/CLI code.  My hope is that once I have a working
>>
> Ah, this is very interesting -- when I've spoken with Enthought on the
> matter we've largely ignored C++ for this and were talking about a C#
> backend.
>> compiler, I can refactor to merge most of the methods back together
>> (eliminating the duplicate code) while still not being very intrusive
>> to the original codebase.
>>
>
> Have you considered writing a transform instead of adding more methods
> to the nodes? E.g. something like Cython/CodeWriter.py for the .NET
> backend? That would be even less intrusive...ideally, one would just
> configure the pipeline differently depending on what the backend is.

+1 The transform could go through and replace any nodes that need
special casing with special dot_net versions (that only need to know
how to generate their code). This would keep things nice and modular.
(The real question is how much code could be shared between the
CPython bindings and the IronPython ones--the raw C code generation
would probably be virtually identical.)

> (I once suggested that the CPython generation was also moved into a
> seperate generator class like this as well; of course, that's a lot of
> unnecessary work)
>
> I'm just not too sure about having "if dotnet:" all over the place
> (eventually). Of course, you should just do whatever gives results
> quickest at this point (and I very much trust your judgement), I'm just
> making sure the option is mentioned.

On this note, it would be cool if, eventually, Jython could be
supported through similar mechanisms.

- Robert
Dag Sverre Seljebotn | 5 Aug 2010 11:39
Picon
Picon

Re: [Cython] working on Cython for IronPython

Robert Bradshaw wrote:
> On Thu, Aug 5, 2010 at 12:08 AM, Dag Sverre Seljebotn
> <dagss@...> wrote:
>   
>> On 08/05/2010 06:01 AM, Carl Witty wrote:
>>     
>>> I'm working with Enthought on a branch of Cython to target .NET for
>>> IronPython.  I'm maintaining a repository at
>>> http://bitbucket.org/cwitty/cython-for-ironpython (I just started
>>> writing code, so there's not much interesting there yet).
>>>
>>>       
>> Great, I think this is fantastic news.
>>     
>
> I'm excited about this too.
>
>   
>>> I'm hopeful that eventually this code will be merged back into Cython;
>>> I don't want to fork Cython.
>>>
>>>       
>> I hope this stays in Cython as well.
>>     
>>> I'm starting by essentially duplicating all of the generate_* methods
>>> into generate_dotnet_* methods, and modifying the duplicated methods
>>> to generate C++/CLI code.  My hope is that once I have a working
>>>
>>>       
>> Ah, this is very interesting -- when I've spoken with Enthought on the
>> matter we've largely ignored C++ for this and were talking about a C#
>> backend.
>>     
>>> compiler, I can refactor to merge most of the methods back together
>>> (eliminating the duplicate code) while still not being very intrusive
>>> to the original codebase.
>>>
>>>       
>> Have you considered writing a transform instead of adding more methods
>> to the nodes? E.g. something like Cython/CodeWriter.py for the .NET
>> backend? That would be even less intrusive...ideally, one would just
>> configure the pipeline differently depending on what the backend is.
>>     
>
> +1 The transform could go through and replace any nodes that need
> special casing with special dot_net versions (that only need to know
> how to generate their code). This would keep things nice and modular.
> (The real question is how much code could be shared between the
> CPython bindings and the IronPython ones--the raw C code generation
> would probably be virtually identical.)
>   
I was actually thinking that one had the "transform" (or rather "sink") 
just emit/serialize code directly when visiting a node. The "return 
value" could either be string fragments, or if that is too inefficient, 
nothing at all.

def visit_BinopNode(self, node):
    # Very naive, I know things are more complicated...
    return '%s %s %s' % (self.visit(node.operand1), node.operator, 
self.visit(node.operand2))

def visit_FuncNode(self, node):
    # Here we would return a StringIOTree instead (see 
Cython/StringIOTree.py),
    # for efficiency.

or, perhaps just:

def visit_BinopNode(self, node):
    self.visit(node.operand1)
    self.put(node.operator)
    self.visit(node.operand2)

So generate_... would never be called, on *any* node, when doing .NET 
output.

One could deal with shared CPython/.NET code by creating a superclass:

class CLikeCodeGenerator:
    # pull out stuff that is common between CPython and .NET here

class DotNetCodeGenerator(CLikeCodeGenerator):
    # .NET-specific nodes handled here

class CPythonCodeGenerator(CLikeCodeGenerator):
    def visit_Node(self, node):
        # for now, just call generate_... if the node wasn't handled
        # by CLikeCodeGenerator

This makes it easy to plug in Java as well without making Nodes.py and 
ExprNodes.py even larger and more tangled and more full of conditionals...

Dag Sverre
Dag Sverre Seljebotn | 5 Aug 2010 11:43
Picon
Picon

Re: [Cython] working on Cython for IronPython

Dag Sverre Seljebotn wrote:
> Robert Bradshaw wrote:
>   
>> On Thu, Aug 5, 2010 at 12:08 AM, Dag Sverre Seljebotn
>> <dagss@...> wrote:
>>   
>>     
>>> On 08/05/2010 06:01 AM, Carl Witty wrote:
>>>     
>>>       
>>>> I'm working with Enthought on a branch of Cython to target .NET for
>>>> IronPython.  I'm maintaining a repository at
>>>> http://bitbucket.org/cwitty/cython-for-ironpython (I just started
>>>> writing code, so there's not much interesting there yet).
>>>>
>>>>       
>>>>         
>>> Great, I think this is fantastic news.
>>>     
>>>       
>> I'm excited about this too.
>>
>>   
>>     
>>>> I'm hopeful that eventually this code will be merged back into Cython;
>>>> I don't want to fork Cython.
>>>>
>>>>       
>>>>         
>>> I hope this stays in Cython as well.
>>>     
>>>       
>>>> I'm starting by essentially duplicating all of the generate_* methods
>>>> into generate_dotnet_* methods, and modifying the duplicated methods
>>>> to generate C++/CLI code.  My hope is that once I have a working
>>>>
>>>>       
>>>>         
>>> Ah, this is very interesting -- when I've spoken with Enthought on the
>>> matter we've largely ignored C++ for this and were talking about a C#
>>> backend.
>>>     
>>>       
>>>> compiler, I can refactor to merge most of the methods back together
>>>> (eliminating the duplicate code) while still not being very intrusive
>>>> to the original codebase.
>>>>
>>>>       
>>>>         
>>> Have you considered writing a transform instead of adding more methods
>>> to the nodes? E.g. something like Cython/CodeWriter.py for the .NET
>>> backend? That would be even less intrusive...ideally, one would just
>>> configure the pipeline differently depending on what the backend is.
>>>     
>>>       
>> +1 The transform could go through and replace any nodes that need
>> special casing with special dot_net versions (that only need to know
>> how to generate their code). This would keep things nice and modular.
>> (The real question is how much code could be shared between the
>> CPython bindings and the IronPython ones--the raw C code generation
>> would probably be virtually identical.)
>>   
>>     
> I was actually thinking that one had the "transform" (or rather "sink") 
> just emit/serialize code directly when visiting a node. The "return 
> value" could either be string fragments, or if that is too inefficient, 
> nothing at all.
>
> def visit_BinopNode(self, node):
>     # Very naive, I know things are more complicated...
>     return '%s %s %s' % (self.visit(node.operand1), node.operator, 
> self.visit(node.operand2))
>
> def visit_FuncNode(self, node):
>     # Here we would return a StringIOTree instead (see 
> Cython/StringIOTree.py),
>     # for efficiency.
>
> or, perhaps just:
>
> def visit_BinopNode(self, node):
>     self.visit(node.operand1)
>     self.put(node.operator)
>     self.visit(node.operand2)
>
> So generate_... would never be called, on *any* node, when doing .NET 
> output.
>
> One could deal with shared CPython/.NET code by creating a superclass:
>   
Actually, in the first pass, one could implement DotNetCodeGenerator 
like this:

class DotNetCodeGenerator(CLikeCodeGenerator):
    # Default fallback to CPython generation
    def visit_Node(self, node):
        node.generate_...(self.code)

    def visit_ExprNode(self, node):
        # Slightly different logic I believe...
        node.generate_...(self.code)

    # Any .NET specific stuff
    def visit_FuncNode(self, node): ...

Dag Sverre

> class CLikeCodeGenerator:
>     # pull out stuff that is common between CPython and .NET here
>
> class DotNetCodeGenerator(CLikeCodeGenerator):
>     # .NET-specific nodes handled here
>
> class CPythonCodeGenerator(CLikeCodeGenerator):
>     def visit_Node(self, node):
>         # for now, just call generate_... if the node wasn't handled
>         # by CLikeCodeGenerator
>
>
> This makes it easy to plug in Java as well without making Nodes.py and 
> ExprNodes.py even larger and more tangled and more full of conditionals...
>
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev@...
> http://codespeak.net/mailman/listinfo/cython-dev
>   

Robert Bradshaw | 5 Aug 2010 12:56
Favicon

Re: [Cython] working on Cython for IronPython

On Thu, Aug 5, 2010 at 2:39 AM, Dag Sverre Seljebotn
<dagss@...> wrote:
> Robert Bradshaw wrote:
>> On Thu, Aug 5, 2010 at 12:08 AM, Dag Sverre Seljebotn
>> <dagss@...> wrote:
>>
>>> On 08/05/2010 06:01 AM, Carl Witty wrote:
>>>
>>>> I'm working with Enthought on a branch of Cython to target .NET for
>>>> IronPython.  I'm maintaining a repository at
>>>> http://bitbucket.org/cwitty/cython-for-ironpython (I just started
>>>> writing code, so there's not much interesting there yet).
>>>>
>>>>
>>> Great, I think this is fantastic news.
>>>
>>
>> I'm excited about this too.
>>
>>
>>>> I'm hopeful that eventually this code will be merged back into Cython;
>>>> I don't want to fork Cython.
>>>>
>>>>
>>> I hope this stays in Cython as well.
>>>
>>>> I'm starting by essentially duplicating all of the generate_* methods
>>>> into generate_dotnet_* methods, and modifying the duplicated methods
>>>> to generate C++/CLI code.  My hope is that once I have a working
>>>>
>>>>
>>> Ah, this is very interesting -- when I've spoken with Enthought on the
>>> matter we've largely ignored C++ for this and were talking about a C#
>>> backend.
>>>
>>>> compiler, I can refactor to merge most of the methods back together
>>>> (eliminating the duplicate code) while still not being very intrusive
>>>> to the original codebase.
>>>>
>>>>
>>> Have you considered writing a transform instead of adding more methods
>>> to the nodes? E.g. something like Cython/CodeWriter.py for the .NET
>>> backend? That would be even less intrusive...ideally, one would just
>>> configure the pipeline differently depending on what the backend is.
>>>
>>
>> +1 The transform could go through and replace any nodes that need
>> special casing with special dot_net versions (that only need to know
>> how to generate their code). This would keep things nice and modular.
>> (The real question is how much code could be shared between the
>> CPython bindings and the IronPython ones--the raw C code generation
>> would probably be virtually identical.)
>>
> I was actually thinking that one had the "transform" (or rather "sink")
> just emit/serialize code directly when visiting a node. The "return
> value" could either be string fragments, or if that is too inefficient,
> nothing at all.
>
> def visit_BinopNode(self, node):
>    # Very naive, I know things are more complicated...
>    return '%s %s %s' % (self.visit(node.operand1), node.operator,
> self.visit(node.operand2))
>
> def visit_FuncNode(self, node):
>    # Here we would return a StringIOTree instead (see
> Cython/StringIOTree.py),
>    # for efficiency.
>
> or, perhaps just:
>
> def visit_BinopNode(self, node):
>    self.visit(node.operand1)
>    self.put(node.operator)
>    self.visit(node.operand2)
>
> So generate_... would never be called, on *any* node, when doing .NET
> output.
>
> One could deal with shared CPython/.NET code by creating a superclass:
>
> class CLikeCodeGenerator:
>    # pull out stuff that is common between CPython and .NET here
>
> class DotNetCodeGenerator(CLikeCodeGenerator):
>    # .NET-specific nodes handled here
>
> class CPythonCodeGenerator(CLikeCodeGenerator):
>    def visit_Node(self, node):
>        # for now, just call generate_... if the node wasn't handled
>        # by CLikeCodeGenerator
>
>
> This makes it easy to plug in Java as well without making Nodes.py and
> ExprNodes.py even larger and more tangled and more full of conditionals...

Ah, yes, I remember discussing this before. If that could be pulled
off, it would be nice and clean. The fact that memory management is
handled for you (I think) makes things much simpler.

My one concerns is if (taken to the extreem) having a visitor that
knows how to generate code for every type (and the requisite
internals) is preferable to encapsulating this knowledge on a
node-by-node basis. Perhaps.

- Robert
Josh Allen | 5 Aug 2010 15:07
Picon
Gravatar

Re: [Cython] Package compiling in pure Python mode

Sorry for the delayed response; it seems that my GMail ate your reply.

I was trying to set up the 0.13 beta to install alongside the stable 0.12.1. However, I was not able to get it to work this way, so I decided to go ahead and install the 0.13 beta in place of 0.12.1.

When I did this, I found that this bug seems to have been addressed. For the record, when I switched from .py to .pyx (and added cdefs and cpdefs as appropriate) 0.12.1 still failed with the same error.

Thanks for your help!

Josh

On Wed, Jul 28, 2010 at 1:03 PM, Josh Allen <chemejosh-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
Hello again,

I'm working on a Python/Cython project with a large number of modules, several of which I have Cythonized. I have collected all of the modules into a single package. Attempting to compile these modules has revealed a couple of potential issues, which I will outline below for a small test case.

First, the directory layout for the test case, which contains two modules "foo" and "bar" in a package "bug":

/bug/__init__.py
/bug/bar.pxd
/bug/bar.py
/bug/foo.pxd
/bug/foo.py
/setup.py
/test.py

Next, the contents of the files:

# /bug/foo.py:
class Foo:  
    def hello(self):
        print 'Hello from foo.Foo.hello()'

# /bug/foo.pxd:
cdef class Foo:
    cpdef hello(self)

# /bug/bar.pxd:
from foo cimport Foo
cdef class Bar:
    cdef public Foo foo
    cpdef hello(self)

# /bug/bar.py:
import cython
from foo import Foo
class Bar:  
    def hello(self):
        cython.declare(f=Foo)
        f = Foo()
        print 'Hello from bar.Bar.hello()'

# setup.py
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
setup(
    cmdclass={'build_ext': build_ext},
    ext_modules=[Extension('bug.foo', ['bug/foo.py']), Extension('bug.bar', ['bug/bar.py'])]
)

# test.py
from bug.foo import Foo
from bug.bar import Bar
f = Foo()
f.hello() # Should print 'Hello from Foo.hello()'
b = Bar()
b.hello() # Should print 'Hello from Bar.hello()'

If I try to compile as is -- by running "python setup.py build_ext --inplace" from the top-level directory --, Cython fails at the line "from foo import Foo" in bar.py with the error "Assignment to non-lvalue 'Foo'". If I remove this line, then everything compiles and runs properly. However, I need this line to exist in order to run the file in pure Python mode.

I also tried moving the setup.py file within the bug package, adjusting its content accordingly, and running from within the bug folder. (This is probably not a good location for the setup.py file, but I was trying to see if anything would work.) This also causes Cython to fail, but this time at the line "cython.declare(f=Foo)" in bar.py with the error "Unknown type". However, running cython from the console on bar.py while within the bug directory does produce a valid C file.

Both of these cases are for Python 2.6.5 and Cython 0.12.1 on Ubuntu 10.04. I also attempted to try it for the Cython 0.13 beta, but the problem seems to persist. (Incidentally, is running "cython -V" with the 0.13 beta supposed to print "Cython version 0.12.1"?)

Let me know if there's any other info you need.

<div>
<p>Sorry for the delayed response; it seems that my GMail ate your reply.<br><br>I was trying to set up the 0.13 beta to install alongside the stable 0.12.1. However, I was not able to get it to work this way, so I decided to go ahead and install the 0.13 beta in place of 0.12.1.<br><br>When I did this, I found that this bug seems to have been addressed. For the record, when I switched from .py to .pyx (and added cdefs and cpdefs as appropriate) 0.12.1 still failed with the same error.<br><br>Thanks for your help!<br><br>Josh<br><br></p>
<div class="gmail_quote">On Wed, Jul 28, 2010 at 1:03 PM, Josh Allen <span dir="ltr">&lt;<a href="mailto:chemejosh@...">chemejosh@...</a>&gt;</span> wrote:<br><blockquote class="gmail_quote">
Hello again,<br><br>I'm working on a Python/Cython project with a large number of modules, several of which I have Cythonized. I have collected all of the modules into a single package. Attempting to compile these modules has revealed a couple of potential issues, which I will outline below for a small test case.<br><br>First, the directory layout for the test case, which contains two modules "foo" and "bar" in a package "bug":<br><br>/bug/__init__.py<br>/bug/bar.pxd<br>
/bug/bar.py<br>
/bug/foo.pxd<br>/bug/foo.py<br>/setup.py<br>/test.py<br><br>Next, the contents of the files:<br><br># /bug/foo.py:<br>class Foo:&nbsp;&nbsp; <br>&nbsp;&nbsp;&nbsp; def hello(self):<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; print 'Hello from foo.Foo.hello()'<br><br># /bug/foo.pxd:<br>

cdef class Foo:<br>&nbsp;&nbsp;&nbsp; cpdef hello(self)<br><br># /bug/bar.pxd:<br>from foo cimport Foo<br>cdef class Bar:<br>&nbsp;&nbsp;&nbsp; cdef public Foo foo<br>&nbsp;&nbsp;&nbsp; cpdef hello(self)<br><br># /bug/bar.py:<br>import cython<br>from foo import Foo<br>

class Bar:&nbsp;&nbsp; <br>&nbsp;&nbsp;&nbsp; def hello(self):<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; cython.declare(f=Foo)<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; f = Foo()<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; print 'Hello from bar.Bar.hello()'<br><br># setup.py<br>from distutils.core import setup<br>from distutils.extension import Extension<br>

from Cython.Distutils import build_ext<br>setup(<br>&nbsp;&nbsp;&nbsp; cmdclass={'build_ext': build_ext},<br>&nbsp;&nbsp;&nbsp; ext_modules=[Extension('bug.foo', ['bug/foo.py']), Extension('bug.bar', ['bug/bar.py'])]<br>

)<br><br># test.py<br>from bug.foo import Foo<br>from bug.bar import Bar<br>f = Foo()<br>f.hello() # Should print 'Hello from Foo.hello()'<br>b = Bar()<br>b.hello() # Should print 'Hello from Bar.hello()'<br><br>If I try to compile as is -- by running "python setup.py build_ext --inplace" from the top-level directory --, Cython fails at the line "from foo import Foo" in bar.py with the error "Assignment to non-lvalue 'Foo'". If I remove this line, then everything compiles and runs properly. However, I need this line to exist in order to run the file in pure Python mode.<br><br>I also tried moving the setup.py file within the bug package, adjusting its content accordingly, and running from within the bug folder. (This is probably not a good location for the setup.py file, but I was trying to see if anything would work.) This also causes Cython to fail, but this time at the line "cython.declare(f=Foo)" in bar.py with the error "Unknown type". However, running cython from the console on bar.py while within the bug directory does produce a valid C file.<br><br>Both of these cases are for Python 2.6.5 and Cython 0.12.1 on Ubuntu 10.04. I also attempted to try it for the Cython 0.13 beta, but the problem seems to persist. (Incidentally, is running "cython -V" with the 0.13 beta supposed to print "Cython version 0.12.1"?)<br><br>Let me know if there's any other info you need.<br>
</blockquote>
</div>
<br>
</div>
Carl Witty | 5 Aug 2010 18:58
Picon

Re: [Cython] working on Cython for IronPython

On Thu, Aug 5, 2010 at 12:08 AM, Dag Sverre Seljebotn
<dagss@...> wrote:
> On 08/05/2010 06:01 AM, Carl Witty wrote:
>> I'm starting by essentially duplicating all of the generate_* methods
>> into generate_dotnet_* methods, and modifying the duplicated methods
>> to generate C++/CLI code.  My hope is that once I have a working
>>
> Ah, this is very interesting -- when I've spoken with Enthought on the
> matter we've largely ignored C++ for this and were talking about a C#
> backend.

I convinced the Enthought people that a C++/CLI backend was the best
first step.  For a variety of reasons (mostly the fact that Cython's
"extern from" describes the API, not the ABI, but also catching C++
exceptions and turning them into Python exceptions) a C# backend would
have to generate C or C++ wrappers for every operation on "extern
from" values/types (function call, global variable read, global
variable write, struct member read, struct member write, ...).  On the
other hand, a C++/CLI backend can handle "extern from" in exactly the
same way that the current C/C++ backend does.

Once the C++/CLI backend works, a C# backend is a possible next step.
All of the code generation for C# to interface with IronPython would
be exactly the same as C++/CLI (modulo minor syntactic issues that can
easily be localized).  Even if the ultimate goal is a C# backend, I
think that very little of the work on a C++/CLI backend would be
wasted.

>> compiler, I can refactor to merge most of the methods back together
>> (eliminating the duplicate code) while still not being very intrusive
>> to the original codebase.
>>
>
> Have you considered writing a transform instead of adding more methods
> to the nodes? E.g. something like Cython/CodeWriter.py for the .NET
> backend? That would be even less intrusive...ideally, one would just
> configure the pipeline differently depending on what the backend is.

I did consider this briefly.  Given the choice between writing a
backend basically from scratch, or being able to copy a working
backend and incrementally modify it, the latter seemed a lot easier.
:)

Once I have a working compiler, and it's time to clean up the code for
submission, I'll definitely consider this approach.  Based on the tiny
amount of work I've done so far, though, I suspect that I may be able
to share a lot of code with the existing backend that would end up
duplicated if I made my backend into a transform.

> (I once suggested that the CPython generation was also moved into a
> seperate generator class like this as well; of course, that's a lot of
> unnecessary work)

Sounds like a good idea to me.  Also, if the existing backend and the
C++/CLI backend were both transforms, that would make it a lot easier
for them to share code (they could both inherit from a generic "code
generation" transform, for instance).  But I doubt if I'll have time
to work on this...

> I'm just not too sure about having "if dotnet:" all over the place
> (eventually). Of course, you should just do whatever gives results
> quickest at this point (and I very much trust your judgement), I'm just
> making sure the option is mentioned.

Keeping the number of "if dotnet:" down is definitely an important
long-term goal.  (For now, I'm adding lots of them, but I expect to
remove them and handle the cases some other way before I submit the
code.)

> Dag Sverre

Carl
Carl Witty | 5 Aug 2010 19:13
Picon

Re: [Cython] working on Cython for IronPython

On Thu, Aug 5, 2010 at 1:27 AM, Robert Bradshaw
<robertwb@...> wrote:
> +1 The transform could go through and replace any nodes that need
> special casing with special dot_net versions (that only need to know
> how to generate their code). This would keep things nice and modular.
> (The real question is how much code could be shared between the
> CPython bindings and the IronPython ones--the raw C code generation
> would probably be virtually identical.)

I do like this idea.

My tentative plan for merging the existing backend and the C++/CLI
backend back together is to move most of the C code fragments from
Nodes.py/ExprNodes.py into more methods on Code.py's CCodeWriter.
Then I would split the generic parts of CCodeWriter into a new base
class BaseCodeWriter, and add a new subclass CPPCLICodeWriter.  Then
(hopefully) a single conditional in ModuleNode.generate_c_code, where
it creates the rootwriter object, could handle most of the differences
between the backends.  For nodes where the code generation difference
is too large to be reasonably hidden in Code.py, your transform sounds
like a reasonable way to avoid "if dotnet:".

> On this note, it would be cool if, eventually, Jython could be
> supported through similar mechanisms.

That would be very cool.  (I don't know anything about Jython
internals or the JVM foreign function interface, so I don't know how
hard it would be.)

> - Robert

Carl

Gmane