Nathan Goldbaum | 21 Oct 17:52 2015
Picon

[Cython] 'Class' redeclared error if C++ class & file name match those of the wrapper class

Hi all,

I'm forwarding this at the request of David Nemesky ("savior" on Freenode), who I've been chatting with on the cython IRC channel. He was having trouble signing up to the cython-devel mailing list, so I've gone ahead and sent his bug report in for him. His report follows below.

Apparently this error happens under very specific circumstances, but I managed to run into them. When wrapping a C++ class, Cython reports a "'ClassName' redeclared" error, if:
- the name of the .pyx is the same as the name of the .pxd
- the name of the python class is the same as the name of the C++ class
 
Mismatch in either the file- or the class name results in a successful compilation.
 
I have modified my fork of the cython-cpp-test repo to make the issue easily reproducible. In https://github.com/DavidNemeskey/cython-cpp-test/, try `python setup.py build_ext --inplace` (do not forget to delete all .cpp and .so files between calls) in the following branches:
- cython_bug (diff file, diff class) -> OK
- diff_file_same_class -> OK
- same_file_diff_class -> OK
- same_file_same_class -> ERROR (https://bpaste.net/show/2c1af5fa5ee2)
 
Since Cython compiles xxx.pyx to xxx.cpp in this case, in order to avoid problems with testclass.cpp being overwritten, I implemented the class in testclass.hpp (in the original repo, there was a .h and a .cpp file).
However, implementing everything in the header is not a pre-requisite of the bug, as is evident from branch same_file_same_class_diff_c.
<div><div dir="ltr">
<div>Hi all,</div>
<div><br></div>
<div>I'm forwarding this at the request of David Nemesky ("savior" on Freenode), who I've been chatting with on the cython IRC channel. He was having trouble signing up to the cython-devel mailing list, so I've gone ahead and sent his bug report in for him. His report follows below.</div>
<div><br></div>
<div>Apparently this error happens under very specific circumstances, but I managed to run into them. When wrapping a C++ class, Cython reports a "'ClassName' redeclared" error, if:</div>
<div>- the name of the .pyx is the same as the name of the .pxd</div>
<div>- the name of the python class is the same as the name of the C++ class</div>
<div>&nbsp;</div>
<div>Mismatch in either the file- or the class name results in a successful compilation.</div>
<div>&nbsp;</div>
<div>I have modified my fork of the cython-cpp-test repo to make the issue easily reproducible. In <a href="https://github.com/DavidNemeskey/cython-cpp-test/">https://github.com/DavidNemeskey/cython-cpp-test/</a>, try `python setup.py build_ext --inplace` (do not forget to delete all .cpp and .so files between calls) in the following branches:</div>
<div>- cython_bug (diff file, diff class) -&gt; OK</div>
<div>- diff_file_same_class -&gt; OK</div>
<div>- same_file_diff_class -&gt; OK</div>
<div>- same_file_same_class -&gt; ERROR (<a href="https://bpaste.net/show/2c1af5fa5ee2">https://bpaste.net/show/2c1af5fa5ee2</a>)</div>
<div>&nbsp;</div>
<div>Since Cython compiles xxx.pyx to xxx.cpp in this case, in order to avoid problems with testclass.cpp being overwritten, I implemented the class in testclass.hpp (in the original repo, there was a .h and a .cpp file).</div>
<div>However, implementing everything in the header is not a pre-requisite of the bug, as is evident from branch same_file_same_class_diff_c.</div>
</div></div>
Carlos Pita | 18 Oct 19:45 2015
Picon
Gravatar

[Cython] [Refactor] Emulated types

Hi,

I'm in the process of refactoring the emulated types in Shadow, in order to simplify a bit that part of the module. Ultimately I want to add support for declaring typed memory views in pure python (currently it's only possible to declare them as strings, Shadow does implement the slicing part -tho it's messy here- but the parser is unable to get them right from decorators). But before that I'd like to simplify things a bit, as I've said, and before that I need to get a good grasp of what I'm trying to simplify, so I'd better hear your opinion about the following points first:

1) Shadow allows creating array types like this: cython.array(cython.int, 10) but the compiler will complain that cython.array is an unknown type.

2) Furthermore Shadow allows to instantiate every type: cython.int(10), cython.array(cython.int, 20)(), etc. The compiler rejects all these constructs, so I see little point in supporting this feature.

I think that disallowing these features (AFAICS completely unsupported by the compiler) will simplify the code a lot and will allow to merge the "shameless copy" section in a more coherent fashion. Is there any compelling reason to keep them? What is/was their rationale?

Cheers
--
Carlos
<div><div dir="ltr">
<div>
<div>
<div>
<div>Hi,<br><br>
</div>I'm in the process of refactoring the emulated types in Shadow, in order to simplify a bit that part of the module. Ultimately I want to add support for declaring typed memory views in pure python (currently it's only possible to declare them as strings, Shadow does implement the slicing part -tho it's messy here- but the parser is unable to get them right from decorators). But before that I'd like to simplify things a bit, as I've said, and before that I need to get a good grasp of what I'm trying to simplify, so I'd better hear your opinion about the following points first:<br><br>
</div>1) Shadow allows creating array types like this: <span tabindex="-1" class="">cython</span>.array(<span tabindex="-1" class="">cython</span>.int, 10) but the compiler will complain that <span tabindex="-1" class="">cython</span>.array is an unknown type.<br><br>
</div>2) Furthermore Shadow allows to instantiate every type: <span tabindex="-1" class="">cython</span>.int(10), <span tabindex="-1" class="">cython</span>.array(<span tabindex="-1" class="">cython</span>.int, 20)(), etc. The compiler rejects all these constructs, so I see little point in supporting this feature.<br><br>
</div>
<div>I think that disallowing these features (<span tabindex="-1" class="">AFAICS</span> completely unsupported by the compiler) will simplify the code a lot and will allow to merge the "shameless copy" section in a more coherent fashion. Is there any compelling reason to keep them? What is/was their rationale?<br>
</div>
<div><br></div>
<div>Cheers<br>--<br>
</div>
<div>Carlos</div>
</div></div>
Picon

[Cython] Bug in list comprehension

Hello,

I've think I've found a bug in Cython (or is this just a funny feature?). The error occurs when the variable name used as source for the iteration is also used as a value.

Given the files test1.pyx (cython) and test2.py (python):

test1.pyx:
def test():
    x = range(10)
    return [x for x in x]

test2.py:
def test():
    x = range(10)
    return [x for x in x]

The results in running them are:
>>> import test1, test2
>>> test1.test()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "test1.pyx", line 4, in test1.test (/Users/leandro/.pyxbld/temp.macosx-10.6-intel-3.4/pyrex/test1.c:666)
    return [x for x in x]
UnboundLocalError: local variable 'x' referenced before assignment
>>> test2.test()
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>>

A regular for loop will behave correctly:

code:
def test():
    x = range(10)
    ret = []
    for x in x:
        ret.append(x)
    return ret

result:
>>> import test1, test2
>>> test1.test()
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> test2.test()
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>>

Cheers, Leandro
<div><div dir="ltr">Hello,<div><br></div>
<div>I've think I've found a bug in Cython (or is this just a funny feature?). The error occurs when the variable name used as source for the iteration is also used as a value.</div>
<div><br></div>
<div>Given the files test1.pyx (cython) and test2.py (python):</div>
<div><br></div>
<div>test1.pyx:</div>
<div>
<div>def test():</div>
<div>&nbsp; &nbsp; x = range(10)</div>
<div>&nbsp; &nbsp; return [x for x in x]</div>
</div>
<div><br></div>
<div>test2.py:</div>
<div>
<div>def test():</div>
<div>&nbsp; &nbsp; x = range(10)</div>
<div>&nbsp; &nbsp; return [x for x in x]</div>
</div>
<div><br></div>
<div>The results in running them are:</div>
<div>
<div>&gt;&gt;&gt; import test1, test2</div>
<div>&gt;&gt;&gt; test1.test()</div>
<div>Traceback (most recent call last):</div>
<div>&nbsp; File "&lt;stdin&gt;", line 1, in &lt;module&gt;</div>
<div>&nbsp; File "test1.pyx", line 4, in test1.test (/Users/leandro/.pyxbld/temp.macosx-10.6-intel-3.4/pyrex/test1.c:666)</div>
<div>&nbsp; &nbsp; return [x for x in x]</div>
<div>UnboundLocalError: local variable 'x' referenced before assignment</div>
<div>&gt;&gt;&gt; test2.test()</div>
<div>[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]</div>
<div>&gt;&gt;&gt;</div>
</div>
<div><br></div>
<div>A regular for loop will behave correctly:</div>
<div><br></div>
<div>code:</div>
<div>
<div>def test():</div>
<div>&nbsp; &nbsp; x = range(10)</div>
<div>&nbsp; &nbsp; ret = []</div>
<div>&nbsp; &nbsp; for x in x:</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; ret.append(x)</div>
<div>&nbsp; &nbsp; return ret</div>
</div>
<div><br></div>
<div>result:</div>
<div>
<div>&gt;&gt;&gt; import test1, test2</div>
<div>&gt;&gt;&gt; test1.test()</div>
<div>[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]</div>
<div>&gt;&gt;&gt; test2.test()</div>
<div>[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]</div>
<div>&gt;&gt;&gt;</div>
</div>
<div><br></div>
<div>Cheers, Leandro</div>
</div></div>
Kevin Thornton | 16 Oct 21:50 2015
Picon

[Cython] Exposing C++'s <iterator>

Hello,

I was wanting to use C++s' back_inserter in Cython, and I got it working.  I'm writing the list to ask about what more I should do before submitting a pull request.

In short, the following definitions were sufficient:

##Define the basic iterator types (this is incomplete)
cdef extern from "<iterator>" namespace "std":
    cdef cppclass iterator[Category,T,Distance,Pointer,Reference]:
        pass
    cdef cppclass output_iterator_tag:
        pass
    #The goal here is to get back_inserter working, as it is handy.
    cdef cppclass back_insert_iterator[T](iterator[output_iterator_tag,void,void,void,void]):
        pass
    back_insert_iterator[CONTAINER] back_inserter[CONTAINER](CONTAINER &)

##Not shown here, but I've gotten C++11's move algorithm to work, too,
##using same kind of prototype
cdef extern from "<algorithm>" namespace "std":
    OUTPUT copy[INPUT,OUTPUT](INPUT,INPUT,OUTPUT)

I've worked up a tiny test package that is on GitHub: https://github.com/molpopgen/cython_cpp_iterators

The tests work, and I can indeed copy amongst C++ containers.  

I've also gotten the C++ move algorithm to work.  The prototype is the same as for 'copy' above.

I'd be willing to define the remaining classes from <iterator> and add a .pxd file.

Any input would be appreciated.

Thanks,

Kevin


<div><div dir="ltr">Hello,<div><br></div>
<div>I was wanting to use C++s' back_inserter in Cython, and I got it working.&nbsp; I'm writing the list to ask about what more I should do before submitting a pull request.</div>
<div><br></div>
<div>In short, the following definitions were sufficient:</div>
<div><br></div>
<div>
<div>##Define the basic iterator types (this is incomplete)</div>
<div>cdef extern from "&lt;iterator&gt;" namespace "std":</div>
<div>&nbsp; &nbsp; cdef cppclass iterator[Category,T,Distance,Pointer,Reference]:</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; pass</div>
<div>&nbsp; &nbsp; cdef cppclass output_iterator_tag:</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; pass</div>
<div>&nbsp; &nbsp; #The goal here is to get back_inserter working, as it is handy.</div>
<div>&nbsp; &nbsp; cdef cppclass back_insert_iterator[T](iterator[output_iterator_tag,void,void,void,void]):</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; pass</div>
<div>&nbsp; &nbsp; back_insert_iterator[CONTAINER] back_inserter[CONTAINER](CONTAINER &amp;)</div>
<div><br></div>
<div>##Not shown here, but I've gotten C++11's move algorithm to work, too,</div>
<div>##using same kind of prototype</div>
<div>cdef extern from "&lt;algorithm&gt;" namespace "std":</div>
<div>&nbsp; &nbsp; OUTPUT copy[INPUT,OUTPUT](INPUT,INPUT,OUTPUT)</div>
</div>
<div><br></div>
<div>I've worked up a tiny test package that is on GitHub:&nbsp;<a href="https://github.com/molpopgen/cython_cpp_iterators">https://github.com/molpopgen/cython_cpp_iterators</a>
</div>
<div><br></div>
<div>The tests work, and I can indeed copy amongst C++ containers. &nbsp;</div>
<div><br></div>
<div>I've also gotten the C++ move algorithm to work.&nbsp; The prototype is the same as for 'copy' above.</div>
<div><br></div>
<div>I'd be willing to define the remaining classes from &lt;iterator&gt; and add a .pxd file.</div>
<div><br></div>
<div>Any input would be appreciated.</div>
<div><br></div>
<div>Thanks,</div>
<div><br></div>
<div>Kevin</div>
<div><br></div>
<div><br></div>
</div></div>
Carlos Pita | 16 Oct 21:05 2015
Picon
Gravatar

[Cython] [Bug] Unable to cast to python type.

Hi,

import cython as cy
y = cy.cast('list', x)

fails to compile with "AttributeError: 'TypecastNode' object has no attribute 'typecheck'".

But the following examples do compile:

y = <list> x

y = cy.cast('int', x)

Cheers
--
Carlos

<div><div dir="ltr">
<div>
<div>
<div>
<div>
<div>
<div>Hi,<br>
</div>
<div>
<br>import cython as cy<br>y = cy.cast('list', x)<br><br>
</div>fails to compile with "AttributeError: 'TypecastNode' object has no attribute 'typecheck'".<br><br>
</div>But the following examples do compile:<br><br>
</div>y = &lt;list&gt; x<br><br>
</div>y = cy.cast('int', x)<br><br>
</div>Cheers<br>--<br>
</div>Carlos<br><div><div><div><br></div></div></div>
</div></div>
Carlos Pita | 16 Oct 20:53 2015
Picon
Gravatar

[Cython] [Bug] Generator comprehension not compiling

Hi,

in cython 0.24.0a0 this:

----------
import cython as cy

<at> cy.cfunc
def f(cats):
    ''.join(c for c in cats)
----------

fails to compile with: Compiler crash in AnalyseDeclarationsTransform.

Nevertheless the following close variations do compile:

----------
import cython as cy

<at> cy.cfunc
def f(cats):
    ''.join([c for c in cats])
----------

----------
cdef f(cats):
    ''.join(c for c in cats)
----------

Cheers
--
Carlos


<div><div dir="ltr">
<div>
<div>
<div>
<div>
<div>Hi,<br><br>
</div>in cython 0.24.0a0 this:<br><br>----------<br>import cython as cy<br><br> <at> cy.cfunc<br>def f(cats):<br>&nbsp;&nbsp;&nbsp; ''.join(c for c in cats)<br>----------<br><br>
</div>fails to compile with: Compiler crash in AnalyseDeclarationsTransform.<br><br>
</div>Nevertheless the following close variations do compile:<br><br>----------<br>import cython as cy<br><br> <at> cy.cfunc<br>def f(cats):<br>&nbsp;&nbsp;&nbsp; ''.join([c for c in cats])<br>----------<br><br>----------<br>cdef f(cats):<br>&nbsp;&nbsp;&nbsp; ''.join(c for c in cats)<br>----------<br><br>
</div>Cheers<br>--<br>
</div>Carlos<br><div><div>
<br><div><br></div>
</div></div>
</div></div>
Stefan Behnel | 11 Oct 16:30 2015
Picon

[Cython] can we deprecate for-from loops?

Hi!

The syntax construct "for i from 0 <= i < 10" has been silently outdated
for years. Can we start issuing a warning that normal range() loops are
preferred?

While I don't see it going to go away any time soon, I'd like to make users
aware that it's an unnecessary syntax wart that's best avoided and that
should really not be used in new code.

Stefan

[Cython] git master fails to compile scipy

hi,
since ab78f93b3ffa88183a0d2aae6b692e394c51f860 scipy does not build anymore:
the failing file is:
https://github.com/scipy/scipy/blob/master/scipy/sparse/csgraph/_reordering.pyx

error:
Traceback (most recent call last):
  File "/usr/bin/cython", line 8, in <module>
    main(command_line = 1)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Main.py",
line 704, in main
    result = compile(sources, options)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Main.py",
line 679, in compile
    return compile_multiple(source, options)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Main.py",
line 657, in compile_multiple
    result = run_pipeline(source, options, context=context)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Main.py",
line 487, in run_pipeline
    err, enddata = Pipeline.run_pipeline(pipeline, source)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Pipeline.py",
line 365, in run_pipeline
    data = phase(data)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Pipeline.py",
line 53, in generate_pyx_code_stage
    module_node.process_implementation(options, result)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/ModuleNode.py",
line 118, in process_implementation
    self.generate_c_code(env, options, result)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/ModuleNode.py",
line 349, in generate_c_code
    self.body.generate_function_definitions(env, code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
line 436, in generate_function_definitions
    stat.generate_function_definitions(env, code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
line 436, in generate_function_definitions
    stat.generate_function_definitions(env, code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/FusedNode.py",
line 785, in generate_function_definitions
    stat.generate_function_definitions(env, code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
line 3068, in generate_function_definitions
    FuncDefNode.generate_function_definitions(self, env, code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
line 1935, in generate_function_definitions
    self.generate_function_body(env, code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
line 1691, in generate_function_body
    self.body.generate_execution_code(code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
line 442, in generate_execution_code
    stat.generate_execution_code(code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/UtilNodes.py",
line 321, in generate_execution_code
    self.body.generate_execution_code(code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
line 6419, in generate_execution_code
    self.body.generate_execution_code(code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
line 442, in generate_execution_code
    stat.generate_execution_code(code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
line 5865, in generate_execution_code
    if_clause.generate_execution_code(code, end_label, is_last=i == last)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
line 5908, in generate_execution_code
    self.body.generate_execution_code(code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
line 442, in generate_execution_code
    stat.generate_execution_code(code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
line 4800, in generate_execution_code
    self.generate_assignment_code(code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
line 5091, in generate_assignment_code
    self.lhs.generate_assignment_code(self.rhs, code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/ExprNodes.py",
line 3897, in generate_assignment_code
    self.generate_buffer_setitem_code(rhs, code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/ExprNodes.py",
line 3905, in generate_buffer_setitem_code
    buffer_entry, ptrexpr = self.buffer_lookup_code(code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/ExprNodes.py",
line 3873, in buffer_lookup_code
    for ivar in self.indices]
AttributeError: 'NoneType' object has no attribute 'signed'
Jason Madden | 9 Oct 18:04 2015
Gravatar

[Cython] Bug - PyPy tuple leak in __Pyx_PyObject_CallOneArg?

Hello,

I'm one of the maintainers of the gevent concurrency library, which has recently been ported to run on PyPy.
Under PyPy, a small portion of the code is compiled with Cython in order to get the desired atomic semantics
(specifically, a semaphore class). We recently had a user report an easily reproducible leak of tuples of
one element. Tracking it down, it appears that __Pyx_PyObject_CallOneArg creates a new tuple under
PyPy, but neglects to free it. This was tested with Cython 0.23.3 and PyPy 2.6.1.

Our Cython code contained a loop like this, and every iteration of the loop leaked a tuple:

    for link in links:
        link(self)

The C output for that last line looked like this:

    __Pyx_PyObject_CallOneArg(__pyx_t_10, ((PyObject *)__pyx_v_self));...

And, under PyPy, the implementation of __Pyx_PyObject_CallOneArg is different than it is under CPython.
Specifically, with Cython 0.23.3 this is the code:

    #if CYTHON_COMPILING_IN_CPYTHON
    ...
    #else
    static CYTHON_INLINE PyObject* __Pyx_PyObject_CallOneArg(PyObject *func, PyObject *arg) {
        PyObject* args = PyTuple_Pack(1, arg);
        return (likely(args)) ? __Pyx_PyObject_Call(func, args, NULL) : NULL;
    }
    #endif

PyTuple_Pack is documented as returning a new reference
(https://docs.python.org/2/c-api/tuple.html#c.PyTuple_Pack), so it seems to me like this code
should be decrementing the reference to the tuple before returning (much like the
__Pyx__PyObject_CallOneArg function does under CPython).

Changing the loop to avoid the use of CallOneArg seemed to resolve the tuple leak, presumably at some
execution time cost:

   args = (self,)
   for link in links:
       link(*self)

There's some additional background at https://bitbucket.org/pypy/pypy/issues/2149/memory-leak-for-python-subclass-of-cpyext#comment-22347393

Am I interpreting this correctly to be a bug, or could there be something wrong in the way we're handling
callbacks? Please let me know if there's any further information I can provide.

Thanks,
Jason
Stephen LARROQUE | 8 Oct 16:35 2015
Picon

Re: [Cython] cython-devel Digest, Vol 57, Issue 5

Hello Masood,

I can only answer the second issue. You need to explicitly include the *.pyx files in your MANIFEST.in, because else this .pyx files aren't automatically managed by the distutils package so it doesn't know it has to add the .pyx files.

I have recently packaged two libraries, with different hierarchy layouts, so you can maybe use one of them as an example to fix your packaging:

First project with a usual package structure:

Second project with a flatout structure:

The commands I use to locally build the package are:

python setup.py sdist --formats=gztar,zip bdist_wininst --plat-name=win32
python setup.py sdist bdist_egg bdist_wheel --plat-name=win32

If your packaging config is correct, you should see the .pyx files in those builds.

And to build all other formats and upload to pypi, I use the python library Twine:

twine upload dist/*

Hope this can help you.

Best regards,
Stephen Larroque


2015-10-08 12:00 GMT+02:00 <cython-devel-request-+ZN9ApsXKcEdnm+yROfE0A@public.gmane.org>:
Send cython-devel mailing list submissions to
        cython-devel-+ZN9ApsXKcEdnm+yROfE0A@public.gmane.org

To subscribe or unsubscribe via the World Wide Web, visit
        https://mail.python.org/mailman/listinfo/cython-devel
or, via email, send a message with subject or body 'help' to
        cython-devel-request-+ZN9ApsXKcEdnm+yROfE0A@public.gmane.org

You can reach the person managing the list at
        cython-devel-owner-+ZN9ApsXKcEdnm+yROfE0A@public.gmane.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of cython-devel digest..."


Today's Topics:

   1. Cython coverage and multiple projects (Masood Malekghassemi)


----------------------------------------------------------------------

Message: 1
Date: Wed, 7 Oct 2015 16:06:01 -0700
From: Masood Malekghassemi <atash <at> google.com>
To: cython-devel-+ZN9ApsXKcEdnm+yROfE0A@public.gmane.org
Subject: [Cython] Cython coverage and multiple projects
Message-ID:
        <CALweS8f=xeQfhmydXR5C70mbO1=1hibmsTgBPuvCuiUEqE=g8w-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
Content-Type: text/plain; charset="utf-8"

I'd asked this of Robert Bradshaw earlier and was pointed in this direction
(specifically towards Stefan), so here goes!

I'm having trouble with getting Cython to play nicely with the coverage.py
tool via the Cython.Coverage plug-in.

We have tests that are running from a different setup.py project than the
one which contains the Cython files (the projects respectively being for
the duration of this discussion the 'tests' project and the 'source'
project). The generated coverage file contains the line data for individual
.pyx files, but the paths of the .pyx files themselves *within* the
.coverage file appear to be rooted in the tests project rather than the
site-packages folder in which the source project's files were installed
(i.e. paths that never exist throughout the edit-build-test cycle).

After editing the .coverage file manually to point to the source project
directory, coverage.py reports that the .pyx files are not python files and
errors out... but the .coveragerc file contains the plug-in module via
`[run]\n plugins = Cython.Coverage`, so I'd have expected the Cython
coverage.py plug-in to catch those files and handle them instead of letting
the rest of coverage.py get confused over them.

Furthermore, this is against the 4.0 coverage.py release rather than the
4.0b2 coverage.py release (which appears to be the version against which
the latest edit to the Cython code base was made). I don't see any API
differences between the two in the plug-ins, but I might have missed
something so maybe that's relevant.

A secondary issue: the .pyx files are never copied over to the installation
directory, so even if the paths were 'correct' (with respect to being
copied into the site-packages folder or being accessible via developer mode
pip-installs), they wouldn't be found. This seems like it wouldn't
necessarily be an issue from a cursory glance at the source code as the
file tracer in the Cython coverage.py plug-in appears to spit out blank
lines in such a scenario.

Complicating matters further, we're running the tests through pytest. That
said, I've given up on trying to get pytest-cov to play nicely with this
until I can get a manual invocation of `coverage report ...` to produce
output describing the .pyx files.

Thanks for reading this far. Got any pointers?


----------------------------------------------------------------
Context: https://github.com/soltanmm/grpc/tree/cy-wip/src/python
Basic steps to run the tests:

   - cd $GRPC_REPOSITORY_ROOT
   - make
   - CONFIG=opt ./tools/run_tests/build_python.sh 2.7
   - CONFIG=opt PYVER=2.7 ./run_tests/run_python.sh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20151007/11793594/attachment-0001.html>

------------------------------

Subject: Digest Footer

_______________________________________________
cython-devel mailing list
cython-devel-+ZN9ApsXKcEdnm+yROfE0A@public.gmane.org
https://mail.python.org/mailman/listinfo/cython-devel


------------------------------

End of cython-devel Digest, Vol 57, Issue 5
*******************************************

<div>
<div dir="ltr">Hello Masood,<div><br></div>
<div>I can only answer the second issue. You need to explicitly include the *.pyx files in your MANIFEST.in, because else this .pyx files aren't automatically managed by the distutils package so it doesn't know it has to add the .pyx files.</div>
<div><br></div>
<div>I have recently packaged two libraries, with different hierarchy layouts, so you can maybe use one of them as an example to fix your packaging:</div>
<div><br></div>
<div>First project with a usual package structure:</div>
<div>
<a href="https://github.com/lrq3000/unireedsolomon/blob/master/MANIFEST.in">https://github.com/lrq3000/unireedsolomon/blob/master/MANIFEST.in</a><br>
</div>
<div><a href="https://github.com/lrq3000/unireedsolomon/blob/master/setup.py">https://github.com/lrq3000/unireedsolomon/blob/master/setup.py</a></div>
<div><br></div>
<div>Second project with a flatout structure:</div>
<div>
<a href="https://github.com/lrq3000/reedsolomon/blob/master/MANIFEST.in">https://github.com/lrq3000/reedsolomon/blob/master/MANIFEST.in</a><br>
</div>
<div><a href="https://github.com/lrq3000/reedsolomon/blob/master/setup.py">https://github.com/lrq3000/reedsolomon/blob/master/setup.py</a></div>
<div><br></div>
<div>The commands I use to locally build the package are:</div>
<div><br></div>
<div>
<div>python setup.py sdist --formats=gztar,zip bdist_wininst --plat-name=win32</div>
<div>python setup.py sdist bdist_egg bdist_wheel --plat-name=win32</div>
</div>
<div><br></div>
<div>If your packaging config is correct, you should see the .pyx files in those builds.</div>
<div><br></div>
<div>And to build all other formats and upload to pypi, I use the python library Twine:</div>
<div><br></div>
<div>twine upload dist/*<br>
</div>
<div><br></div>
<div>Hope this can help you.</div>
<div><br></div>
<div>Best regards,</div>
<div>Stephen Larroque</div>
<div><br></div>
</div>
<div class="gmail_extra">
<br><div class="gmail_quote">2015-10-08 12:00 GMT+02:00  <span dir="ltr">&lt;<a href="mailto:cython-devel-request@..." target="_blank">cython-devel-request@...</a>&gt;</span>:<br><blockquote class="gmail_quote">Send cython-devel mailing list submissions to<br>
&nbsp; &nbsp; &nbsp; &nbsp; <a href="mailto:cython-devel@...">cython-devel@...</a><br><br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
&nbsp; &nbsp; &nbsp; &nbsp; <a href="https://mail.python.org/mailman/listinfo/cython-devel" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/cython-devel</a><br>
or, via email, send a message with subject or body 'help' to<br>
&nbsp; &nbsp; &nbsp; &nbsp; <a href="mailto:cython-devel-request <at> python.org">cython-devel-request@...</a><br><br>
You can reach the person managing the list at<br>
&nbsp; &nbsp; &nbsp; &nbsp; <a href="mailto:cython-devel-owner@...">cython-devel-owner@...</a><br><br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of cython-devel digest..."<br><br><br>
Today's Topics:<br><br>
&nbsp; &nbsp;1. Cython coverage and multiple projects (Masood Malekghassemi)<br><br><br>
----------------------------------------------------------------------<br><br>
Message: 1<br>
Date: Wed, 7 Oct 2015 16:06:01 -0700<br>
From: Masood Malekghassemi &lt;<a href="mailto:atash@...">atash <at> google.com</a>&gt;<br>
To: <a href="mailto:cython-devel@...">cython-devel@...</a><br>
Subject: [Cython] Cython coverage and multiple projects<br>
Message-ID:<br>
&nbsp; &nbsp; &nbsp; &nbsp; &lt;CALweS8f=xeQfhmydXR5C70mbO1=1hibmsTgBPuvCuiUEqE=<a href="mailto:g8w@...">g8w@...</a>&gt;<br>
Content-Type: text/plain; charset="utf-8"<br><br>
I'd asked this of Robert Bradshaw earlier and was pointed in this direction<br>
(specifically towards Stefan), so here goes!<br><br>
I'm having trouble with getting Cython to play nicely with the coverage.py<br>
tool via the Cython.Coverage plug-in.<br><br>
We have tests that are running from a different setup.py project than the<br>
one which contains the Cython files (the projects respectively being for<br>
the duration of this discussion the 'tests' project and the 'source'<br>
project). The generated coverage file contains the line data for individual<br>
.pyx files, but the paths of the .pyx files themselves *within* the<br>
.coverage file appear to be rooted in the tests project rather than the<br>
site-packages folder in which the source project's files were installed<br>
(i.e. paths that never exist throughout the edit-build-test cycle).<br><br>
After editing the .coverage file manually to point to the source project<br>
directory, coverage.py reports that the .pyx files are not python files and<br>
errors out... but the .coveragerc file contains the plug-in module via<br>
`[run]\n plugins = Cython.Coverage`, so I'd have expected the Cython<br>
coverage.py plug-in to catch those files and handle them instead of letting<br>
the rest of coverage.py get confused over them.<br><br>
Furthermore, this is against the 4.0 coverage.py release rather than the<br>
4.0b2 coverage.py release (which appears to be the version against which<br>
the latest edit to the Cython code base was made). I don't see any API<br>
differences between the two in the plug-ins, but I might have missed<br>
something so maybe that's relevant.<br><br>
A secondary issue: the .pyx files are never copied over to the installation<br>
directory, so even if the paths were 'correct' (with respect to being<br>
copied into the site-packages folder or being accessible via developer mode<br>
pip-installs), they wouldn't be found. This seems like it wouldn't<br>
necessarily be an issue from a cursory glance at the source code as the<br>
file tracer in the Cython coverage.py plug-in appears to spit out blank<br>
lines in such a scenario.<br><br>
Complicating matters further, we're running the tests through pytest. That<br>
said, I've given up on trying to get pytest-cov to play nicely with this<br>
until I can get a manual invocation of `coverage report ...` to produce<br>
output describing the .pyx files.<br><br>
Thanks for reading this far. Got any pointers?<br><br><br>
----------------------------------------------------------------<br>
Context: <a href="https://github.com/soltanmm/grpc/tree/cy-wip/src/python" rel="noreferrer" target="_blank">https://github.com/soltanmm/grpc/tree/cy-wip/src/python</a><br>
Basic steps to run the tests:<br><br>
&nbsp; &nbsp;- cd $GRPC_REPOSITORY_ROOT<br>
&nbsp; &nbsp;- make<br>
&nbsp; &nbsp;- CONFIG=opt ./tools/run_tests/build_python.sh 2.7<br>
&nbsp; &nbsp;- CONFIG=opt PYVER=2.7 ./run_tests/run_python.sh<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: &lt;<a href="http://mail.python.org/pipermail/cython-devel/attachments/20151007/11793594/attachment-0001.html" rel="noreferrer" target="_blank">http://mail.python.org/pipermail/cython-devel/attachments/20151007/11793594/attachment-0001.html</a>&gt;<br><br>
------------------------------<br><br>
Subject: Digest Footer<br><br>
_______________________________________________<br>
cython-devel mailing list<br><a href="mailto:cython-devel@...">cython-devel@...</a><br><a href="https://mail.python.org/mailman/listinfo/cython-devel" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/cython-devel</a><br><br><br>
------------------------------<br><br>
End of cython-devel Digest, Vol 57, Issue 5<br>
*******************************************<br>
</blockquote>
</div>
<br>
</div>
</div>

[Cython] Cython coverage and multiple projects

I'd asked this of Robert Bradshaw earlier and was pointed in this direction (specifically towards Stefan), so here goes!

I'm having trouble with getting Cython to play nicely with the coverage.py tool via the Cython.Coverage plug-in.

We have tests that are running from a different setup.py project than the one which contains the Cython files (the projects respectively being for the duration of this discussion the 'tests' project and the 'source' project). The generated coverage file contains the line data for individual .pyx files, but the paths of the .pyx files themselves within the .coverage file appear to be rooted in the tests project rather than the site-packages folder in which the source project's files were installed (i.e. paths that never exist throughout the edit-build-test cycle).

After editing the .coverage file manually to point to the source project directory, coverage.py reports that the .pyx files are not python files and errors out... but the .coveragerc file contains the plug-in module via `[run]\n plugins = Cython.Coverage`, so I'd have expected the Cython coverage.py plug-in to catch those files and handle them instead of letting the rest of coverage.py get confused over them.

Furthermore, this is against the 4.0 coverage.py release rather than the 4.0b2 coverage.py release (which appears to be the version against which the latest edit to the Cython code base was made). I don't see any API differences between the two in the plug-ins, but I might have missed something so maybe that's relevant.

A secondary issue: the .pyx files are never copied over to the installation directory, so even if the paths were 'correct' (with respect to being copied into the site-packages folder or being accessible via developer mode pip-installs), they wouldn't be found. This seems like it wouldn't necessarily be an issue from a cursory glance at the source code as the file tracer in the Cython coverage.py plug-in appears to spit out blank lines in such a scenario.

Complicating matters further, we're running the tests through pytest. That said, I've given up on trying to get pytest-cov to play nicely with this until I can get a manual invocation of `coverage report ...` to produce output describing the .pyx files.

Thanks for reading this far. Got any pointers?


----------------------------------------------------------------
Basic steps to run the tests:
  • cd $GRPC_REPOSITORY_ROOT
  • make
  • CONFIG=opt ./tools/run_tests/build_python.sh 2.7
  • CONFIG=opt PYVER=2.7 ./run_tests/run_python.sh
<div><div dir="ltr">
<div><span>I'd asked this of Robert Bradshaw earlier and was pointed in this direction (specifically towards Stefan), so here goes!</span></div>
<div><span><br></span></div>
<div>
<span>I'm having trouble with getting Cython to play nicely with the coverage.py tool via the Cython.Coverage plug-in.</span><br>
</div>
<div><br></div>
<div>We have tests that are running from a different setup.py project than the one which contains the Cython files (the projects respectively being for the duration of this discussion the 'tests' project and the 'source' project). The generated coverage file contains the line data for individual .pyx files, but the paths of the .pyx files themselves&nbsp;within&nbsp;the .coverage file appear to be rooted in the tests project rather than the site-packages folder in which the source project's files were installed (i.e. paths that never exist throughout the edit-build-test cycle).<div><br></div>
<div>After editing the .coverage file manually to point to the source project directory, coverage.py reports that the .pyx files are not python files and errors out... but the .coveragerc file contains the plug-in module via `[run]\n plugins = Cython.Coverage`, so I'd have expected the Cython coverage.py plug-in to catch those files and handle them instead of letting the rest of coverage.py get confused over them.</div>
<div><br></div>
<div>Furthermore, this is against the 4.0 coverage.py release rather than the 4.0b2 coverage.py release (which appears to be the version against which the latest edit to the Cython code base was made). I don't see any API differences between the two in the plug-ins, but I might have missed something so maybe that's relevant.<br><div><br></div>
<div>A secondary issue: the .pyx files are never copied over to the installation directory, so even if the paths were 'correct' (with respect to being copied into the site-packages folder or being accessible via developer mode pip-installs), they wouldn't be found. This seems like it wouldn't necessarily be an issue from a cursory glance at the source code as the file tracer in the Cython coverage.py plug-in appears to spit out blank lines in such a scenario.</div>
</div>
<div><br></div>
<div>Complicating matters further, we're running the tests through pytest. That said, I've given up on trying to get pytest-cov to play nicely with this until I can get a manual invocation of `coverage report ...` to produce output describing the .pyx files.</div>
<div><br></div>
<div>Thanks for reading this far. Got any pointers?</div>
<div><br></div>
<div><br></div>
<div>----------------<span>----------------</span><span>----------------</span><span>----------------</span>
</div>
<div>
<div>
<span>Context:&nbsp;</span><span><a href="https://github.com/soltanmm/grpc/tree/cy-wip/src/python">https://github.com/soltanmm/grpc/tree/cy-wip/src/python</a></span>
</div>
<div><span>Basic steps to run the tests:</span></div>
<div><ul>
<li><span>cd $GRPC_REPOSITORY_ROOT</span></li>
<li><span>make</span></li>
<li>
<span>CONFIG=opt ./tools/run_tests/build_python.sh 2.7</span><br>
</li>
<li><span>CONFIG=opt PYVER=2.7 ./run_tests/run_python.sh</span></li>
</ul></div>
</div>
</div>
</div></div>

Gmane