Picon

[Cython] Bug in list comprehension

Hello,

I've think I've found a bug in Cython (or is this just a funny feature?). The error occurs when the variable name used as source for the iteration is also used as a value.

Given the files test1.pyx (cython) and test2.py (python):

test1.pyx:
def test():
    x = range(10)
    return [x for x in x]

test2.py:
def test():
    x = range(10)
    return [x for x in x]

The results in running them are:
>>> import test1, test2
>>> test1.test()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "test1.pyx", line 4, in test1.test (/Users/leandro/.pyxbld/temp.macosx-10.6-intel-3.4/pyrex/test1.c:666)
    return [x for x in x]
UnboundLocalError: local variable 'x' referenced before assignment
>>> test2.test()
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>>

A regular for loop will behave correctly:

code:
def test():
    x = range(10)
    ret = []
    for x in x:
        ret.append(x)
    return ret

result:
>>> import test1, test2
>>> test1.test()
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> test2.test()
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>>

Cheers, Leandro
<div><div dir="ltr">Hello,<div><br></div>
<div>I've think I've found a bug in Cython (or is this just a funny feature?). The error occurs when the variable name used as source for the iteration is also used as a value.</div>
<div><br></div>
<div>Given the files test1.pyx (cython) and test2.py (python):</div>
<div><br></div>
<div>test1.pyx:</div>
<div>
<div>def test():</div>
<div>&nbsp; &nbsp; x = range(10)</div>
<div>&nbsp; &nbsp; return [x for x in x]</div>
</div>
<div><br></div>
<div>test2.py:</div>
<div>
<div>def test():</div>
<div>&nbsp; &nbsp; x = range(10)</div>
<div>&nbsp; &nbsp; return [x for x in x]</div>
</div>
<div><br></div>
<div>The results in running them are:</div>
<div>
<div>&gt;&gt;&gt; import test1, test2</div>
<div>&gt;&gt;&gt; test1.test()</div>
<div>Traceback (most recent call last):</div>
<div>&nbsp; File "&lt;stdin&gt;", line 1, in &lt;module&gt;</div>
<div>&nbsp; File "test1.pyx", line 4, in test1.test (/Users/leandro/.pyxbld/temp.macosx-10.6-intel-3.4/pyrex/test1.c:666)</div>
<div>&nbsp; &nbsp; return [x for x in x]</div>
<div>UnboundLocalError: local variable 'x' referenced before assignment</div>
<div>&gt;&gt;&gt; test2.test()</div>
<div>[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]</div>
<div>&gt;&gt;&gt;</div>
</div>
<div><br></div>
<div>A regular for loop will behave correctly:</div>
<div><br></div>
<div>code:</div>
<div>
<div>def test():</div>
<div>&nbsp; &nbsp; x = range(10)</div>
<div>&nbsp; &nbsp; ret = []</div>
<div>&nbsp; &nbsp; for x in x:</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; ret.append(x)</div>
<div>&nbsp; &nbsp; return ret</div>
</div>
<div><br></div>
<div>result:</div>
<div>
<div>&gt;&gt;&gt; import test1, test2</div>
<div>&gt;&gt;&gt; test1.test()</div>
<div>[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]</div>
<div>&gt;&gt;&gt; test2.test()</div>
<div>[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]</div>
<div>&gt;&gt;&gt;</div>
</div>
<div><br></div>
<div>Cheers, Leandro</div>
</div></div>
Kevin Thornton | 16 Oct 21:50 2015
Picon

[Cython] Exposing C++'s <iterator>

Hello,

I was wanting to use C++s' back_inserter in Cython, and I got it working.  I'm writing the list to ask about what more I should do before submitting a pull request.

In short, the following definitions were sufficient:

##Define the basic iterator types (this is incomplete)
cdef extern from "<iterator>" namespace "std":
    cdef cppclass iterator[Category,T,Distance,Pointer,Reference]:
        pass
    cdef cppclass output_iterator_tag:
        pass
    #The goal here is to get back_inserter working, as it is handy.
    cdef cppclass back_insert_iterator[T](iterator[output_iterator_tag,void,void,void,void]):
        pass
    back_insert_iterator[CONTAINER] back_inserter[CONTAINER](CONTAINER &)

##Not shown here, but I've gotten C++11's move algorithm to work, too,
##using same kind of prototype
cdef extern from "<algorithm>" namespace "std":
    OUTPUT copy[INPUT,OUTPUT](INPUT,INPUT,OUTPUT)

I've worked up a tiny test package that is on GitHub: https://github.com/molpopgen/cython_cpp_iterators

The tests work, and I can indeed copy amongst C++ containers.  

I've also gotten the C++ move algorithm to work.  The prototype is the same as for 'copy' above.

I'd be willing to define the remaining classes from <iterator> and add a .pxd file.

Any input would be appreciated.

Thanks,

Kevin


<div><div dir="ltr">Hello,<div><br></div>
<div>I was wanting to use C++s' back_inserter in Cython, and I got it working.&nbsp; I'm writing the list to ask about what more I should do before submitting a pull request.</div>
<div><br></div>
<div>In short, the following definitions were sufficient:</div>
<div><br></div>
<div>
<div>##Define the basic iterator types (this is incomplete)</div>
<div>cdef extern from "&lt;iterator&gt;" namespace "std":</div>
<div>&nbsp; &nbsp; cdef cppclass iterator[Category,T,Distance,Pointer,Reference]:</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; pass</div>
<div>&nbsp; &nbsp; cdef cppclass output_iterator_tag:</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; pass</div>
<div>&nbsp; &nbsp; #The goal here is to get back_inserter working, as it is handy.</div>
<div>&nbsp; &nbsp; cdef cppclass back_insert_iterator[T](iterator[output_iterator_tag,void,void,void,void]):</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; pass</div>
<div>&nbsp; &nbsp; back_insert_iterator[CONTAINER] back_inserter[CONTAINER](CONTAINER &amp;)</div>
<div><br></div>
<div>##Not shown here, but I've gotten C++11's move algorithm to work, too,</div>
<div>##using same kind of prototype</div>
<div>cdef extern from "&lt;algorithm&gt;" namespace "std":</div>
<div>&nbsp; &nbsp; OUTPUT copy[INPUT,OUTPUT](INPUT,INPUT,OUTPUT)</div>
</div>
<div><br></div>
<div>I've worked up a tiny test package that is on GitHub:&nbsp;<a href="https://github.com/molpopgen/cython_cpp_iterators">https://github.com/molpopgen/cython_cpp_iterators</a>
</div>
<div><br></div>
<div>The tests work, and I can indeed copy amongst C++ containers. &nbsp;</div>
<div><br></div>
<div>I've also gotten the C++ move algorithm to work.&nbsp; The prototype is the same as for 'copy' above.</div>
<div><br></div>
<div>I'd be willing to define the remaining classes from &lt;iterator&gt; and add a .pxd file.</div>
<div><br></div>
<div>Any input would be appreciated.</div>
<div><br></div>
<div>Thanks,</div>
<div><br></div>
<div>Kevin</div>
<div><br></div>
<div><br></div>
</div></div>
Carlos Pita | 16 Oct 21:05 2015
Picon
Gravatar

[Cython] [Bug] Unable to cast to python type.

Hi,

import cython as cy
y = cy.cast('list', x)

fails to compile with "AttributeError: 'TypecastNode' object has no attribute 'typecheck'".

But the following examples do compile:

y = <list> x

y = cy.cast('int', x)

Cheers
--
Carlos

<div><div dir="ltr">
<div>
<div>
<div>
<div>
<div>
<div>Hi,<br>
</div>
<div>
<br>import cython as cy<br>y = cy.cast('list', x)<br><br>
</div>fails to compile with "AttributeError: 'TypecastNode' object has no attribute 'typecheck'".<br><br>
</div>But the following examples do compile:<br><br>
</div>y = &lt;list&gt; x<br><br>
</div>y = cy.cast('int', x)<br><br>
</div>Cheers<br>--<br>
</div>Carlos<br><div><div><div><br></div></div></div>
</div></div>
Carlos Pita | 16 Oct 20:53 2015
Picon
Gravatar

[Cython] [Bug] Generator comprehension not compiling

Hi,

in cython 0.24.0a0 this:

----------
import cython as cy

<at> cy.cfunc
def f(cats):
    ''.join(c for c in cats)
----------

fails to compile with: Compiler crash in AnalyseDeclarationsTransform.

Nevertheless the following close variations do compile:

----------
import cython as cy

<at> cy.cfunc
def f(cats):
    ''.join([c for c in cats])
----------

----------
cdef f(cats):
    ''.join(c for c in cats)
----------

Cheers
--
Carlos


<div><div dir="ltr">
<div>
<div>
<div>
<div>
<div>Hi,<br><br>
</div>in cython 0.24.0a0 this:<br><br>----------<br>import cython as cy<br><br> <at> cy.cfunc<br>def f(cats):<br>&nbsp;&nbsp;&nbsp; ''.join(c for c in cats)<br>----------<br><br>
</div>fails to compile with: Compiler crash in AnalyseDeclarationsTransform.<br><br>
</div>Nevertheless the following close variations do compile:<br><br>----------<br>import cython as cy<br><br> <at> cy.cfunc<br>def f(cats):<br>&nbsp;&nbsp;&nbsp; ''.join([c for c in cats])<br>----------<br><br>----------<br>cdef f(cats):<br>&nbsp;&nbsp;&nbsp; ''.join(c for c in cats)<br>----------<br><br>
</div>Cheers<br>--<br>
</div>Carlos<br><div><div>
<br><div><br></div>
</div></div>
</div></div>
Stefan Behnel | 11 Oct 16:30 2015
Picon

[Cython] can we deprecate for-from loops?

Hi!

The syntax construct "for i from 0 <= i < 10" has been silently outdated
for years. Can we start issuing a warning that normal range() loops are
preferred?

While I don't see it going to go away any time soon, I'd like to make users
aware that it's an unnecessary syntax wart that's best avoided and that
should really not be used in new code.

Stefan

[Cython] git master fails to compile scipy

hi,
since ab78f93b3ffa88183a0d2aae6b692e394c51f860 scipy does not build anymore:
the failing file is:
https://github.com/scipy/scipy/blob/master/scipy/sparse/csgraph/_reordering.pyx

error:
Traceback (most recent call last):
  File "/usr/bin/cython", line 8, in <module>
    main(command_line = 1)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Main.py",
line 704, in main
    result = compile(sources, options)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Main.py",
line 679, in compile
    return compile_multiple(source, options)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Main.py",
line 657, in compile_multiple
    result = run_pipeline(source, options, context=context)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Main.py",
line 487, in run_pipeline
    err, enddata = Pipeline.run_pipeline(pipeline, source)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Pipeline.py",
line 365, in run_pipeline
    data = phase(data)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Pipeline.py",
line 53, in generate_pyx_code_stage
    module_node.process_implementation(options, result)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/ModuleNode.py",
line 118, in process_implementation
    self.generate_c_code(env, options, result)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/ModuleNode.py",
line 349, in generate_c_code
    self.body.generate_function_definitions(env, code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
line 436, in generate_function_definitions
    stat.generate_function_definitions(env, code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
line 436, in generate_function_definitions
    stat.generate_function_definitions(env, code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/FusedNode.py",
line 785, in generate_function_definitions
    stat.generate_function_definitions(env, code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
line 3068, in generate_function_definitions
    FuncDefNode.generate_function_definitions(self, env, code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
line 1935, in generate_function_definitions
    self.generate_function_body(env, code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
line 1691, in generate_function_body
    self.body.generate_execution_code(code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
line 442, in generate_execution_code
    stat.generate_execution_code(code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/UtilNodes.py",
line 321, in generate_execution_code
    self.body.generate_execution_code(code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
line 6419, in generate_execution_code
    self.body.generate_execution_code(code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
line 442, in generate_execution_code
    stat.generate_execution_code(code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
line 5865, in generate_execution_code
    if_clause.generate_execution_code(code, end_label, is_last=i == last)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
line 5908, in generate_execution_code
    self.body.generate_execution_code(code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
line 442, in generate_execution_code
    stat.generate_execution_code(code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
line 4800, in generate_execution_code
    self.generate_assignment_code(code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
line 5091, in generate_assignment_code
    self.lhs.generate_assignment_code(self.rhs, code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/ExprNodes.py",
line 3897, in generate_assignment_code
    self.generate_buffer_setitem_code(rhs, code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/ExprNodes.py",
line 3905, in generate_buffer_setitem_code
    buffer_entry, ptrexpr = self.buffer_lookup_code(code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/ExprNodes.py",
line 3873, in buffer_lookup_code
    for ivar in self.indices]
AttributeError: 'NoneType' object has no attribute 'signed'
Jason Madden | 9 Oct 18:04 2015
Gravatar

[Cython] Bug - PyPy tuple leak in __Pyx_PyObject_CallOneArg?

Hello,

I'm one of the maintainers of the gevent concurrency library, which has recently been ported to run on PyPy.
Under PyPy, a small portion of the code is compiled with Cython in order to get the desired atomic semantics
(specifically, a semaphore class). We recently had a user report an easily reproducible leak of tuples of
one element. Tracking it down, it appears that __Pyx_PyObject_CallOneArg creates a new tuple under
PyPy, but neglects to free it. This was tested with Cython 0.23.3 and PyPy 2.6.1.

Our Cython code contained a loop like this, and every iteration of the loop leaked a tuple:

    for link in links:
        link(self)

The C output for that last line looked like this:

    __Pyx_PyObject_CallOneArg(__pyx_t_10, ((PyObject *)__pyx_v_self));...

And, under PyPy, the implementation of __Pyx_PyObject_CallOneArg is different than it is under CPython.
Specifically, with Cython 0.23.3 this is the code:

    #if CYTHON_COMPILING_IN_CPYTHON
    ...
    #else
    static CYTHON_INLINE PyObject* __Pyx_PyObject_CallOneArg(PyObject *func, PyObject *arg) {
        PyObject* args = PyTuple_Pack(1, arg);
        return (likely(args)) ? __Pyx_PyObject_Call(func, args, NULL) : NULL;
    }
    #endif

PyTuple_Pack is documented as returning a new reference
(https://docs.python.org/2/c-api/tuple.html#c.PyTuple_Pack), so it seems to me like this code
should be decrementing the reference to the tuple before returning (much like the
__Pyx__PyObject_CallOneArg function does under CPython).

Changing the loop to avoid the use of CallOneArg seemed to resolve the tuple leak, presumably at some
execution time cost:

   args = (self,)
   for link in links:
       link(*self)

There's some additional background at https://bitbucket.org/pypy/pypy/issues/2149/memory-leak-for-python-subclass-of-cpyext#comment-22347393

Am I interpreting this correctly to be a bug, or could there be something wrong in the way we're handling
callbacks? Please let me know if there's any further information I can provide.

Thanks,
Jason
Stephen LARROQUE | 8 Oct 16:35 2015
Picon

Re: [Cython] cython-devel Digest, Vol 57, Issue 5

Hello Masood,

I can only answer the second issue. You need to explicitly include the *.pyx files in your MANIFEST.in, because else this .pyx files aren't automatically managed by the distutils package so it doesn't know it has to add the .pyx files.

I have recently packaged two libraries, with different hierarchy layouts, so you can maybe use one of them as an example to fix your packaging:

First project with a usual package structure:

Second project with a flatout structure:

The commands I use to locally build the package are:

python setup.py sdist --formats=gztar,zip bdist_wininst --plat-name=win32
python setup.py sdist bdist_egg bdist_wheel --plat-name=win32

If your packaging config is correct, you should see the .pyx files in those builds.

And to build all other formats and upload to pypi, I use the python library Twine:

twine upload dist/*

Hope this can help you.

Best regards,
Stephen Larroque


2015-10-08 12:00 GMT+02:00 <cython-devel-request-+ZN9ApsXKcEdnm+yROfE0A@public.gmane.org>:
Send cython-devel mailing list submissions to
        cython-devel-+ZN9ApsXKcEdnm+yROfE0A@public.gmane.org

To subscribe or unsubscribe via the World Wide Web, visit
        https://mail.python.org/mailman/listinfo/cython-devel
or, via email, send a message with subject or body 'help' to
        cython-devel-request-+ZN9ApsXKcEdnm+yROfE0A@public.gmane.org

You can reach the person managing the list at
        cython-devel-owner-+ZN9ApsXKcEdnm+yROfE0A@public.gmane.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of cython-devel digest..."


Today's Topics:

   1. Cython coverage and multiple projects (Masood Malekghassemi)


----------------------------------------------------------------------

Message: 1
Date: Wed, 7 Oct 2015 16:06:01 -0700
From: Masood Malekghassemi <atash <at> google.com>
To: cython-devel-+ZN9ApsXKcEdnm+yROfE0A@public.gmane.org
Subject: [Cython] Cython coverage and multiple projects
Message-ID:
        <CALweS8f=xeQfhmydXR5C70mbO1=1hibmsTgBPuvCuiUEqE=g8w-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
Content-Type: text/plain; charset="utf-8"

I'd asked this of Robert Bradshaw earlier and was pointed in this direction
(specifically towards Stefan), so here goes!

I'm having trouble with getting Cython to play nicely with the coverage.py
tool via the Cython.Coverage plug-in.

We have tests that are running from a different setup.py project than the
one which contains the Cython files (the projects respectively being for
the duration of this discussion the 'tests' project and the 'source'
project). The generated coverage file contains the line data for individual
.pyx files, but the paths of the .pyx files themselves *within* the
.coverage file appear to be rooted in the tests project rather than the
site-packages folder in which the source project's files were installed
(i.e. paths that never exist throughout the edit-build-test cycle).

After editing the .coverage file manually to point to the source project
directory, coverage.py reports that the .pyx files are not python files and
errors out... but the .coveragerc file contains the plug-in module via
`[run]\n plugins = Cython.Coverage`, so I'd have expected the Cython
coverage.py plug-in to catch those files and handle them instead of letting
the rest of coverage.py get confused over them.

Furthermore, this is against the 4.0 coverage.py release rather than the
4.0b2 coverage.py release (which appears to be the version against which
the latest edit to the Cython code base was made). I don't see any API
differences between the two in the plug-ins, but I might have missed
something so maybe that's relevant.

A secondary issue: the .pyx files are never copied over to the installation
directory, so even if the paths were 'correct' (with respect to being
copied into the site-packages folder or being accessible via developer mode
pip-installs), they wouldn't be found. This seems like it wouldn't
necessarily be an issue from a cursory glance at the source code as the
file tracer in the Cython coverage.py plug-in appears to spit out blank
lines in such a scenario.

Complicating matters further, we're running the tests through pytest. That
said, I've given up on trying to get pytest-cov to play nicely with this
until I can get a manual invocation of `coverage report ...` to produce
output describing the .pyx files.

Thanks for reading this far. Got any pointers?


----------------------------------------------------------------
Context: https://github.com/soltanmm/grpc/tree/cy-wip/src/python
Basic steps to run the tests:

   - cd $GRPC_REPOSITORY_ROOT
   - make
   - CONFIG=opt ./tools/run_tests/build_python.sh 2.7
   - CONFIG=opt PYVER=2.7 ./run_tests/run_python.sh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20151007/11793594/attachment-0001.html>

------------------------------

Subject: Digest Footer

_______________________________________________
cython-devel mailing list
cython-devel-+ZN9ApsXKcEdnm+yROfE0A@public.gmane.org
https://mail.python.org/mailman/listinfo/cython-devel


------------------------------

End of cython-devel Digest, Vol 57, Issue 5
*******************************************

<div>
<div dir="ltr">Hello Masood,<div><br></div>
<div>I can only answer the second issue. You need to explicitly include the *.pyx files in your MANIFEST.in, because else this .pyx files aren't automatically managed by the distutils package so it doesn't know it has to add the .pyx files.</div>
<div><br></div>
<div>I have recently packaged two libraries, with different hierarchy layouts, so you can maybe use one of them as an example to fix your packaging:</div>
<div><br></div>
<div>First project with a usual package structure:</div>
<div>
<a href="https://github.com/lrq3000/unireedsolomon/blob/master/MANIFEST.in">https://github.com/lrq3000/unireedsolomon/blob/master/MANIFEST.in</a><br>
</div>
<div><a href="https://github.com/lrq3000/unireedsolomon/blob/master/setup.py">https://github.com/lrq3000/unireedsolomon/blob/master/setup.py</a></div>
<div><br></div>
<div>Second project with a flatout structure:</div>
<div>
<a href="https://github.com/lrq3000/reedsolomon/blob/master/MANIFEST.in">https://github.com/lrq3000/reedsolomon/blob/master/MANIFEST.in</a><br>
</div>
<div><a href="https://github.com/lrq3000/reedsolomon/blob/master/setup.py">https://github.com/lrq3000/reedsolomon/blob/master/setup.py</a></div>
<div><br></div>
<div>The commands I use to locally build the package are:</div>
<div><br></div>
<div>
<div>python setup.py sdist --formats=gztar,zip bdist_wininst --plat-name=win32</div>
<div>python setup.py sdist bdist_egg bdist_wheel --plat-name=win32</div>
</div>
<div><br></div>
<div>If your packaging config is correct, you should see the .pyx files in those builds.</div>
<div><br></div>
<div>And to build all other formats and upload to pypi, I use the python library Twine:</div>
<div><br></div>
<div>twine upload dist/*<br>
</div>
<div><br></div>
<div>Hope this can help you.</div>
<div><br></div>
<div>Best regards,</div>
<div>Stephen Larroque</div>
<div><br></div>
</div>
<div class="gmail_extra">
<br><div class="gmail_quote">2015-10-08 12:00 GMT+02:00  <span dir="ltr">&lt;<a href="mailto:cython-devel-request@..." target="_blank">cython-devel-request@...</a>&gt;</span>:<br><blockquote class="gmail_quote">Send cython-devel mailing list submissions to<br>
&nbsp; &nbsp; &nbsp; &nbsp; <a href="mailto:cython-devel@...">cython-devel@...</a><br><br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
&nbsp; &nbsp; &nbsp; &nbsp; <a href="https://mail.python.org/mailman/listinfo/cython-devel" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/cython-devel</a><br>
or, via email, send a message with subject or body 'help' to<br>
&nbsp; &nbsp; &nbsp; &nbsp; <a href="mailto:cython-devel-request <at> python.org">cython-devel-request@...</a><br><br>
You can reach the person managing the list at<br>
&nbsp; &nbsp; &nbsp; &nbsp; <a href="mailto:cython-devel-owner@...">cython-devel-owner@...</a><br><br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of cython-devel digest..."<br><br><br>
Today's Topics:<br><br>
&nbsp; &nbsp;1. Cython coverage and multiple projects (Masood Malekghassemi)<br><br><br>
----------------------------------------------------------------------<br><br>
Message: 1<br>
Date: Wed, 7 Oct 2015 16:06:01 -0700<br>
From: Masood Malekghassemi &lt;<a href="mailto:atash@...">atash <at> google.com</a>&gt;<br>
To: <a href="mailto:cython-devel@...">cython-devel@...</a><br>
Subject: [Cython] Cython coverage and multiple projects<br>
Message-ID:<br>
&nbsp; &nbsp; &nbsp; &nbsp; &lt;CALweS8f=xeQfhmydXR5C70mbO1=1hibmsTgBPuvCuiUEqE=<a href="mailto:g8w@...">g8w@...</a>&gt;<br>
Content-Type: text/plain; charset="utf-8"<br><br>
I'd asked this of Robert Bradshaw earlier and was pointed in this direction<br>
(specifically towards Stefan), so here goes!<br><br>
I'm having trouble with getting Cython to play nicely with the coverage.py<br>
tool via the Cython.Coverage plug-in.<br><br>
We have tests that are running from a different setup.py project than the<br>
one which contains the Cython files (the projects respectively being for<br>
the duration of this discussion the 'tests' project and the 'source'<br>
project). The generated coverage file contains the line data for individual<br>
.pyx files, but the paths of the .pyx files themselves *within* the<br>
.coverage file appear to be rooted in the tests project rather than the<br>
site-packages folder in which the source project's files were installed<br>
(i.e. paths that never exist throughout the edit-build-test cycle).<br><br>
After editing the .coverage file manually to point to the source project<br>
directory, coverage.py reports that the .pyx files are not python files and<br>
errors out... but the .coveragerc file contains the plug-in module via<br>
`[run]\n plugins = Cython.Coverage`, so I'd have expected the Cython<br>
coverage.py plug-in to catch those files and handle them instead of letting<br>
the rest of coverage.py get confused over them.<br><br>
Furthermore, this is against the 4.0 coverage.py release rather than the<br>
4.0b2 coverage.py release (which appears to be the version against which<br>
the latest edit to the Cython code base was made). I don't see any API<br>
differences between the two in the plug-ins, but I might have missed<br>
something so maybe that's relevant.<br><br>
A secondary issue: the .pyx files are never copied over to the installation<br>
directory, so even if the paths were 'correct' (with respect to being<br>
copied into the site-packages folder or being accessible via developer mode<br>
pip-installs), they wouldn't be found. This seems like it wouldn't<br>
necessarily be an issue from a cursory glance at the source code as the<br>
file tracer in the Cython coverage.py plug-in appears to spit out blank<br>
lines in such a scenario.<br><br>
Complicating matters further, we're running the tests through pytest. That<br>
said, I've given up on trying to get pytest-cov to play nicely with this<br>
until I can get a manual invocation of `coverage report ...` to produce<br>
output describing the .pyx files.<br><br>
Thanks for reading this far. Got any pointers?<br><br><br>
----------------------------------------------------------------<br>
Context: <a href="https://github.com/soltanmm/grpc/tree/cy-wip/src/python" rel="noreferrer" target="_blank">https://github.com/soltanmm/grpc/tree/cy-wip/src/python</a><br>
Basic steps to run the tests:<br><br>
&nbsp; &nbsp;- cd $GRPC_REPOSITORY_ROOT<br>
&nbsp; &nbsp;- make<br>
&nbsp; &nbsp;- CONFIG=opt ./tools/run_tests/build_python.sh 2.7<br>
&nbsp; &nbsp;- CONFIG=opt PYVER=2.7 ./run_tests/run_python.sh<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: &lt;<a href="http://mail.python.org/pipermail/cython-devel/attachments/20151007/11793594/attachment-0001.html" rel="noreferrer" target="_blank">http://mail.python.org/pipermail/cython-devel/attachments/20151007/11793594/attachment-0001.html</a>&gt;<br><br>
------------------------------<br><br>
Subject: Digest Footer<br><br>
_______________________________________________<br>
cython-devel mailing list<br><a href="mailto:cython-devel@...">cython-devel@...</a><br><a href="https://mail.python.org/mailman/listinfo/cython-devel" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/cython-devel</a><br><br><br>
------------------------------<br><br>
End of cython-devel Digest, Vol 57, Issue 5<br>
*******************************************<br>
</blockquote>
</div>
<br>
</div>
</div>

[Cython] Cython coverage and multiple projects

I'd asked this of Robert Bradshaw earlier and was pointed in this direction (specifically towards Stefan), so here goes!

I'm having trouble with getting Cython to play nicely with the coverage.py tool via the Cython.Coverage plug-in.

We have tests that are running from a different setup.py project than the one which contains the Cython files (the projects respectively being for the duration of this discussion the 'tests' project and the 'source' project). The generated coverage file contains the line data for individual .pyx files, but the paths of the .pyx files themselves within the .coverage file appear to be rooted in the tests project rather than the site-packages folder in which the source project's files were installed (i.e. paths that never exist throughout the edit-build-test cycle).

After editing the .coverage file manually to point to the source project directory, coverage.py reports that the .pyx files are not python files and errors out... but the .coveragerc file contains the plug-in module via `[run]\n plugins = Cython.Coverage`, so I'd have expected the Cython coverage.py plug-in to catch those files and handle them instead of letting the rest of coverage.py get confused over them.

Furthermore, this is against the 4.0 coverage.py release rather than the 4.0b2 coverage.py release (which appears to be the version against which the latest edit to the Cython code base was made). I don't see any API differences between the two in the plug-ins, but I might have missed something so maybe that's relevant.

A secondary issue: the .pyx files are never copied over to the installation directory, so even if the paths were 'correct' (with respect to being copied into the site-packages folder or being accessible via developer mode pip-installs), they wouldn't be found. This seems like it wouldn't necessarily be an issue from a cursory glance at the source code as the file tracer in the Cython coverage.py plug-in appears to spit out blank lines in such a scenario.

Complicating matters further, we're running the tests through pytest. That said, I've given up on trying to get pytest-cov to play nicely with this until I can get a manual invocation of `coverage report ...` to produce output describing the .pyx files.

Thanks for reading this far. Got any pointers?


----------------------------------------------------------------
Basic steps to run the tests:
  • cd $GRPC_REPOSITORY_ROOT
  • make
  • CONFIG=opt ./tools/run_tests/build_python.sh 2.7
  • CONFIG=opt PYVER=2.7 ./run_tests/run_python.sh
<div><div dir="ltr">
<div><span>I'd asked this of Robert Bradshaw earlier and was pointed in this direction (specifically towards Stefan), so here goes!</span></div>
<div><span><br></span></div>
<div>
<span>I'm having trouble with getting Cython to play nicely with the coverage.py tool via the Cython.Coverage plug-in.</span><br>
</div>
<div><br></div>
<div>We have tests that are running from a different setup.py project than the one which contains the Cython files (the projects respectively being for the duration of this discussion the 'tests' project and the 'source' project). The generated coverage file contains the line data for individual .pyx files, but the paths of the .pyx files themselves&nbsp;within&nbsp;the .coverage file appear to be rooted in the tests project rather than the site-packages folder in which the source project's files were installed (i.e. paths that never exist throughout the edit-build-test cycle).<div><br></div>
<div>After editing the .coverage file manually to point to the source project directory, coverage.py reports that the .pyx files are not python files and errors out... but the .coveragerc file contains the plug-in module via `[run]\n plugins = Cython.Coverage`, so I'd have expected the Cython coverage.py plug-in to catch those files and handle them instead of letting the rest of coverage.py get confused over them.</div>
<div><br></div>
<div>Furthermore, this is against the 4.0 coverage.py release rather than the 4.0b2 coverage.py release (which appears to be the version against which the latest edit to the Cython code base was made). I don't see any API differences between the two in the plug-ins, but I might have missed something so maybe that's relevant.<br><div><br></div>
<div>A secondary issue: the .pyx files are never copied over to the installation directory, so even if the paths were 'correct' (with respect to being copied into the site-packages folder or being accessible via developer mode pip-installs), they wouldn't be found. This seems like it wouldn't necessarily be an issue from a cursory glance at the source code as the file tracer in the Cython coverage.py plug-in appears to spit out blank lines in such a scenario.</div>
</div>
<div><br></div>
<div>Complicating matters further, we're running the tests through pytest. That said, I've given up on trying to get pytest-cov to play nicely with this until I can get a manual invocation of `coverage report ...` to produce output describing the .pyx files.</div>
<div><br></div>
<div>Thanks for reading this far. Got any pointers?</div>
<div><br></div>
<div><br></div>
<div>----------------<span>----------------</span><span>----------------</span><span>----------------</span>
</div>
<div>
<div>
<span>Context:&nbsp;</span><span><a href="https://github.com/soltanmm/grpc/tree/cy-wip/src/python">https://github.com/soltanmm/grpc/tree/cy-wip/src/python</a></span>
</div>
<div><span>Basic steps to run the tests:</span></div>
<div><ul>
<li><span>cd $GRPC_REPOSITORY_ROOT</span></li>
<li><span>make</span></li>
<li>
<span>CONFIG=opt ./tools/run_tests/build_python.sh 2.7</span><br>
</li>
<li><span>CONFIG=opt PYVER=2.7 ./run_tests/run_python.sh</span></li>
</ul></div>
</div>
</div>
</div></div>
Jeroen Demeyer | 5 Oct 16:04 2015
Picon

[Cython] __dealloc__ called even if __cinit__ failed

Hello,

I'm not entirely sure if this is a feature or bug...

I noticed that, if a __cinit__ call raises an exception on an object, 
that __dealloc__ on that same object is still called.

The problem is that __cinit__ is supposed to set up the C-level 
attributes for that object. If this somehow fails, then the object isn't 
in a consistent state yet, so it makes no sense to call __dealloc__ on it.

Do you consider this a Cython bug or should I manually protect my code 
against this?

Thanks,
Jeroen.
Josh Ayers | 3 Oct 18:04 2015

[Cython] Cython 0.23.3 compiler warning with fused types

Cython devs,

Using Cython 0.23.3 and Python 3.4.3, the following Cython code:

ctypedef fused floating:
    float
    double

cpdef add_one(floating [:] array):
    cdef Py_ssize_t i
    for i in range(array.shape[0]):
        array[i] += 1.0

produces the C code below.  It raises a compiler warning because a long
is assigned to a char.

char __pyx_v_kind;
...
long __pyx_t_11;
...
__pyx_t_11 = __Pyx_PyObject_Ord(__pyx_t_8);
...
__pyx_v_kind = __pyx_t_11;

Thanks,
Josh Ayers

Gmane