Stefan Behnel | 2 Feb 18:13 2014
Picon

Re: [Cython] Bug: _PyType_Lookup shortcut in Cython 0.20 breaks lookup of __exit__ in Python 2.6

[forwarding to cython-devel]

Wouter Bolsterlee, 02.02.2014 16:40:
> Cython 0.20 introduced __Pyx_PyObject_LookupSpecial as a shortcut to
> lookup special methods. The commit that introduced this change is
> 8f6412275c4c2d1dcf43ad40306f858b114104ed and can be seen here:
> 
> https://github.com/cython/cython/commit/8f6412275c4c2d1dcf43ad40306f858b114104ed
>         
> It seems this change does not work with Python 2.6. The newly introduced
> function calls _PyType_Lookup(), which fails to lookup (at least)
> __exit__ (on a threading.Lock) instance when run under Python 2.6. The
> exact call is here:
> 
> https://github.com/cython/cython/commit/8f6412275c4c2d1dcf43ad40306f858b114104ed#diff-83083bdce5dcca284394ead83b8d3f99R1053
>         
> For my Plyvel project (https://plyvel.readthedocs.org/ and
> https://github.com/wbolster/plyvel) this issue manifests as follows:
> 
> $ cython --version
>         Cython version 0.20
>         
>         $ python 
>         Python 2.6.8 (unknown, Jan 26 2013, 14:35:25) 
>         [GCC 4.7.2] on linux2
>         Type "help", "copyright", "credits" or "license" for more
> information.
>         >>> import plyvel
>         >>> db = plyvel.DB('/tmp/testdb', create_if_missing=True)
>         >>> db.close()
(Continue reading)

Stefan Behnel | 31 Jan 12:55 2014
Picon

[Cython] Sage build problem

Hi,

There's some kind of C++ enum issue in the Sage build.

Generated code in sage/libs/ppl.cpp:

"""
static PyObject *__pyx_pf_4sage_4libs_3ppl_11MIP_Problem_30solve(
             struct __pyx_obj_4sage_4libs_3ppl_MIP_Problem *__pyx_v_self) {
  enum Parma_Polyhedra_Library::PPL_MIP_Problem_Status __pyx_v_tmp;
"""

gcc error:

"""
sage/libs/ppl.cpp: In function 'PyObject*
__pyx_pf_4sage_4libs_3ppl_11MIP_Problem_30solve(__pyx_obj_4sage_4libs_3ppl_MIP_Problem*)':

sage/libs/ppl.cpp:5514:33: error: 'PPL_MIP_Problem_Status' in namespace
'Parma_Polyhedra_Library' does not name a type
sage/libs/ppl.cpp:5514:67: error: invalid type in declaration before ';' token
"""

The type of "tmp" seems to be inferred. That might be a hint that it could
be a problem in Cython.

The Cython code in question is also a bit funny:

sage/libs/ppl.pyx:1076
"""
(Continue reading)

R. Andrew Ohana | 28 Jan 23:34 2014
Picon

[Cython] Broken list multiplication

It seems like eaca39b00b1451fe6c846a0a670e4c8b39935525 broke some (very simple) instances of list multiplication when there are at least 3 terms in the product. e.g. on my system I'm getting these results:

L = [None] * 2 * 3
print(len(L))
3

L = 2 * [None] * 3
print(len(L))
3

L = [None] * 2 * 3 * 5
print(len(L))
5

--
Andrew
<div><div dir="ltr">
<div>
<div>
<div>
<div>
<div>
<div>It seems like eaca39b00b1451fe6c846a0a670e4c8b39935525 broke some (very simple) instances of list multiplication when there are at least 3 terms in the product. e.g. on my system I'm getting these results:</div>

<div><br></div>L = [None] * 2 * 3<br>
</div>print(len(L))<br>3<br><br>
</div>L = 2 * [None] * 3<br>
</div>print(len(L))<br>3<br><br>
</div>L = [None] * 2 * 3 * 5<br>
</div>print(len(L))<br>5<br clear="all"><div><div><div><div>

<div><div><div>
<br>-- <br><div dir="ltr">Andrew</div>
</div></div></div>
</div></div></div></div>
</div></div>
Syam Gadde | 28 Jan 19:41 2014

[Cython] memoryview refcount error

Hi again,

I'm exploring the typed memoryview feature with numpy and encountered a 
refcount error.  It seems to happen when I create a slice of a 
memoryview, and transpose it.  If I don't do both, I don't get the 
refcount error.  It happens in the error cleanup code.  I suspect that 
any error that causes the function to jump to the error cleanup code 
will do the same thing as the Exception below -- but that seemed the 
easiest way to demonstrate the problem. However, if there truly is an 
error in my code, please let me know! Thanks,

-syam

# BEGIN CODE
import numpy
cimport numpy

class MyException(Exception):
     def __init__(self):
         Exception.__init__(self)

def testfunc():
    a = numpy.arange(12).reshape([3,4])

    cdef numpy.int_t[:,:] a_view = a

    ## here is a slice followed by a transpose
    cdef numpy.int_t[:,:] b = a_view[:,:].T
    ## same thing happens with a more realistic slicing:
    #cdef numpy.int_t[:,:] b = a_view[1:2,:].T
    ## however, there is no error if I do this instead:
    #cdef numpy.int_t[:,:] b = a_view.T
    ## also no error if I do this instead:
    #cdef numpy.int_t[:,:] b = a_view[:,:]

    # The exception below is just to force the function to abort
    # and run the extraneous __PYX_XDEC_MEMVIEW in the error
    # cleanup code:
    #   Fatal Python error: Acquisition count is 0 (line XXXX)
    # Comment out this exception and we don't get the error
    raise MyException()

try:
     testfunc()
except MyException:
     pass
# END CODE

Julian Taylor | 24 Jan 22:26 2014

[Cython] 0.20 tests fail with python3.4b2

hi,

with python3.4b2 numpy_memoryview.acquire_release_cycle fails
looking at the source the test should be disabled, but its still run.

This might be a py3.4 change to doctest, but I know nothing about
doctest, so I wanted to check first if you know something about this.
Interestingly py3.4 runs 64 tests, while python3.3 only runs 32.
the __test__ variable contains the same functions in both python
versions (acquire_release is not in it)

======================================================================
FAIL: acquire_release_cycle (numpy_memoryview)
Doctest: numpy_memoryview.acquire_release_cycle
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.4/doctest.py", line 2187, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for
numpy_memoryview.acquire_release_cycle
  File
"/tmp/buildd/cython-0.20/build/work-dir/memoryview/c/numpy_memoryview/numpy_memoryview.cpython-34m.so",
line unknown line number, in acquire_release_cycle

----------------------------------------------------------------------
File
"/tmp/buildd/cython-0.20/build/work-dir/memoryview/c/numpy_memoryview/numpy_memoryview.cpython-34m.so",
line ?, in numpy_memoryview.acquire_release_cycle
Failed example:
    acquire_release_cycle(a)
Expected:
    deallocated!
Got nothing
Stefan Behnel | 23 Jan 21:10 2014
Picon

Re: [Cython] segfault due to using DECREF instead of XDECREF

Victor Makarov, 23.01.2014 21:02:
> I'd like to fix this if you don't mind.

Oh, I certainly don't mind. :)

Stefan

Stefan Behnel | 23 Jan 20:49 2014
Picon

[Cython] C code churn and benchmark diffs

Hi,

to get a better idea of what new optimisations (or C code changes in
general) actually bring, I've added a diffing step to the benchmark runner
jobs. The bigger ones now generate an additional output file
"cfiles.diff.gz" that contains a diff of the C files generated in the
current run against those of the last (successful) run. E.g. here:

https://sage.math.washington.edu:8091/hudson/view/bench/job/cython-devel-pybenchmarks-neu-py3k/

(That one should start looking a lot smaller tomorrow :)

While doing that, I noticed that there are a couple of unnecessary
differences in the C files that are due to dict iteration, which is
randomised in Py3.3+ (for security reasons). I fixed (most of?) them, so
repeated Cython runs should now generate pretty much identical output
files, if you strip the file header comment (diff can do that). The rest is
generated deterministically in source code order already.

It should be possible to do the same thing for the Sage build, I guess.
That would also be interesting to look at from time to time, especially
when things start failing for some reason.

Stefan
Syam Gadde | 21 Jan 23:00 2014

[Cython] segfault due to using DECREF instead of XDECREF

(apologies to the moderator for multiple postings -- I'm trying to figure out which of my email addresses is
actually being exported to the public!)

Hi,

It seems that cython's static code analysis is failing on the following
code.  It thinks that a particular variable (myobject2) must not be
NULL, but in fact it is, so the __Pyx_DECREF_SET segfaults.  This came
from third-party code, so I could fix it with an explicit initialization
to None, but I'd love to have Cython deal with it directly.  Thanks for
your help!

#<<<BEGIN tmpnone.py>>>
# When compiled with Cython 0.19.2 or 0.20dev, the following code
# segfaults on import

import random

def myfunc():
      ## uncommenting the following fixes things
      #myobject2 = None

      myfalse = random.random() > 2

      if myfalse:
          myobject = None
          myobject2 = None

      if not myfalse:
          # here Cython uses __Pyx_XDECREF_SET
          myobject = None

      print "Made it past myobject assignment"

      if not myfalse or myobject2 is None:
          # here Cython uses Pyx_DECREF_SET because it has determined
          # that __pyx_v_myobject2 can't be null, but it really, really can!
          # (if no one assigned myobject2 yet)
          myobject2 = None

      print "Made it past myobject2 assignment"

myfunc()
#<<<END tmpnone.py>>>

#<<BEGIN setup.py>>>
#<from distutils.core import setup
from Cython.Build import cythonize

setup(
      ext_modules = cythonize("tmpnone.py")
)
#<<END setup.py>>>

[prompt]$ python setup.py build_ext --inplace
[prompt]$ ls tmpnone.*
tmpnone.c  tmpnone.py  tmpnone.so
[prompt]$ python
Python 2.6.6 (r266:84292, Nov 21 2013, 12:39:37)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
  >>> import tmpnone
Made it past myobject assignment
Segmentation fault

Julian Taylor | 22 Jan 00:03 2014

[Cython] bytearray tests fail with default unsigned char

hi,
the bytearray tests are broken when chars are unsigned.

tests/run/bytearraymethods.pyx defines following function:
def bytearray_append(b, char c, int i, object o):

this gets converted to an effective __Pyx_PyInt_AsUnsignedChar which
then errors out when -1 is passed in.
chars are unsigned like they are by default on arm, s390x and powerpc.
This causes a couple build failures in debian:
https://buildd.debian.org/status/package.php?p=cython
https://buildd.debian.org/status/fetch.php?pkg=cython&arch=armel&ver=0.20-1&stamp=1390316252

the tests can be fixed by adding signed char to the interface.

to reproduce on x86 with gcc (note the -funsigned-char to change the
default):

cython tests/run/bytearraymethods.pyx
gcc -funsigned-char bytearraymethods.c -fPIC $(python-config --includes)
$(python-config --libs) -shared -O2 -o bytearraymethods.so

python -c "import doctest; import bytearraymethods;
doctest.testmod(bytearraymethods)"

Traceback (most recent call last):
  File "<string>", line 1, in <module>
NameError: name 'bytearraymethods' is not defined
root <at> ubuntu:/# python -c "import doctest; import bytearraymethods;
doctest.testmod(bytearraymethods)"
**********************************************************************
File "bytearraymethods.so", line ?, in
bytearraymethods.__test__.bytearray_append (line 202)
Failed example:
    b = bytearray_append(b, -1, ord('y'), ord('z'))  # doctest: +ELLIPSIS
Expected:
    Traceback (most recent call last):
    ValueError: ...
Got:
    Traceback (most recent call last):
      File "/usr/lib/python2.7/doctest.py", line 1315, in __run
        compileflags, 1) in test.globs
      File "<doctest bytearraymethods.__test__.bytearray_append (line
202)[14]>", line 1, in <module>
        b = bytearray_append(b, -1, ord('y'), ord('z'))  # doctest:
+ELLIPSIS
      File "bytearraymethods.pyx", line 202, in
bytearraymethods.bytearray_append (bytearraymethods.c:1339)
        def bytearray_append(bytearray b, char c, int i, object o):
    OverflowError: can't convert negative value to char
**********************************************************************

Andriy Kornatskyy | 19 Jan 21:00 2014

[Cython] Compiler crash in RemoveUnreachableCode

The cython compiler crash report below. Steps to reproduce:

1. virtualenv env
2. env/bin/easy_install cython
3. env/bin/easy_install wheezy.http

The wheezy.http has dependency on wheezy.core. If I install those two packages separately there is no
error, only if through dependencies.

Thanks.

Andriy Kornatskyy

  File "Visitor.py", line 178, in Cython.Compiler.Visitor.TreeVisitor._visit (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:4437)
  File "Visitor.py", line 137, in Cython.Compiler.Visitor.TreeVisitor._raise_compiler_error (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:3655)
Cython.Compiler.Errors.CompilerCrash:
Error compiling Cython file:
------------------------------------------------------------
...

"""
"""

__version__ = '0.1.129'
             ^
------------------------------------------------------------

src/wheezy/core/__init__.py:5:14: Compiler crash in RemoveUnreachableCode

ModuleNode.body = StatListNode(__init__.py:5:14)

Compiler crash traceback from this point on:
  File "Visitor.py", line 170, in Cython.Compiler.Visitor.TreeVisitor._visit (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:4275)
  File
“/.../env/lib/python2.7/site-packages/Cython-0.20-py2.7-macosx-10.9-x86_64.egg/Cython/Compiler/ParseTreeTransforms.py",
line 2135, in visit_StatListNode
    if not self.current_directives['remove_unreachable']:
TypeError: 'NoneType' object has no attribute '__getitem__'
Robert Byrnes | 16 Jan 18:43 2014
Picon

[Cython] assignment to reference argument bug, revisited

Are there any plans to fix the "Assignment to reference" bug mentioned here?

https://groups.google.com/forum/?fromgroups=#!topic/cython-users/j58Sp3QMrD4

> Oh, yes, of course. Good old C++, where local T& x is unassignable but
> assigning to an argument T& x is a common idiom. This is a bug in Cython.

This is still broken in 0.19.2, and a casual inspection of 0.20rc1
suggests that it is broken there as well.

Is the fix as simple as this?

--- ExprNodes.py~       2014-01-16 12:31:08.377573000 -0500
+++ ExprNodes.py        2014-01-16 12:30:56.945501000 -0500
 <at>  <at>  -1605,7 +1605,7  <at>  <at>  class NameNode(AtomicExprNode):

         if self.type.is_const:
             error(self.pos, "Assignment to const '%s'" % self.name)
-        if self.type.is_reference:
+        if self.type.is_reference and not self.entry.is_arg:
             error(self.pos, "Assignment to reference '%s'" % self.name)
         if not self.is_lvalue():
             error(self.pos, "Assignment to non-lvalue '%s'"

This seems to work for me, but ...

I wonder if it is also correct to set self.entry.cf_unused = True
for the case of assignment to reference arguments, to avoid CYTHON_UNUSED
annotations in the function definitions and unused variable warnings.

--
Bob


Gmane