Syam Gadde | 27 Feb 16:22 2014

[Cython] line tracing/profiling code objects

Hi all,

I tried using line tracing in conjunction with line_profiler/kernprof to 
attempt to see if line-based profiling works in Cython.  I think it's 
pretty close.  But I think the problem is that line_profiler is getting 
a different PyCodeObject when it wraps the function and when it actually 
gets the trace call.  It adds the initial code object to a map, and 
later when it gets the trace call, decides that the trace call is not 
something it needs to pay attention to because the new code object that 
it gets is not the same as the original one.

The first code object is created at function declaration by 
__Pyx_PyCode_New (called in__Pyx_InitCachedConstants) and is assigned to 
a variable __pyx_k_codeobj_NNN.  The second code object is created, 
essentially, during the first entry into the function (in 
__Pyx_TraceCall, via __Pyx_TraceSetupAndCall).  It seems that setting 
__pyx_frame_code to the initial code object before calling TraceCall() 
would fix this.

Is this easy to do?  I'd do it myself, but I'd need help figuring out 
how to get the name of the appropriate __pyx_k_codeobj_NNN variable from 
within FuncDefNode.generate_function_definitions(), which calls 
put_trace_call().

Thanks for your help...

-syam
Stefan Behnel | 22 Feb 10:02 2014
Picon

[Cython] 0.20.2

Hi,

there are a couple of nice additions and fixes in the current master that
could go out as a 0.20.2 some time soon. I suggest waiting another couple
of days for more bug reports to come in and then just send it out.

I'd like to also fix the buffer formatting bug by then, looks like I got a
hold of it.

Stefan
Gael Varoquaux | 17 Feb 12:46 2014

[Cython] Invalid C++ with Python and C++ iterators (not compiling CLang)

Hi,

I'd like to enquire on the status of the problem raised last year:
https://mail.python.org/pipermail/cython-devel/2013-April/003629.html

To summarize, the following Cython leads to code that doesn't compile
with Clang and is probably invalid C++:

    from libcpp.vector cimport vector as libcpp_vector
    from cython.operator cimport dereference as deref, preincrement as inc

    cdef class TestClass:

	cdef libcpp_vector[float] inst

	def __iter__(self):
	    it = self.inst.begin()
	    while it != self.inst.end():
		yield deref(it)
		inc(it)

This create the following C++ code:

  p->__pyx_v_it.std::vector<float>::iterator::~iterator();

which is invalid, but happens to work on GCC and MSVC. However, Clang
support is starting to become very important, as it is the default
compiler on MacOSX.

The original reporter suggested a valid C++ code (see original email),
(Continue reading)

Joshua Adelman | 6 Feb 16:21 2014
Picon

[Cython] Bug: Memoryview of struct with adjacent string fields does not detect string boundaries

This discussion was initially started on the cython user google group, but I wanted to move the issue over to
the dev list, as per the suggestion on cython_trac. 

Given a numpy recarray containing two or more fixed-length string fields, if those string fields are
adjacent to one another cython does not properly detect the boundary between the string fields. A concise
test case demonstrating the problem is:

```cython
cimport numpy as np

cdef packed struct tstruct:
    np.float32_t a
    np.int16_t b
    char[6] c
    char[4] d

def test_struct(tstruct[:] x):
    pass
```

We then define some data on the python side:

```python
import numpy as np

a = np.recarray(3, dtype=[('a', np.float32),  ('b', np.int16), ('c', '|S6'), ('d', '|S4')])
a[0] = (1.1, 1, 'abcde', 'fgh')
a[1] = (2.1, 2, 'ijklm', 'nop')
a[2] = (3.1, 3, 'qrstu', 'vwx')

(Continue reading)

Stefan Behnel | 5 Feb 18:00 2014
Picon

[Cython] CPython ticket on allowing non-ASCII names for extension modules

There's a ticket about allowing extension modules to have non-ASCII names.

http://bugs.python.org/issue20485

Stefan
Stefan Behnel | 2 Feb 18:13 2014
Picon

Re: [Cython] Bug: _PyType_Lookup shortcut in Cython 0.20 breaks lookup of __exit__ in Python 2.6

[forwarding to cython-devel]

Wouter Bolsterlee, 02.02.2014 16:40:
> Cython 0.20 introduced __Pyx_PyObject_LookupSpecial as a shortcut to
> lookup special methods. The commit that introduced this change is
> 8f6412275c4c2d1dcf43ad40306f858b114104ed and can be seen here:
> 
> https://github.com/cython/cython/commit/8f6412275c4c2d1dcf43ad40306f858b114104ed
>         
> It seems this change does not work with Python 2.6. The newly introduced
> function calls _PyType_Lookup(), which fails to lookup (at least)
> __exit__ (on a threading.Lock) instance when run under Python 2.6. The
> exact call is here:
> 
> https://github.com/cython/cython/commit/8f6412275c4c2d1dcf43ad40306f858b114104ed#diff-83083bdce5dcca284394ead83b8d3f99R1053
>         
> For my Plyvel project (https://plyvel.readthedocs.org/ and
> https://github.com/wbolster/plyvel) this issue manifests as follows:
> 
> $ cython --version
>         Cython version 0.20
>         
>         $ python 
>         Python 2.6.8 (unknown, Jan 26 2013, 14:35:25) 
>         [GCC 4.7.2] on linux2
>         Type "help", "copyright", "credits" or "license" for more
> information.
>         >>> import plyvel
>         >>> db = plyvel.DB('/tmp/testdb', create_if_missing=True)
>         >>> db.close()
(Continue reading)

Stefan Behnel | 31 Jan 12:55 2014
Picon

[Cython] Sage build problem

Hi,

There's some kind of C++ enum issue in the Sage build.

Generated code in sage/libs/ppl.cpp:

"""
static PyObject *__pyx_pf_4sage_4libs_3ppl_11MIP_Problem_30solve(
             struct __pyx_obj_4sage_4libs_3ppl_MIP_Problem *__pyx_v_self) {
  enum Parma_Polyhedra_Library::PPL_MIP_Problem_Status __pyx_v_tmp;
"""

gcc error:

"""
sage/libs/ppl.cpp: In function 'PyObject*
__pyx_pf_4sage_4libs_3ppl_11MIP_Problem_30solve(__pyx_obj_4sage_4libs_3ppl_MIP_Problem*)':

sage/libs/ppl.cpp:5514:33: error: 'PPL_MIP_Problem_Status' in namespace
'Parma_Polyhedra_Library' does not name a type
sage/libs/ppl.cpp:5514:67: error: invalid type in declaration before ';' token
"""

The type of "tmp" seems to be inferred. That might be a hint that it could
be a problem in Cython.

The Cython code in question is also a bit funny:

sage/libs/ppl.pyx:1076
"""
(Continue reading)

R. Andrew Ohana | 28 Jan 23:34 2014
Picon

[Cython] Broken list multiplication

It seems like eaca39b00b1451fe6c846a0a670e4c8b39935525 broke some (very simple) instances of list multiplication when there are at least 3 terms in the product. e.g. on my system I'm getting these results:

L = [None] * 2 * 3
print(len(L))
3

L = 2 * [None] * 3
print(len(L))
3

L = [None] * 2 * 3 * 5
print(len(L))
5

--
Andrew
<div><div dir="ltr">
<div>
<div>
<div>
<div>
<div>
<div>It seems like eaca39b00b1451fe6c846a0a670e4c8b39935525 broke some (very simple) instances of list multiplication when there are at least 3 terms in the product. e.g. on my system I'm getting these results:</div>

<div><br></div>L = [None] * 2 * 3<br>
</div>print(len(L))<br>3<br><br>
</div>L = 2 * [None] * 3<br>
</div>print(len(L))<br>3<br><br>
</div>L = [None] * 2 * 3 * 5<br>
</div>print(len(L))<br>5<br clear="all"><div><div><div><div>

<div><div><div>
<br>-- <br><div dir="ltr">Andrew</div>
</div></div></div>
</div></div></div></div>
</div></div>
Syam Gadde | 28 Jan 19:41 2014

[Cython] memoryview refcount error

Hi again,

I'm exploring the typed memoryview feature with numpy and encountered a 
refcount error.  It seems to happen when I create a slice of a 
memoryview, and transpose it.  If I don't do both, I don't get the 
refcount error.  It happens in the error cleanup code.  I suspect that 
any error that causes the function to jump to the error cleanup code 
will do the same thing as the Exception below -- but that seemed the 
easiest way to demonstrate the problem. However, if there truly is an 
error in my code, please let me know! Thanks,

-syam

# BEGIN CODE
import numpy
cimport numpy

class MyException(Exception):
     def __init__(self):
         Exception.__init__(self)

def testfunc():
    a = numpy.arange(12).reshape([3,4])

    cdef numpy.int_t[:,:] a_view = a

    ## here is a slice followed by a transpose
    cdef numpy.int_t[:,:] b = a_view[:,:].T
    ## same thing happens with a more realistic slicing:
    #cdef numpy.int_t[:,:] b = a_view[1:2,:].T
    ## however, there is no error if I do this instead:
    #cdef numpy.int_t[:,:] b = a_view.T
    ## also no error if I do this instead:
    #cdef numpy.int_t[:,:] b = a_view[:,:]

    # The exception below is just to force the function to abort
    # and run the extraneous __PYX_XDEC_MEMVIEW in the error
    # cleanup code:
    #   Fatal Python error: Acquisition count is 0 (line XXXX)
    # Comment out this exception and we don't get the error
    raise MyException()

try:
     testfunc()
except MyException:
     pass
# END CODE

Julian Taylor | 24 Jan 22:26 2014

[Cython] 0.20 tests fail with python3.4b2

hi,

with python3.4b2 numpy_memoryview.acquire_release_cycle fails
looking at the source the test should be disabled, but its still run.

This might be a py3.4 change to doctest, but I know nothing about
doctest, so I wanted to check first if you know something about this.
Interestingly py3.4 runs 64 tests, while python3.3 only runs 32.
the __test__ variable contains the same functions in both python
versions (acquire_release is not in it)

======================================================================
FAIL: acquire_release_cycle (numpy_memoryview)
Doctest: numpy_memoryview.acquire_release_cycle
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.4/doctest.py", line 2187, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for
numpy_memoryview.acquire_release_cycle
  File
"/tmp/buildd/cython-0.20/build/work-dir/memoryview/c/numpy_memoryview/numpy_memoryview.cpython-34m.so",
line unknown line number, in acquire_release_cycle

----------------------------------------------------------------------
File
"/tmp/buildd/cython-0.20/build/work-dir/memoryview/c/numpy_memoryview/numpy_memoryview.cpython-34m.so",
line ?, in numpy_memoryview.acquire_release_cycle
Failed example:
    acquire_release_cycle(a)
Expected:
    deallocated!
Got nothing
Stefan Behnel | 23 Jan 21:10 2014
Picon

Re: [Cython] segfault due to using DECREF instead of XDECREF

Victor Makarov, 23.01.2014 21:02:
> I'd like to fix this if you don't mind.

Oh, I certainly don't mind. :)

Stefan


Gmane