Lisandro Dalcin | 21 Apr 13:53 2014
Picon

[Cython] MPI_Type_create_hindexed_block() segfaults

I believe the problem is in the following source code line
(file:ompi_datatype_args.c, line:221):

https://bitbucket.org/ompiteam/ompi-svn-mirror/src/v1.8/ompi/datatype/ompi_datatype_args.c?at=v1.8#cl-221

I think you should just remove that bogus line, and that's all.

[dalcinl <at> kw2060 openmpi]$ cat type_hindexed_block.c
#include <mpi.h>
int main(int argc, char *argv[])
{
  MPI_Aint disps[] = {0};
  MPI_Datatype datatype;
  MPI_Init(&argc, &argv);
  MPI_Type_create_hindexed_block(1, 1, disps, MPI_BYTE, &datatype);
  MPI_Finalize();
  return 0;
}
[dalcinl <at> kw2060 openmpi]$ mpicc type_hindexed_block.c
[dalcinl <at> kw2060 openmpi]$ ./a.out
[kw2060:20304] *** Process received signal ***
[kw2060:20304] Signal: Segmentation fault (11)
[kw2060:20304] Signal code: Address not mapped (1)
[kw2060:20304] Failing at address: 0x6a
[kw2060:20304] [ 0] /lib64/libpthread.so.0[0x327c40f750]
[kw2060:20304] [ 1] /lib64/libc.so.6[0x327bc94126]
[kw2060:20304] [ 2]
/home/devel/mpi/openmpi/1.8.0/lib/libmpi.so.1(ompi_datatype_set_args+0x7f1)[0x7f8f0158b62a]
[kw2060:20304] [ 3]
/home/devel/mpi/openmpi/1.8.0/lib/libmpi.so.1(MPI_Type_create_hindexed_block+0x24d)[0x7f8f015cedc8]
(Continue reading)

Martín Gaitán | 4 Apr 19:55 2014
Picon

[Cython] Move IPython's cython extension to cython

As you probably know, IPython has a "cython magic" [1], an extension to interface ipython with cython, allowing to use cython code in the interactive session.

Originally, few of these "language specific magics" (R, cython, octave, etc) were included builtin in IPython, but then the IPython core developers decided break out them (with fair arguments IMHO [2]), either to be included in the "language package" they depend on or as a third party extensions.

At the moment,  cython-magic is the only one language specific extension left to move.
The rest were moved to the packages they depend on [3]

So, this message is to ask if the Cython project would like to import this code, task that  Brian Grangrer (the author of the cython-magic extension and IPython core dev)  entrusted me [3].

Of course, I could help you preparing a PR that migrate the module, its tests and documentation, preserving the git history as far as possible.

let me know your opinion/decision




--
<div><div dir="ltr">
<div class="gmail_quote">
<div dir="ltr">
<div>As you probably know, IPython has a "cython magic" [1], an extension to interface ipython with cython, allowing to use cython code in the interactive session. <br><br>
</div>
<div>Originally, few of these "language specific magics" (R, cython, octave, etc) were included builtin in IPython, but then the IPython core developers decided break out them (with fair arguments IMHO [2]), <span lang="en"><span>either to be included </span></span>in the "language package" they depend on or as a third party extensions. <br><br>At the moment,&nbsp; cython-magic is the only one language specific extension left to move. <br>The rest were moved to the packages they depend on [3]<br>
</div>
<div><br></div>
<div>So, this message is to ask if the Cython project would like to import this code, task that&nbsp; Brian Grangrer (the author of the cython-magic extension and IPython core dev)&nbsp; entrusted me [3]. <br><br>Of course, I could help you preparing a PR that migrate the module, its tests and documentation, preserving the git history as far as possible. <br><br>
</div>
<div>let me know your opinion/decision <br>
</div>
<div><br></div>
<div><div><div>
<div></div>
<div>
<div>cheers<br>
</div>
<div>Martin. <br>
</div>
<div>
<br><br>[1] <a href="http://ipython.org/ipython-doc/dev/config/extensions/cythonmagic.html" target="_blank">http://ipython.org/ipython-doc/dev/config/extensions/cythonmagic.html</a><br>

[2] <a href="https://github.com/ipython/ipython/issues/3150" target="_blank">https://github.com/ipython/ipython/issues/3150</a> <br>[3] <a href="https://github.com/ipython/ipython/issues/3803" target="_blank">https://github.com/ipython/ipython/issues/3803</a><br>

[4] <a href="https://github.com/ipython/ipython/issues/3150#issuecomment-37203217" target="_blank">https://github.com/ipython/ipython/issues/3150#issuecomment-37203217</a><span class="HOEnZb"><br><br clear="all"><div>
<br>-- <br><div dir="ltr">
<a href="http://mgaitan.github.io" target="_blank">mgaitan.github.io</a><br><a href="http://textosyprextextos.com.ar" target="_blank">textosyprextextos.com.ar</a>
</div>
</div></span>
</div>
</div>
</div></div></div>
</div>
</div>
<br><br clear="all"><br>-- <br><div dir="ltr">
<a href="http://mgaitan.github.io" target="_blank">mgaitan.github.io</a><br><a href="http://textosyprextextos.com.ar" target="_blank">textosyprextextos.com.ar</a>
</div>

</div></div>
Syam Gadde | 27 Mar 21:29 2014

[Cython] memoryview slice transpose

I wanted to re-offer for discussion a fix for a bug I reported a while 
back.  The code that breaks is this:

import numpy
cimport numpy

def testfunc():
    a = numpy.arange(12).reshape([3,4])
    cdef numpy.int_t[:,:] a_view = a
    cdef numpy.int_t[:,:] b = a_view[:,:].T

One problem is that when a memoryview slice is decremented, the .memview 
and .data fields are not set to NULL, and any error that causes a jump 
to cleanup code will attempt to decref it again.

The second problem (I think) is that assignment of a transpose of 
memoryview slices to a variable is not resulting in an incref on the 
underlying slice, even though the variable is not a temporary and 
continues to persist.

Here is a patch that solves the problem for me.

https://github.com/SyamGadde/cython/commit/5739d8b908f18c4fc9103ef04e39964d61af3495

The part I am least sure about is removing the "if" qualifications on 
the incref and decref:

   if self.obj.is_name  or self.obj.is_attribute and self.obj.is_memslice_transpose:

that was obviously there for a reason, but it causes problems in the above case.  If there is a less extreme
solution I'd be happy to take it.

Thanks,

-syam
Pauli Virtanen | 18 Mar 23:56 2014
Picon
Picon

[Cython] Fused type signature resolution failures

Hi,

Here are the fused types + memoryview signature resolution failures I
promised earlier (Cython 0.20.1):

(A)

TypeError: No matching signature

----------------------------- asd.pyx
cimport numpy as cnp

ctypedef fused value_t:
    cnp.float32_t
    cnp.float64_t

cpdef foo(value_t x):
    pass
----------------------------- quux.py
import numpy as np
import asd
asd.foo(np.float32(1.0))
-----------------------------

(B)

ValueError: Buffer dtype mismatch, expected 'int64_t' but got 'double'

----------------------------- asd.pyx
cimport numpy as cnp

ctypedef fused idx_t:
    cnp.int32_t
    cnp.int64_t

ctypedef fused value_t:
    cnp.int64_t
    cnp.float64_t

cpdef foo(idx_t[:,:] i, idx_t[:,:] j, value_t[:,:] x):
    pass
----------------------------- quux.py
import numpy as np
import asd
i = np.zeros((3, 3), np.int64)
j = np.zeros((3, 3), np.int64)
x = np.zeros((3, 3), np.float64)
asd.foo(i, j, x)
-----------------------------

(C)

Then some nasty platform-dependent failures:

https://github.com/scipy/scipy/issues/3461

The relevant code is:

https://github.com/scipy/scipy/blob/master/scipy/sparse/_csparsetools.pyx#L202

The code looks nothing special. However, call to `lil_fancy_get` fails
with "TypeError: No matching signature found" when the inputs have types

<class 'int'> <class 'int'> <class 'numpy.ndarray'>
<class 'numpy.ndarray'> <class 'numpy.ndarray'> <class 'numpy.ndarray'>
<class 'numpy.ndarray'> <class 'numpy.ndarray'>

with ndarray dtypes: object object object object int32 int32

The failure occurs only on Christoph Gohlke's Win64 build, but not on
Linux/OSX/MINGW32. This sounds like some integer size combination
issue, but I'm far from sure.

Unfortunately, I'm not easily able to isolate/understand what's going
wrong here.

--

-- 
Pauli Virtane

Pauli Virtanen | 8 Mar 21:11 2014
Picon
Picon

[Cython] Too many instantiations with fused type memoryviews

Hi,

FYI: Cython instantiates fused type routines with memoryview arguments
unnecessarily many times.

Example:
```
ctypedef fused idx_t:
    short
    long

# Cython 0.20.1 instantiates this function 128 times,
# even though only 2 would be needed
def fubar_128(idx_t[:] a,
              idx_t[:] b,
              idx_t[:] c,
              idx_t[:] d,
              idx_t[:] e,
              idx_t[:] f,
              idx_t[:] g):
    print("HALLO")

def fubar_2(idx_t a,
            idx_t b,
            idx_t c,
            idx_t d,
            idx_t e,
            idx_t f,
            idx_t g):
    print("HULLO")
```
$ cython cymod.pyx
$ cython --version
Cython version 0.20.1
$ grep -c 'print("HALLO")' cymod.c
128
$ grep -c 'print("HULLO")' _cymod.c
2

The n**m explosion starts to hurt quite quickly when there are several
array arguments and more than one fused type. I think this issue is
also accompanied by some signature resolution bugs (I'll try to come
up with an example case).

Cheers,

--

-- 
Pauli Virtanen

Stefan Behnel | 3 Mar 17:38 2014
Picon

[Cython] Crash with freelist() and __slots__

Hi,

I was made aware of crashes in the last lxml release, which turned out to
be due to the use of freelists for types that could be subtyped from Python
code. I was able to work around them in lxml, however, the real problem is
in Cython. There was supposed to be a safe guard for that case in the
freelist code based on the object struct size, which increases for subtypes
that have a __dict__. However, if the Python subtype uses an empty
__slots__ declaration, the object struct size will not increase, thus
passing the guard.

The correct fix is to also test if the type being instantiated lives on the
heap and exclude it from the freelist if so.

https://github.com/cython/cython/commit/d2aff824dd0900982a420ebaaa1dac6120a9d72e

This, together with the buffer/memory view related bugs I fixed since the
last release, suggest (at least to me) that we shouldn't wait all too long
with the next bug fix release.

From my POV, all changes in current master are safe enough to go out.

Stefan
Syam Gadde | 27 Feb 16:22 2014

[Cython] line tracing/profiling code objects

Hi all,

I tried using line tracing in conjunction with line_profiler/kernprof to 
attempt to see if line-based profiling works in Cython.  I think it's 
pretty close.  But I think the problem is that line_profiler is getting 
a different PyCodeObject when it wraps the function and when it actually 
gets the trace call.  It adds the initial code object to a map, and 
later when it gets the trace call, decides that the trace call is not 
something it needs to pay attention to because the new code object that 
it gets is not the same as the original one.

The first code object is created at function declaration by 
__Pyx_PyCode_New (called in__Pyx_InitCachedConstants) and is assigned to 
a variable __pyx_k_codeobj_NNN.  The second code object is created, 
essentially, during the first entry into the function (in 
__Pyx_TraceCall, via __Pyx_TraceSetupAndCall).  It seems that setting 
__pyx_frame_code to the initial code object before calling TraceCall() 
would fix this.

Is this easy to do?  I'd do it myself, but I'd need help figuring out 
how to get the name of the appropriate __pyx_k_codeobj_NNN variable from 
within FuncDefNode.generate_function_definitions(), which calls 
put_trace_call().

Thanks for your help...

-syam
Stefan Behnel | 22 Feb 10:02 2014
Picon

[Cython] 0.20.2

Hi,

there are a couple of nice additions and fixes in the current master that
could go out as a 0.20.2 some time soon. I suggest waiting another couple
of days for more bug reports to come in and then just send it out.

I'd like to also fix the buffer formatting bug by then, looks like I got a
hold of it.

Stefan
Gael Varoquaux | 17 Feb 12:46 2014

[Cython] Invalid C++ with Python and C++ iterators (not compiling CLang)

Hi,

I'd like to enquire on the status of the problem raised last year:
https://mail.python.org/pipermail/cython-devel/2013-April/003629.html

To summarize, the following Cython leads to code that doesn't compile
with Clang and is probably invalid C++:

    from libcpp.vector cimport vector as libcpp_vector
    from cython.operator cimport dereference as deref, preincrement as inc

    cdef class TestClass:

	cdef libcpp_vector[float] inst

	def __iter__(self):
	    it = self.inst.begin()
	    while it != self.inst.end():
		yield deref(it)
		inc(it)

This create the following C++ code:

  p->__pyx_v_it.std::vector<float>::iterator::~iterator();

which is invalid, but happens to work on GCC and MSVC. However, Clang
support is starting to become very important, as it is the default
compiler on MacOSX.

The original reporter suggested a valid C++ code (see original email),
which wasn't very convenient as it required a typedef.

Any progress on the issue or suggestion to work around?

Cheers,

Gaël
Joshua Adelman | 6 Feb 16:21 2014
Picon

[Cython] Bug: Memoryview of struct with adjacent string fields does not detect string boundaries

This discussion was initially started on the cython user google group, but I wanted to move the issue over to
the dev list, as per the suggestion on cython_trac. 

Given a numpy recarray containing two or more fixed-length string fields, if those string fields are
adjacent to one another cython does not properly detect the boundary between the string fields. A concise
test case demonstrating the problem is:

```cython
cimport numpy as np

cdef packed struct tstruct:
    np.float32_t a
    np.int16_t b
    char[6] c
    char[4] d

def test_struct(tstruct[:] x):
    pass
```

We then define some data on the python side:

```python
import numpy as np

a = np.recarray(3, dtype=[('a', np.float32),  ('b', np.int16), ('c', '|S6'), ('d', '|S4')])
a[0] = (1.1, 1, 'abcde', 'fgh')
a[1] = (2.1, 2, 'ijklm', 'nop')
a[2] = (3.1, 3, 'qrstu', 'vwx')

test_struct(a)
```

This results in the error:

---------------------------------------------------------------------------
ValueError       Traceback (most recent call last)

<ipython-input-12-ac01118a36a7> in <module>()
----> 1 test_struct(a)

ValueError: Expected a dimension of size 6, got 10

If we swap the order of the fields in the recarray and `tstruct` to (a,c,b,d) so that there is a numerical
field between the string fields, then the function can parse the memory view correctly. 

The relevant line of code that catches the incorrect value of `enc_count` is: 
https://github.com/cython/cython/blob/master/Cython/Utility/Buffer.c#L468 

``` 
if (ctx->enc_count != ctx->head->field->type->arraysize[0]) { 
            PyErr_Format(PyExc_ValueError, 
                         "Expected a dimension of size %zu, got %zu", 
                         ctx->head->field->type->arraysize[0], ctx->enc_count); 
            return -1; 
        } 
``` 

My naive guess is that there is something going on in: 
https://github.com/cython/cython/blob/master/Cython/Utility/Buffer.c#L738 

since that appears to be the only place where `enc_count` is being incremented. That would seem like the
place where a boundary between two string fields might not be properly handled (the comment in the line
above "Continue pooling same type" is suggestive.

I'll cross-post this on the cython trac once I have access and will then submit a pull request on Github of a
test case once I have the trac issue number.

Josh

Stefan Behnel | 5 Feb 18:00 2014
Picon

[Cython] CPython ticket on allowing non-ASCII names for extension modules

There's a ticket about allowing extension modules to have non-ASCII names.

http://bugs.python.org/issue20485

Stefan

Gmane