david.froger.ml | 29 Jan 18:36 2016

cython, waf and development mode

Hello,

I'm trying to improve the development cycle of a `.py` and `.pyx` code. 
The
code is built with `waf,` which has several advantages:
   - configure step to check for dependencies, provide build options
   - manage debug, release, profile configurations
   - fast parallel builds, with pretty output
   - compiler errors well reported (not the long Python traceback of
     `pyximport`)

But my problem is that this splits the code in two parts during 
development:
   - `.py` files are in source directory
   - `.so` files (compiled from `.pyx` files) are in `waf` build 
directory

So it's hard to import the package in development mode, ie without 
installing
it (which would put together `.py` and `.so` files).

Here is a simple Cython package for example:

     foo/
         __init__.py # Contains:  from . import bar
         bar.pyx

I use `waf` to build the Python extension from `bar.pyx`:

     foo/
(Continue reading)

Alexander Kain | 28 Jan 21:16 2016
Picon

inlining C code in .pyx files

Right now, to call an external function, we do something like this:

cimport filtering_c
def lfilter_c(np.ndarray[DTYPE_t, ndim=1] b, 
               np.ndarray[DTYPE_t, ndim=1] a, 
               np.ndarray[DTYPE_t, ndim=1] y,
               np.ndarray[DTYPE_t, ndim=1] zb,
               np.ndarray[DTYPE_t, ndim=1] za):
    filtering_c.lfilter(<double *>b.data, b.shape[0],
                        <double *>a.data, a.shape[0],
                        <double *>y.data, y.shape[0],
                        <double *>zb.data,
                        <double *>za.data)

This requires the C source file filtering_c.c, header file filter_c.h, and pdx file filtering_c.pxd.

I am wondering whether it would be possible to do something like this:

cdef void fun(double *b, unsigned Nb,
             double *a, unsigned Na,
             double *y, unsigned Ny,
             double *zb, double *za):
    cython.inline("""
    {
      unsigned i;
      for(i=0; i<Ny; i++) {
        double *bp = b+1;
        double *ap = a+1;
        double acc = 0;
        unsigned j;
        for(j=1; j<Nb; j++)
          acc += (*bp++) * (*zb++);
          ...
      }
    }
    """)
                        
def lfilter_c_local(np.ndarray[DTYPE_t, ndim=1] b, 
               np.ndarray[DTYPE_t, ndim=1] a, 
               np.ndarray[DTYPE_t, ndim=1] y,
               np.ndarray[DTYPE_t, ndim=1] zb,
               np.ndarray[DTYPE_t, ndim=1] za):
    fun(<double *>b.data, b.shape[0],
                        <double *>a.data, a.shape[0],
                        <double *>y.data, y.shape[0],
                        <double *>zb.data,
                        <double *>za.data)

In other words, could there be a way to inline C-code directly in .pyx files?

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
smbat.makiyan | 26 Jan 11:41 2016

operator & is called earlier than constructor

Hi All,

Few month ago you were fixing a bug with overridden operator& while deleting the object. Thank you very much for this, the fix works well. Unfortunately we hit similar issue again, this time when constructing the object.
Let me describe the problem once again. On the project I am working for we are using technology that is very close to Microsoft COM. There we use a smart pointer class for reference counting which is almost identical to well known CComPtr.

Unfortunately it brought us a big annoyance - overridden operator&

template <typename T>
class MyPtr
{
private:
    /// Current referenced interface
    T * _reference;

public:
 /// Allows passing this smart ptr as argument to methods which expect
 /// a T or a void, in order to fill it in. (e.g. QueryInterface). NOTE: if the internal
 /// reference is non-NULL it will be Released.
 /// to this object.
 /// \deprecated Please use the outArg() method (or inOutArg()) instead. 
    T** operator& ()
 {
 if (_reference)
 {
  _reference->Release()
  _reference = NULL;
 }
 return &_reference;
 }

... 

Well, it is deprecated but it still there. The issue is that operator&() returns not the object's address, but address of its internal member

We are using it in cython objects like this

cdef class MyClass:
 cdef MyPtr[IInterface] obj

Cython generates the following code for creation an object:

static PyObject* __pyx_tp_new_8MyModule_MyClass(PyTypeObject* t, PyObject* a, PyObject* k) {
  struct __pyx_obj_8MyModule_MyClass *p;
  new((void*)&(p->obj)) MyPtr < IInterface > ();
}

The issue is that operator & is called earlier than constructor. Luckily this works for most of the cases, someone (tp_new?) clears the memory, Smart Pointer’s internal reference gets NULL, so Release() is not called. One more lucky point is that pointer’s constructor does no work, so even when placement new() gets wrong address nothing breaks. 
Unfortunately when we tried to integrate new smart pointer version we got number of crashes. Internal structure of the pointer has changed and calling operator& before constructor now fails.
Is there any way to tweak tp_new function generation so that if calculates objects address in some other way without calling operator&()?

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Bingo | 26 Jan 09:23 2016
Picon

Cython3 losed define DLIMPROT in xx.h file

When I run "cython3 xxx.pyx",generate xxx.h, xx.c

I had met same problem mentioned in https://groups.google.com/forum/#!topic/cython-users/DQ2thclt3yI

I found if we added
#ifndef DL_IMPORT
  #define DL_IMPORT(t) t
#endif

in xx.c originally 
then, we could compile successfully!

But I think it's inconvenient. Could I solve this problem without adding those statement after I had ran cython3?

Any answer will be appreciate.  

Thanks.

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Antony Lee | 25 Jan 08:24 2016
Picon

Suggestion: allow conditional distutils directives

Something like

IF UNAME_SYSNAME == "Linux":
    #distutils: define_macros = FOOBAR=42
    pass # to avoid completely confusing the parser, this may help.
ELSE:
    #distutils: define_macros = FOOBAR=123
    pass

The goal being mostly to pass different compilation options/link options depending on the platform, without having to put these options in setup.py.

Thoughts?

Antony

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Marvin | 24 Jan 23:50 2016
Picon

Bug with Typed Memoryview and unicode string type?

Hi,

I'm using Cython 0.23.4 with Python 3.5 and run into a problem with typed memoryview. I'm compiling with the c_string_type compiler directive set to 'unicode' and creating a memory view from an int*. Even so the code compiles it will raise a TypeError('expected bytes, str found') at runtime time. With c_string_type set to 'bytes' everything is fine. Is this a bug in Cython or am I doing something wrong?

I attached a small example. To run it you can run test.sh

Best
Marvin

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Attachment (cython-string_type.zip): application/zip, 1945 bytes
Konstantin Bolshakov | 21 Jan 22:57 2016
Picon

SystemError: Parent module '' not loaded, cannot perform relative import

Hello.
Everything worked, where some two weeks ago my module began build with error.
OS Windows 7 x64. Python 3.4.3.
setup.py:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize

extensions = [
    Extension("program1", ["program1.py"]),
]
setup(
    name="program1",
    ext_modules=cythonize(extensions),
)

program1.py
# -*- coding: utf-8 -*-

import os
import sys
import time
import random
# etc...

C:\Python34\python.exe setup1.py build_ext --inplace
# build without error.

> python
Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:43:06) [MSC v.1600 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import  program1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "test\program1.py", line 3, in init test.program1 (program1.c:83009)
    import os
SystemError: Parent module '' not loaded, cannot perform relative import

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Isaac Ge | 22 Jan 09:17 2016
Picon
Gravatar

What is the C++ operator& function in Python syntax?

I check the http://docs.cython.org/src/userguide/wrapping_CPlusPlus.html#c-operators-not-compatible-with-python-syntax

But it only mentions dereference and preincrement, I can not find out what is the C++ operator& called in Python syntax.

Thank you!

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Yury V. Zaytsev | 19 Jan 16:39 2016

Temporaries cause broken code for std::make_unique<>(), any hints?

Hi,

I've implemented a wrapper for std::make_unique<>() as follows:

     from libcpp.memory cimport unique_ptr

     cdef extern from "<memory>" namespace "std" nogil:
         unique_ptr[T] make_unique[T]() except +
         unique_ptr[T] make_unique[T](T&) except +

Very unfortunately, this forces Cython to generate a temporary for 
handling the exceptions, which, obviously breaks the whole thing:

     try {
       __pyx_t_10 = std::make_unique<XXX>(__pyx_v_uv);
     } catch(...) {
       __Pyx_CppExn2PyErr(); /* ... */
     }
     __pyx_v_pyuv->thisptr = __pyx_t_10;

When I remove "except +", then the generated code is correct, but, of 
course, possible exceptions (std::bad_alloc or any exception thrown by the 
constructor of T) are not handled.

Is it somehow possible to force Cython to generate a move, as in

     __pyx_v_pyuv->thisptr = std::move(__pyx_t_10);

Maybe it actually makes sense to always generate a move if the compiler 
supports it? I can't see any obvious disadvantages to not doing so, at 
least for the specific case of exception handling temporaries...

Thanks for considering this!

-- 
Sincerely yours,
Yury V. Zaytsev

--

-- 

--- 
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Yury V. Zaytsev | 19 Jan 14:01 2016

Possible memory leak in support for memoryviews

Hi,

I'm trying to check my Cython modules with LeakSanitizer, and I'm getting 
the following report for every module that uses memoryviews:

Direct leak of 3072 byte(s) in 1 object(s) allocated from:
     #0 0x7fd58f0a75b1 in __interceptor_calloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x945b1)
     #1 0x458543 in dictresize Objects/dictobject.c:643
     #2 0x7fd58ac83e8d in initleaktest2 /home/zaytsev/src/cython/leak/leaktest2.cpp:14131
     #3 0x4ced57 in _PyImport_LoadDynamicModule Python/importdl.c:53

The correspondent line from leaktest2.cpp:14131 looks as follows:

     if (__Pyx_SetVtable(__pyx_type___pyx_memoryview.tp_dict, __pyx_vtabptr_memoryview) < 0) { /* ...
*/ }

I attach the reproducer:

     make

     python -c "import leaktest1; leaktest1.main()"
     # no leak report that looks like the above

     python -c "import leaktest2; leaktest2.main()"
     # leak report that looks like the above

Unfortunately, your Python should be linked with -fsanitize=address, or 
else you can try LD_PRELOAD'ing ASan, but I've faced so many issues along 
this path that I don't recommend it.

Additionally, you need GCC 5 to compile in order to use the latest 
available version of sanitizers for GCC.

I would appreciate hints as to whether this is a genuine leak, and in how 
far this can be avoided at the level of Cython.

Many thanks!

-- 
Sincerely yours,
Yury V. Zaytsev

--

-- 

--- 
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Attachment (reproducer.tar.xz): application/x-xz, 830 bytes
remi.cambuzat | 19 Jan 04:52 2016
Picon

Import .pxd object into anoter .pxd file

Good afternoon.

I'm facing an include problem that drive me mad.

As you will see here in my file Arm.pxd, I'm trying to use two object ArmParameters and MusclesParameters, that are also cython classes from the package ArmModel.

However even if the compilation of the package ArmModel is going without error (compiler script here : Cythonize_ArmModel.py).
I got some strange error when I'm running my test script runTest.py.
Here is the log :

python runTest.py 
 
--------------Muscles---------------
init MusclesParameters
--------------ArmParam---------------
init ArmParameters
--------------Arm---------------
(array([0, 1, 2, 3, 4, 5]), array([ 0, 11, 22, 33]))
     ---------Arm cython---------
Traceback (most recent call last):
  File "runTest.py", line 28, in <module>
    from ArmModel import Arm as Arm
  File "MotorControlModel/ArmModel/MusclesParameters.pxd", line 4, in init MotorControlModel.ArmModel.Arm (MotorControlModel/ArmModel/Arm.c:8141)
ImportError: No module named MotorControlModel.ArmModel.MusclesParameters

The strange thing is that I have my all of the files I need in my package ArmModel. My files MusclesParameters.pxd,ArmParameters.pxd and Arm.pxd are also inside it with their respective .c and.so files.
So I really don't understand why I have this problem.

I read a lot of documentation on Sharing Extension Types and implemented their solution and took into account this post.
Unfortunately I still get the error.

Could you help me on this before It mate me blow a fuse ^^.

If the answer is on the web could you link me to it, as I wasn't able to find something relevant with google.

Many Thanks <3 :D








--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Gmane