Kevin Sheppard | 11 Feb 16:32 2016
Picon

Compiler error on Python 3.x when using Docstrings defined in .pxi files

I am using some composition to build multiple modules.  The docstring is module-dependent and this seems to cause an error.  My code has the following pattern (greatly reduced):

=== file.pxi

FUNC_DOCSTRING = """
Something here
"""

=== module.pxd

include "file.pxi"

def func():
    FUNC_DOCSTRING
    
    return None



When I compile code like this, I see the error


Compiler crash traceback from this point on:
  File "Cython/Compiler/Visitor.py", line 183, in Cython.Compiler.Visitor.TreeVisitor._visit (/home/ilan/minonda/conda-bld/work/Cython-0.23.4/Cython/Compiler/Visitor.c:4646)
  File "/data/unixhome/ksheppard/anaconda/envs/py35/lib/python3.5/site-packages/Cython/Compiler/AnalysedTreeTransforms.py", line 74, in visit_FuncDefNode
    if not self.all_docstrings and '>>>' not in node.doc:
TypeError: a bytes-like object is required, not 'str'
Traceback (most recent call last):
  File "setup.py", line 196, in <module>
    ext_modules = cythonize(extensions)
  File "/data/unixhome/ksheppard/anaconda/envs/py35/lib/python3.5/site-packages/Cython/Build/Dependencies.py", line 877, in cythonize
    cythonize_one(*args)
  File "/data/unixhome/ksheppard/anaconda/envs/py35/lib/python3.5/site-packages/Cython/Build/Dependencies.py", line 997, in cythonize_one
    raise CompileError(None, pyx_file)
Cython.Compiler.Errors.CompileError: ./randomstate/dsfmt.pyx


The offending code is 

if not self.all_docstrings and '>>>' not in node.doc:
    return node


and the problem is that node.doc is Bytes (technically String.EncodedString) but '>>>' is not.  This appears to be caused by Cython reading the .pxi file as binary (I think, 'rb').


The actual code that crashes is in this module: https://github.com/bashtage/ng-numpy-randomstate/tree/1-11-release

Specifically 

https://github.com/bashtage/ng-numpy-randomstate/blob/1-11-release/randomstate/interface/dSFMT/dSFMT.pxi#L165

and 

https://github.com/bashtage/ng-numpy-randomstate/blob/1-11-release/randomstate/randomstate.pyx#L384


Any ideas on how to fix this?

Thanks,
Kevin

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Antony Lee | 8 Feb 09:55 2016
Picon

Save a few spurious clang warnings in the generated code

Compiling the following code

#cython: boundscheck=False
def main(double[:, ::1] x):
    return x[0]

with clang generates a bunch of warnings.  A few of them can be trivially avoided by marking a branch as knowingly dead:

testmod.c:1356:15: warning: code will never be executed [-Wunreachable-code]
    if (0 && (__pyx_tmp_idx < 0 || __pyx_tmp_idx >= __pyx_tmp_shape)) {
              ^~~~~~~~~~~~~
testmod.c:1356:9: note: silence by adding parentheses to mark code as explicitly dead
    if (0 && (__pyx_tmp_idx < 0 || __pyx_tmp_idx >= __pyx_tmp_shape)) {
        ^
        /* DISABLES CODE */ ( )

(i.e., write "if ((0) && ...)" instead of "if (0 &&)")

This heuristic was introduced in clang at this commit: http://lists.llvm.org/pipermail/cfe-commits/Week-of-Mon-20140324/102207.html

Antony

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Marko Loparic | 7 Feb 11:02 2016
Picon

looking for a cython solution for pure python code in two modules

Hello,

I am translating a C code into cython and struggling trying to find a way to obtain the same performance when adding some modularity. The good news is that -- already in the C code -- 99% of the time is spent in a single loop. Simplifying it and writing without thinking about cython we get to

import numpy as np
class Problem:
    def __init__(self, n, m):
        self.n = n
        self.m = m
        self.price = np.linspace(1, 2, n)
        self.amount = np.linspace(3, 4, m)
    def value(self, i, j):    
        return self.price[i] * self.amount(j)
def compute(problem):
    value = 0
    for i in range(problem.n):
        for j in range(problem.m):
            value += problem.value(i, j)
    return value
p = Problem(5000, 5000)
print 'value =', compute(p)

The actual value routine is bigger but is restricted to the manipulation of integers and arrays, like the one shown here. The compute algorithm is also more complex, but also consists on integer and array manipulation plus calls to routines of the problem class/module passing integer arguments.

My goals are (from the most important to the least important):

1. Achieve C performance (ie. "nogil code", white cython html) for the loop. If possible, the value method/function should be inlined (so to avoid a C function call).

2. Separate the value function/method from the compute code in two different modules. The compute code ---a library, with a generic algorithm -- is to be used by different problems, each one with a different value routine. I know that in order to have my first goal met the link between the two modules has to be done in compilation time. The 'problem' object does not need to be a cython class. It can also be a cython module with functions.

3. Have these two modules in a cython pure python source code, so as to be able to run the code without cython (and without having two different versions of the same code).

In our tests so far we have only been able to achieve goal 1 using C structures (instead of numpy annotated arrays). This forbids (as far as I understand) the usage of pure python mode.

Do you have something to suggest?

Thanks a lot!
Marko

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Walter White | 5 Feb 14:40 2016
Picon

Problem with Cython, extern "C" and unresolved external symbol

Hello,

thanks Robert for answering my previous questions on the public stuct.
Now this works fine, but there are other problems now.
#pyximport uses parameter language='c++' I have a header.h file like
---
int Test1(int input); extern "C" { _declspec(dllexport) int Test1_C(int input) { return Test1(input); } }
--
The cython pyx code is like: cdef extern from "header.h": cdef int Test1(int input) cdef int Test1_C(int input) --
Now when I am trying to run my Python code I get the
following error message: error LNK2019: unresolved external symbol "int __cdecl Test1(int)" (?Test1 <at> <at> YAHH <at> Z) referenced in function Test1_C


These are things that might cause the problem but I could not solve it:I am not sure if this related to what is mentioned here:
http://docs.cython.org/src/userguide/wrapping_CPlusPlus.html#access-to-c-only-functions
Is it?

The cpp-file states "using namespace std".
Does this mean I have to use
cdef extern from "header.h" namespace "std" ?

I am using pyximport with the following pyxbld file:

---
def make_ext(modname, pyxfilename):    return Extension(name=modname,

    sources=[pyxfilename],

    include_dirs = include_dirs,

    library_dirs = library_dirs,

    language='c++')

---

Any help is very much appreciated!

Joe


--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Vladimir Bashkardin | 5 Feb 00:50 2016
Picon

Accidentaly missing return type declaration in extern void C function

Hello,

I encountered the following problem recently with Cython 0.23.4.

I have a simple C void function. I declare it in a Cython wrapper file as cdef extern dummy_func(...). Accidentally, I forgot to declare its return type. It turns out, that Cython thinks that the function returns a PyObject pointer in that case. The corresponding Cython wrapper function then segfaults when it tries to decrement a reference to that non-existent object. I attached a simple example that demonstrates that. You can compile it with compile.sh script (tested on Linux only). The crash should be reproducible by running dummy.py script. I tested it with gcc 4.3.4, icc 15.0.1, and icc 16.0.1 on Linux x86_64 platform.

I think that it would be good to have a warning from Cython in such case that it makes an assumption about the function's return type. Right now, -Wextra does not produce any warnings.

Thank you.

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Attachment (cyvoid.tgz): application/x-gtar, 911 bytes
Jonatan Bording | 4 Feb 21:33 2016
Picon

cpdef Special Methods of Extension Types

Is it in the plans to support cpdef for special methods of extension types like __add__, __iadd__, __mul__, etc? 
It would be nice to have for faster operators on extension types.


--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Nikolaus Rath | 3 Feb 04:41 2016

Unexpected explicit cast in generated code

Hi,

Consider this code:

,---- | from libc.stdio cimport printf | from libc.stdint cimport 
int64_t |  | cdef double time_scale = 1 << 30 |  | def test(val): 
|     cdef double newval |     cdef int64_t intval |  |     intval 
= val |     newval = intval / time_scale |     printf("New value: 
%.17e\n", newval) `---- 

When compiled with just `cython`, the generated code is:

  if (unlikely(__pyx_v_5mylib_time_scale == 0)) { 
    PyErr_SetString(PyExc_ZeroDivisionError, "float division"); 
    {__pyx_filename = __pyx_f[0]; __pyx_lineno = 11; __pyx_clineno 
    = __LINE__; goto __pyx_L1_error;} 
  } __pyx_v_newval = (__pyx_v_intval / __pyx_v_5mylib_time_scale); 

When this is compiled with -Wconversion, gcc correctly warns that 
the automatic conversion of intval to double may alter the value.

However, when compiling with `cython -3`, the resulting code is

  if (unlikely(__pyx_v_5mylib_time_scale == 0)) { 
    PyErr_SetString(PyExc_ZeroDivisionError, "float division"); 
    {__pyx_filename = __pyx_f[0]; __pyx_lineno = 11; __pyx_clineno 
    = __LINE__; goto __pyx_L1_error;} 
  } __pyx_v_newval = (((double)__pyx_v_intval) / 
  __pyx_v_5mylib_time_scale); 

Here Cython has inserted an explicit cast, so the developer no 
longer gets a warning about the possible alteration of the value.

Is this really intended behavior? I think -3 should effect only 
Python variables, but in this case both operands are C values.

(Tested with Cython 0.24.0a0 from git)

Best,
-Nikolaus

(No Cc on replies please, I'm reading the list)
-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

--

-- 

--- 
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Daniel Burchardt | 2 Feb 21:32 2016
Picon

Cython and c++

Hi,

I try to run v8 in python and i can't run it... :(

setup.py
from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
from Cython.Distutils import build_ext

import os
from distutils.sysconfig import get_config_vars

(opt,) = get_config_vars('OPT')
os
.environ['OPT'] = " ".join(
    flag
for flag in opt.split() if flag != '-Wstrict-prototypes'
)

setup
(
    name
= 'helloworld',
    ext_modules
= cythonize(
 
Extension(
     
"helloworld",
     
["helloworld.pyx"],
      language
= "c++",
      libraries
= ["v8", "stdc++"],
      library_dirs
= ["v8"],
      include_dirs
= ["v8/include"],
      runtime_library_dirs
= ["v8"],
      extra_compile_args
= ["-std=c++11"],
      extra_link_args
= ["-std=c++11"],
     
#cmdclass = {'build_ext': build_ext}
 
)
   
)
)

helloworld.pyx
from libcpp cimport bool

cdef
extern from "v8.h" namespace "v8":
  cdef cppclass V8
:
       
<at> staticmethod
       
bool InitializeICU(const char *icu_data_file)
       
# <at> staticmethod
       
#void InitializeExternalStartupData(const char* directory_path)

cdef
class Script:
 
def ini(self):
        cdef
char* xyz = 'bla'

       
return V8.InitializeICU(xyz)

 
def test(self):
       
self.init()

       
return "wow"

Compilation process is OK but when i try import helloworld i see:

Traceback (most recent call last):
 
File "test.py", line 1, in <module>
   
import helloworld
ImportError: /opt/cython/helloworld.so: undefined symbol: _ZN2v82V813InitializeICUEPKc



--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Valcortez | 2 Feb 01:36 2016
Picon

Possible ref counting bug (<refcnt 0 at ...>)

Hi,

I'm having a issue with a parameter of a function passed using partial being replaced by a tuple with a single value which prints as <refcnt 0 at ...>. I believe this happens when a second thread is the only one keeping a ref to the partial function. Lemme first describe my code. The following version of the code does work fine:

cdef object _log_callback = None
cdef
int log_level = AV_LOG_WARNING

cdef
void gil_call_callback(char *line, int level):
    cdef
object callback
    callback
= _log_callback
   
if callback is None:
       
return
    callback
(tcode(line), _loglevel_inverse[level])

cdef
void call_callback(char *line, int level) nogil:
   
with gil:
        gil_call_callback
(line, level)

cdef
void _log_callback_func(void* ptr, int level, const char* fmt, va_list vl) nogil:
   
if fmt == NULL or level > log_level:
       
return
    do_stuff
...
    call_callback
(line, level)



The reason for this 3 function deep structure is that I want _log_callback_func to exit quickly if fmt == NULL or level > log_level. But if in this function I acquire the gil then upon the function call the gil is synced which I believe will slow down the call even if we never actually acquire the gil when we return early (see the cleaned generated c code all the way at the end to see what I mean about the additional gil stuff added when the gil is acquired anywhere in the func).

Anyway, here's a short version of the callback that gets set to _log_callback:

def _dshow_log_callback(log, message, level):
    log
.append((message.encode('utf8'), level))

cdef
int list_dshow_opts(list log):
   
global _log_callback
    _log_callback
= partial(_dshow_log_callback, log)


Quite often what happens is that _log_callback_func or its called functions is the only place that holds a reference to the generated partial callback because during a _log_callback_func call the global _log_callback maybe be replaced.

Anyway, so far the code works. The problem arises with the following version of the above code:

cdef void gil_call_callback(object callback, char *line, int level) nogil:
   
with gil:
        callback
(tcode(line), _loglevel_inverse[level])

cdef
void call_callback(object callback, char *line, int level) nogil:
   
if callback is None or level > log_level:
       
return
    gil_call_callback
(callback, line, level)

cdef
void _log_callback_func(void* ptr, int level, const char* fmt, va_list vl) nogil:
    do_stuff
...
    call_callback
(_log_callback, line, level)

You'll notice, the difference here is that _log_callback gets captured by _log_callback_func as a parameter without having the gil. The reason for writing the code this way is not completely sane, but mainly I was trying to get the _log_callback value without the gil so I can check whether it's None and exit early quickly in that case.

The problem is that the line log.append((message.encode('utf8'), level)) raises an exception: tuple doesn't have a append method and when inspecting what the value of log is, instead of being a list it prints as (<refcnt 0 at ...>, ). I think this happens only when _log_callback_func is the only place that still has access to the callback.

I wonder whether it's an oversight that the compiler doesn't complain about capturing a global as a function param without the gil? For instance, if I did:

cdef ... nogil:
    cdef
object callback
   
with gil:
        callback
= _log_callback

It complains about python temporaries without gil. In either case, getting the global _log_callback without the gil seems to make the partial function parameter  that is not held elsewhere die too early.

Thanks,
Matt




Here's the generated c code of the working version from above of the functions showing how as long as there is a with gil statement in the function, even if the gil is not acquired and we return before that, gil sync stuff still happens (e.g. PyGILState_Ensure):

static void __pyx_f_10ffpyplayer_5tools_call_callback(char *__pyx_v_line, int __pyx_v_level) {
  __Pyx_RefNannyDeclarations
 
#ifdef WITH_THREAD
 
PyGILState_STATE __pyx_gilstate_save;
 
#endif
  __Pyx_RefNannySetupContext
("call_callback", 1);

 
{
   
{
       
#ifdef WITH_THREAD
       
PyGILState_STATE __pyx_gilstate_save = PyGILState_Ensure();
       
#endif
          __pyx_f_10ffpyplayer_5tools_gil_call_callback
(__pyx_v_line, __pyx_v_level);
       
}
       
/*finally:*/ {
         
/*normal exit:*/{
           
#ifdef WITH_THREAD
           
PyGILState_Release(__pyx_gilstate_save);
           
#endif
           
goto __pyx_L8;
         
}
          __pyx_L8
:;
       
}
   
}
 
}
 
/*finally:*/ {
   
/*normal exit:*/{
     
#ifdef WITH_THREAD
      __pyx_gilstate_save
= PyGILState_Ensure();
     
#endif
     
goto __pyx_L5;
   
}
    __pyx_L5
:;
 
}

 
/* function exit code */
 
#ifdef WITH_THREAD
 
PyGILState_Release(__pyx_gilstate_save);
 
#endif
}

static void __pyx_f_10ffpyplayer_5tools__log_callback_func(void *__pyx_v_ptr, int __pyx_v_level, char const *__pyx_v_fmt, va_list __pyx_v_vl) {
 
int __pyx_t_1;
 
int __pyx_t_2;
  __pyx_t_2
= ((__pyx_v_fmt == NULL) != 0);
 
if (!__pyx_t_2) {
 
} else {
    __pyx_t_1
= __pyx_t_2;
   
goto __pyx_L4_bool_binop_done;
 
}
  __pyx_t_2
= ((__pyx_v_level > __pyx_v_10ffpyplayer_5tools_log_level) != 0);
  __pyx_t_1
= __pyx_t_2;
  __pyx_L4_bool_binop_done
:;
 
if (__pyx_t_1) {
   
goto __pyx_L0;
 
}
  __pyx_f_10ffpyplayer_5tools_call_callback
(__pyx_v_line, __pyx_v_level);
  __pyx_L0
:;
}


--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Nikolaus Rath | 1 Feb 23:26 2016

Generate system include directives?

Hello,

Is there a way to tell Cython that it should generate an includes 
for a system header file, instead of a local header file?

I.e., in some cases I would like 

  cdef extern from "pthread.h" nogil:
      # bla

to generate

#include <pthread.h>

instead of

#include "pthread.h"

Best,
-Nikolaus

 
(No Cc on replies please, I'm reading the list)
-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

--

-- 

--- 
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Nikolaus Rath | 1 Feb 22:58 2016

Inclusion of semaphore.h in generated code

Hello,

Is there a reason why Cython includes semaphore.h in the generated 
C code? And if so, is there some way to avoid that?

I have trouble compiling the generated code under MacOS-X, because 
MacOS doesn't have full semphore support, and the compatibility 
layer that I'm trying to use 
(https://raw.githubusercontent.com/osxfuse/sshfs/master/compat/darwin_compat.h) 
wants to use it's own definition of sem_t - so doesn't compile if 
semphore.h is included.

Best,
-Nikolaus

(No Cc on replies please, I'm reading the list)
-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

--

-- 

--- 
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Gmane