Eleftherios Garyfallidis | 6 Mar 03:31 2014
Picon

Using prange in simple example performs slower than without parallelism

Hello,

I am trying to increase the performance of the following simple function using Cython 0.19:

cimport cython
from cython.parallel import parallel, prange

from libc.math cimport sqrt, exp

<at> cython.boundscheck(False)
<at> cython.wraparound(False)
def test1(double [:, :, ::1] arr):

    cdef:
        int i, j, k, I, J, K
        double value = 0

    I = arr.shape[0]
    J = arr.shape[1]
    K = arr.shape[2]

    with nogil:

        for i in range(I):
            for j in range(J):
                for k in range(K):

                    value = <double>(i + j + k)

                    arr[i, j, k] = sqrt(value)

<at> cython.boundscheck(False)
<at> cython.wraparound(False)
def test2(double [:, :, ::1] arr):

    cdef:
        int i, j, k, I, J, K
        double value = 0


    I = arr.shape[0]
    J = arr.shape[1]
    K = arr.shape[2]

    with nogil, parallel():

        for i in prange(I):
            for j in range(J):
                for k in range(K):

                    value = <double>(i + j + k)

                    arr[i, j, k] = sqrt(value)

So, test1 does not use openmp but test2 does. But, it seems I am doing something wrong and test2 is not gaining any speed. Actually, it even does worse than test1.

For this benchmark, I am using a notebook with 2 core i7 cpus with multithreading. Here is how I made the comparisons.

A = np.ones((10000, 100, 100))

In [12]: %timeit -n 100 test1(A)
100 loops, best of 3: 136 ms per loop

In [11]: %timeit -n 100 test2(A)
100 loops, best of 3: 152 ms per loop

Let me know what I am missing and apologies for the naivety of the question.

Thank you in advance.

Best,
Eleftherios


--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Marcel Martin | 2 Mar 21:58 2014
Picon

constructing a bytes object and_PyBytes_Resize

Hello,

in one of my functions, I would like to construct a bytes object whose
final length I do not know at allocation time. I know an upper limit to
the size and want to resize the object to the correct size afterwards,
using _PyBytes_Resize.

def f():
  cdef bytes result = PyBytes_FromStringAndSize(NULL, 10)
  cdef char* b = result
  b[0] = 'h'
  b[1] = 'i'
  _PyBytes_Resize(result, 2)
  return result

The line with _PyBytes_Resize gives me this message:
  Cannot convert Python object to 'PyObject **'

Apart from not getting the syntax right, I'm aware that _PyBytes_Resize
starts with an underscore, so perhaps I should use something else.
What's the correct approach here?

Regards,
Marcel

--

-- 

--- 
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Joel Daniels | 28 Feb 19:25 2014
Picon

How to pass a file opened in Python to a C function for writing?

Hi there, first time posting here.  I'm kind of a noob with Cython, so bear with me.  :)

I've written a function in C to write to a file, but I need to pass it a file I've already opened in Python.  I assume this is possible, but I can't figure out what the syntax should look like, and Googling hasn't turned anything up.

Thanks in advance!

--
 
---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Ian Bell | 28 Feb 11:57 2014
Picon

Bug in default encoding with fused types on py3

I have a function in an extension type that has a prototype like

cpdef set_Fluid(self, string_like Fluid)

where the fused type string_like is

ctypedef fused string_like:

    cython.bytes

    cython.unicode


Basically I need to take an ascii string and pass it to a C++ function get_Fluid_index(string name).  If I do

cpdef set_Fluid(self, string_like Fluid):
    self.Fluid = Fluid.encode('ascii')


    if self.Fluid.startswith('REFPROP-'):
        _add_REFPROP_fluid(self.Fluid)
    self.iFluid = _get_Fluid_index(self.Fluid)

everything is ok, the string_like object is encoded to ascii, all is well.  But If I remove the .encode('ascii') from self.Fluid = Fluid.encode('ascii'), the string is encoded improperly.

I have set the default encoding in my file:

#cython: embedsignature = True, c_string_type=str, c_string_encoding=ascii

so the .encode('ascii') should not be necessary, right?

Thanks,
Ian

--
 
---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Ian Bell | 28 Feb 11:41 2014
Picon

Bug in use of fused types in initializer of extension type

I'm trying to use a fused type

ctypedef fused string_like:

    cython.bytes

    cython.unicode


in an extension type initializer which looks like:

def __init__(self, string_like Fluid, dict StateDict, object phase = None):

But when I try to compile, I get compile errors shown below.  I was able to workaround by changing the string_like type to object, but I think this a bug.  If not, it would be good if the error message was less obtuse.

Error compiling Cython file:
------------------------------------------------------------
...

    Instead in this class all that is handled internally. So the call to update
    sets the internal variables in the most computationally efficient way possible
    """

    def __init__(self, string_like Fluid, dict StateDict, object phase = None):
   ^
------------------------------------------------------------

CoolProp\CoolProp.pyx:843:4: Special method __defaults__ has wrong number of arguments (0 declared, 1 or more expected)

Error compiling Cython file:
------------------------------------------------------------
...

    Instead in this class all that is handled internally. So the call to update
    sets the internal variables in the most computationally efficient way possible
    """

    def __init__(self, string_like Fluid, dict StateDict, object phase = None):
   ^
------------------------------------------------------------

CoolProp\CoolProp.pyx:843:4: Special method __defaults__ has wrong number of arguments (0 declared, 1 or more expected)

--
 
---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Will Mayner | 28 Feb 01:03 2014
Picon

Best way to convert numpy array to C++ vector?

Hi,

What is the best way to convert a numpy array to a C++ std::vector?

I'm trying to wrap a C++ library for computing the Earth Mover's Distance in Cython so that it can be used to compute the EMD of two numpy arrays from normal Python code, and the library works with C++ vectors. From looking around here I've found that I can probably hack it together by
  • first converting the numpy array to a C-style array by passing the data pointer, as described here;
  • then converting that to a vector, as described here
But I was wondering if there is a better (easier to write, more efficient) way. Any ideas/comments would be greatly appreciated. Thanks!


— Will

--
 
---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Czarek Tomczak | 27 Feb 14:36 2014
Picon

Static analysis of code that makes calls to python standard libraries?

Hello Cython enthusiasts,

When writing new functions and classes in Cython, typos like this will be detected by compiler:

cdef void Some():
    asd() # nonexistent function

But what about static checking when using methods/functions from python standard library? The code below compiles fine:

cdef void Some():
    os.asd()

What are my options to detect errors like this? In Python there are tools like pychecker/pylint that can help with that, but I don't think they are compatible with Cython (I've used google, correct me if I'm wrong).

I've found that mypy can do static checking when calling standard libraries, a quote from http://www.mypy-lang.org/:

Most commonly used libraries also have statically typed interface definitions that allow the type checker to check code that uses the libraries.

Could Cython somehow reuse mypy statically typed interface definitions? Any other ideas?

Best regards,
Czarek

--
 
---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Anthony Scopatz | 27 Feb 01:24 2014
Picon

ANN: XDress v0.4

Hello All, 

I am extremely pleased to be able to announce the version 0.4 release of xdress.  This version contains much anticipated full support for Clang as a parser!  This is almost entirely due to the efforts of Geoffrey Irving.  Please thank him the next time you get a chance :)

This release also contains a lot of other goodies that you can read about in the release notes below.

Happy Generating!
Anthony

XDress 0.4 Release Notes

XDress is a numpy-aware automatic wrapper generator for C/C++ written in pure Python. Currently, xdress may generate Python bindings (via Cython) for C++ classes, functions, and certain variable types. It also contains idiomatic wrappers for C++ standard library containers (sets, vectors, maps). In the future, other tools and bindings will be supported.

The main enabling feature of xdress is a dynamic type system that was designed with the purpose of API generation in mind.

Release highlights:

  • Clang support! All kudos to Geoffrey Irving!
  • NumPy dtypes may be created independently of C++ STL vectors
  • A complete test suite refactor
  • Arbitrary source code locations
  • Global run control files
  • A plethora of useful bug fixes

This version of xdress is not 100% backwards compatible with previous versions of xdress. We apologize in the name of progress. It represents ans impressive 245 files changed, 44917 aggregate line insertions (+), and 7893 deletions (-).

Please visit the website for more information: http://xdress.org/

Ask questions on the mailing list: https://groups.google.com/forum/#!forum/xdress

Download the code from GitHub: http://github.com/xdress/xdress

XDress is free & open source (BSD 2-clause license) and requires Python 2.7+, NumPy 1.5+, Cython 0.19+, and optionally Clang, GCC-XML, pycparser, dOxygen, or lxml.

New Features

Clang Support

Through the herculean efforts of Geoffrey Irving xdress finally has full, first-class Clang/LLVM support! This is major advancement as it allows xdress to wrap more modern versions of C++ than GCC-XML can handle. Because of deficiencies in the existing libclang and Python bindings it was necessary for us to fork libclang for xdress in the short term. We hope to integrate these changes upstream. Clang versions 3.2 - 3.4 are supported.

Independent NumPy Dtypes

In previous versions of xdress, to create a dtype of type T the user needed to declare the desire for a wrapper of an STL vector of type T. These two desires have now been separated. It is now possible to create a dtype via the dtypes run control parameter. STL vectors are still wrapped via dtypes. See the dtypes module for more information.

Shiny New Test Suite

The xdress test suite has been completely revamped to include both unit and integration tests which are run for all available parsers. The integration tests are accomplished though two fake projects - cproj and cppproj - on which the xdress CLI is run. These tests are now fully platform independent, unlike the previous BASH-based test suite.

Source Paths

Source file paths are now given by either their absolute or relative path. This allows source code to be located anywhere on the user’s file system and enable the wrapping of dependencies or externally supplied libraries as needed. The run control parametersourcedir has been deprecated.

Global Run Control Files

It is sometimes useful to be able to set system-wide run control parameters. XDress will now search the following files in order of increasing precedence.

  • $HOME/.xdressrc
  • $HOME/.xdressrc.py
  • $HOME/.config/xdressrc
  • $HOME/.config/xdressrc.py

$HOME is the user’s home directory. Settings in the project run control file take precedence over the values here.

Major Bug Fixes

  • Debug file now always written when in debug mode.
  • STL sets of custom types now allowed.
  • Template parameters now allowed to be enum values.
  • Allow classes with no default constructor.

Join in the Fun!

If you are interested in using xdress on your project (and need help), contributing back to xdress, starting up a development team, or writing your own code generation plugin tool, please let us know. Participation is very welcome!

Authors

  • Anthony Scopatz
  • Geoffrey Irving *
  • James Casbon *
  • Kevin Tew *
  • Spencer Lyon
  • John Wiggins
  • Matt McCormick
  • Brad Buran
  • Chris Harris *
  • Gerald Dalley *
  • Micky Latowicki *
  • Mike C. Fletcher *
  • Robert Schwarz *

An * indicates a first time contributor.

Links

  1. Homepage - http://xdress.org/
  2. Mailing List - https://groups.google.com/forum/#!forum/xdress
  3. GitHub Organization - https://github.com/xdress

--
 
---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Jonathan Buck | 26 Feb 17:33 2014
Picon

Problem linking gomp and pthread for OpenMP

In my ongoing effort to get familiar with Cython, I'm trying to use Cython's OpenMP capabilities, but I'm having difficulty getting it to compile.

I've got a setup file as follows:

from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize

setup(
    ext_modules = cythonize([Extension("cython_sum_p", ["cython_sum_p.pyx"],
    extra_compile_args=["-fopenmp"],
    extra_link_args=["-fopenmp"])])
)


Then I try to build the extension module:
python cython_sum_parallel_setup.py build_ext --inplace

And I get errors about pthread and gopenmp.


running build_ext
building 'cython_sum_p' extension
C:\Anaconda\Scripts\gcc.bat -DMS_WIN64 -mdll -O -Wall -IC:\Anaconda\include -IC:\Anaconda\PC -c cython_sum_p.c -o build\temp.win-amd64-2.7\Release\cython_sum_p.
o -fopenmp
cython_sum_p.c: In function '__Pyx_BufFmt_ProcessTypeChunk':
cython_sum_p.c:13707:26: warning: unknown conversion type character 'z' in format [-Wformat]
cython_sum_p.c:13707:26: warning: unknown conversion type character 'z' in format [-Wformat]
cython_sum_p.c:13707:26: warning: too many arguments for format [-Wformat-extra-args]
cython_sum_p.c:13759:20: warning: unknown conversion type character 'z' in format [-Wformat]
cython_sum_p.c:13759:20: warning: unknown conversion type character 'z' in format [-Wformat]
cython_sum_p.c:13759:20: warning: too many arguments for format [-Wformat-extra-args]
cython_sum_p.c: In function '__pyx_buffmt_parse_array':
cython_sum_p.c:13819:25: warning: unknown conversion type character 'z' in format [-Wformat]
cython_sum_p.c:13819:25: warning: format '%d' expects argument of type 'int', but argument 3 has type 'size_t' [-Wformat]
cython_sum_p.c:13819:25: warning: too many arguments for format [-Wformat-extra-args]
cython_sum_p.c: In function '__Pyx_GetBufferAndValidate':
cython_sum_p.c:14000:7: warning: unknown conversion type character 'z' in format [-Wformat]
cython_sum_p.c:14000:7: warning: format '%s' expects argument of type 'char *', but argument 3 has type 'Py_ssize_t' [-Wformat]
cython_sum_p.c:14000:7: warning: unknown conversion type character 'z' in format [-Wformat]
cython_sum_p.c:14000:7: warning: too many arguments for format [-Wformat-extra-args]
cython_sum_p.c: In function '__Pyx_RaiseArgtupleInvalid':
cython_sum_p.c:14172:18: warning: unknown conversion type character 'z' in format [-Wformat]
cython_sum_p.c:14172:18: warning: format '%s' expects argument of type 'char *', but argument 5 has type 'Py_ssize_t' [-Wformat]
cython_sum_p.c:14172:18: warning: unknown conversion type character 'z' in format [-Wformat]
cython_sum_p.c:14172:18: warning: too many arguments for format [-Wformat-extra-args]
cython_sum_p.c: In function '__Pyx_RaiseTooManyValuesError':
cython_sum_p.c:14707:18: warning: unknown conversion type character 'z' in format [-Wformat]
cython_sum_p.c:14707:18: warning: too many arguments for format [-Wformat-extra-args]
cython_sum_p.c: In function '__Pyx_RaiseNeedMoreValuesError':
cython_sum_p.c:14713:18: warning: unknown conversion type character 'z' in format [-Wformat]
cython_sum_p.c:14713:18: warning: format '%s' expects argument of type 'char *', but argument 3 has type 'Py_ssize_t' [-Wformat]
cython_sum_p.c:14713:18: warning: too many arguments for format [-Wformat-extra-args]
cython_sum_p.c: In function '__Pyx_ValidateAndInit_memviewslice':
cython_sum_p.c:15213:22: warning: unknown conversion type character 'z' in format [-Wformat]
cython_sum_p.c:15213:22: warning: format '%s' expects argument of type 'char *', but argument 3 has type 'Py_ssize_t' [-Wformat]
cython_sum_p.c:15213:22: warning: unknown conversion type character 'z' in format [-Wformat]
cython_sum_p.c:15213:22: warning: too many arguments for format [-Wformat-extra-args]
writing build\temp.win-amd64-2.7\Release\cython_sum_p.def
C:\Anaconda\Scripts\gcc.bat -DMS_WIN64 -shared -s build\temp.win-amd64-2.7\Release\cython_sum_p.o build\temp.win-amd64-2.7\Release\cython_sum_p.def -LC:\Anacond
a\libs -LC:\Anaconda\PCbuild\amd64 -lpython27 -lmsvcr90 -o H:\training\Python\2014Seminar\Demos\cython_sum_p.pyd -fopenmp
c:/anaconda/mingw/bin/../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot find -lgomp
c:/anaconda/mingw/bin/../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot find -lpthread
collect2.exe: error: ld returned 1 exit status
error: command 'gcc' failed with exit status 1

If it's relevant, I'm on a Windows 7 x86-64 system. I'm running the latest build of Python 2.7.6 from Anaconda v1.9.0. Is this just a simple matter of including additional directories in my setup file?

Thanks,
Jonathan

--
 
---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
yarden.katz | 26 Feb 05:47 2014
Picon

Problem calling clapack from Cython

Hi all,

I'm trying to simply call a C-LAPACK function (in this case, matrix multiplication) from Cython.  I don't want to dynamically link to an existing C-LAPACK installation, so I've packaged the necessary headers/*.c files for that function with my code.  I compile it using setup.py and then try to call it from a Cython (*.pyx) file.

I made a self-contained example here:

https://github.com/yarden/learn_cython

The README.txt explains how to compile and "run.py" is the test script. The key code from that example is below. I basically include the *.c / *.h files in the src/ and include/ directories and then define the external function as:

cdef extern from "f2c.h":
   ctypedef int integer
   ctypedef double doublereal


# Import lapack functions.
cdef extern from "clapack.h":
   cdef extern from "f2c.h":
       pass
   integer c_dgemm "dgemm_"(char *transa, char *transb, integer *m, integer *
                             n, integer *k, doublereal *alpha, doublereal *a, integer *lda,
                             doublereal *b, integer *ldb, doublereal *beta, doublereal *c__,
                             integer *ldc)


I then call the function as follows:

cdef int main():
    # Form of matrix
    cdef char transa_val = 'N'
    cdef char *transa = &transa_val

    cdef char transb_val = 'N'
    cdef char *transb = &transb_val

    # Number of rows of matrix A
    cdef int m = 3
    cdef int n = 3
    cdef int k = 3

    cdef double alpha = 1.0
    cdef np.ndarray[double, ndim=2, mode="c"] a = \
      np.array([[1, 2, 3],
                [4, 5, 6],
                [7, 8, 9]], dtype=float)

    cdef np.ndarray[double, ndim=2, mode="c"] b = \
      np.array([[1, 0, 0],
                [0, 0, 0],
                [1, 1, 1]], dtype=float)

    # result is a double pointer
    cdef np.ndarray[double, ndim=2, mode="c"] c = \
      np.empty([3, 3], dtype=float)


    cdef int lda = m
    cdef int ldb = n

    cdef double beta = 1.0

    cdef int ldc = 1

    print "Multipling: "
    print a
    print " times "
    print b

    c_dgemm(transa, transb, &m, &n, &k, &alpha, &a[0,0], &lda, &b[0,0], &ldb, &beta, &c[0,0], &ldc)

    return 0

main()

Then I compile and run this as follows:

$ python setup.py sdist
$ python setup.py build_ext
$ python run.py

This always yields the error:

ImportError: undefined symbol: dgemm_

Even though though the function body and header is included. What did I do wrong here?

Thanks very much in advance for your help.

--
 
---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Antoine Martin | 23 Feb 12:51 2014
Picon

easiest way to access nested structs

Hi,

I am trying to interface with some C code that uses a structure like this one:

typedef struct param
{
    int a
    struct
    {
        int b
        int c
    } sub;
} param;

I have my "param" object in Cython and all I really want to do is the equivalent of:
param.sub.b = 1

Nothing that I've tried even compiles (ie: sticking the "sub" struct in its own typedef fails with error: unknown type name ‘sub’)

What is the easiest way of doing this without adding a C file to access all the "sub" fields? (which is tedious)

Thanks
Antoine

--
 
---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Gmane