Stefan Behnel | 16 Aug 08:40 2014
Picon

[Cython] python-ideas discussion on "multiple dispatch", also operator overloading

Hi,

there's a discussion going on on python-ideas about generic functions and
multiple dispatch. Might be interesting for the fused functions
implementation in Cython.

http://thread.gmane.org/gmane.comp.python.ideas/28790/focus=28848

Cython can definitely improve its own dispatch implementation.

The reply I linked to is by Nick Coghlan, who also makes an interesting
comment on operator overloading methods and NotImplemented:

"""
A *lot* of folks make the mistake
of raising TypeError or NotImplementedError directly in their operator
overload implementations, rather than returning the NotImplemented
singleton that tells the interpreter to try the other type. Even some
of the CPython builtins get that wrong, since the sq_concat and
sq_repeat slots in C don't properly support the type coercion dance,
so you *have* to raise the exception yourself if you're only
implementing those without implementing nb_add and nb_mul (types
defined in Python automatically populate both sets of C level slots if
you define __add__ or __mul__).
"""

I think Cython could help its users here to a certain extent, if only by
warning about raising TypeError explicitly in these methods and encouraging
better behaviour.

(Continue reading)

Andriy Kornatskyy | 14 Aug 07:20 2014

[Cython] error LNK2001: unresolved external symbol PyInit_init

Here is an issue when trying to install wheezy.template (it uses cythonize) into environment with cython.

Host: windows 7 64bit Python version: 3.4.1 32 bit wheezy.template version: 0.1.151

When installing either from pip3 or downloading the source with python3 setup.py install

c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\BIN\link.exe /DLL /nologo /INCREMENTAL:NO
/LIBPATH:C:\Python34\libs /LIBPATH:C:\Python34\PCbuild /EXPORT: PyInit_init
build\temp.win32-3.4\Release\src\wheezy\template__init.obj /O
UT:build\lib.win32-3.4\wheezy\template__init.pyd /IMPLIB:build\temp.win32-3.4
\Release\src\wheezy\template__init.lib /MANIFESTFILE:build\temp.win32-3.4\Rel ease\src\wheezy\template__init.pyd.manifest
LINK : error LNK2001: unresolved external symbol PyInit_init
build\temp.win32-3.4\Release\src\wheezy\template__init__.lib : fatal error LNK1120: 1 unresolved externals
error: command 'c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\BIN\link.exe' failed with
exit status 1120

Some details are here:
https://bitbucket.org/akorn/wheezy.template/issue/10/installation-on-windows-fails

Thanks.

Andriy Kornatskyy

Leon Bottou | 12 Aug 18:06 2014

[Cython] [Re] OpenMP thread private variable not recognized (bug report + discussion)

On Tue, 12 Aug 2014 14:26:31, Sturla Molden wrote:
> Cython does not do an error here:[...
> - i is recognized as private
> - r is recognized as reduction
> - w is (correctly) recognized as shared

Not according to the documentation.
http://docs.cython.org/src/userguide/parallelism.html documentation for
cython.parallel.parallel says "A contained prange will be a worksharing loop
that is not parallel, so any variable assigned to in the parallel section is
also private to the prange. Variables that are private in the parallel block
are unavailable after the parallel block.".  Variable w is such a variable.

Furthermore, if cython is correct, why does GCC report an error on the
cython generated C code?  

My point here is that there is a bug because (a) cython does not behave as
documented, and (b) it generates invalid C code despite not reporting an
error.

> Personally I prefer to avoid OpenMP and just use Python threads and an
> internal function (closure) or an internal class. If you start to use
OpenMP,
> Apple's libdispatch ("GCD"), Intel TBB, or Intel clikplus, you will soon
discover
> that they are all variations over the same theme: a thread pool and a
closure.

I am making heavy uses of OpenBlas which also uses OpenMP.
Using the same queue manager prevents lots of CPU provisioning problem.
(Continue reading)

Marcus Brinkmann | 12 Aug 04:57 2014
Picon

[Cython] bug: constructor declarations not working

Hi,

I want to declare and define C++ classes from Cython.  Sure, I could 
write them in native C++, but it's just small glue code, and I prefer to 
let Cython handle the GIL and all the other good stuff.

First, for reference, the following works (c.f. 
tests/run/cpp_classes_def.pyx):

== cython --cplus example1.pyx ============
cdef cppclass Foo "Foo":
   int _foo
   __init__(int foo) nogil:
     this._foo = foo
   __dealloc__() nogil:
     pass
===========================================

This will emit a declaration for Foo and the implementations of the 
constructor and destructor (output is truncated to show relevant parts 
only):

struct Foo {
   int _foo;
    Foo(int);
   virtual  ~Foo(void);
};
  Foo::Foo(int __pyx_v_foo) {
   this->_foo = __pyx_v_foo;
}
(Continue reading)

Leon Bottou | 11 Aug 03:20 2014

[Cython] OpenMP thread private variable not recognized (bug report + discussion)


The attached cython program uses an extension class to represent a unit of 
work. The with parallel block temporarily gets the gil to allocate the object, 
then release the gil and performs the task with a for i in prange(...) 
statement. My expectation was to have w recognized as a thread private 
variable.

    with nogil, parallel():
        with gil: 
            w = Worker(n)                 # should be thread private
            with nogil: 
                for i in prange(0,m): 
                    r += w.run()          # should be reduction
            w = None                      # is this needed?

Cythonize (0.20.2) works without error but produces an incorrect C file.

hello.c: In function ‘__pyx_pf_5hello_run’:
hello.c:2193:42: error: expected identifier before ‘)’ token
hello.c: At top level:

The erroneous line is:

#pragma omp parallel private() reduction(+:__pyx_v_r) private(__pyx_t_5, 
__pyx_t_4, __pyx_t_3) firstprivate(__pyx_t_1, __pyx_t_2) 
private(__pyx_filename, __pyx_lineno, __pyx_clineno) 
shared(__pyx_parallel_why, __pyx_parallel_exc_type, __pyx_parallel_exc_value, 
__pyx_parallel_exc_tb)

where you can see that the first private() clause has no argument. The 
(Continue reading)

J Robert Ray | 9 Aug 04:04 2014
Picon

[Cython] Bug: operator[] does not respect except +

I apologize for not having a working demonstration, but I have found this bug and verified it is still an issue in Cython 0.20.2. No try/catch block is generated for operator[]. In contrast, a try/catch block does get generated for operator().

If it is any use, here is a paraphrasing of the pyx code and the generated cpp code.

cdef extern from "blah.h" namespace "blah":
    cdef cppclass Blah:
        int operator[](int) except +

...

cdef class PyBlah:
    ...
    def demo(self, int index):
        return deref(self.thisptr)[index]

...

/* "Blah.pyx":107
 *     def demo(self, int index):
 *         return deref(self.thisptr)[index]             # <<<<<<<<<<<<<<
 */
  __Pyx_XDECREF(__pyx_r);
  __pyx_t_1 = __Pyx_PyInt_From_int(((*__pyx_v_self->thisptr)[__pyx_v_index])); if (unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 107; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
  __Pyx_GOTREF(__pyx_t_1);
  __pyx_r = __pyx_t_1;
  __pyx_t_1 = 0;
  goto __pyx_L0;


Please let me know if I'm doing anything wrong. Thanks!
<div><div dir="ltr">I apologize for not having a working demonstration, but I have found this bug and verified it is still an issue in Cython 0.20.2. No try/catch block is generated for operator[]. In contrast, a try/catch block does get generated for operator().<div>

<br>
</div>
<div>If it is any use, here is a paraphrasing of the pyx code and the generated cpp code.</div>
<div>
<div><br></div>
<div>cdef extern from "blah.h" namespace "blah":</div>
<div>&nbsp; &nbsp; cdef cppclass Blah:</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; int operator[](int) except +</div>
<div><br></div>
<div>...</div>
<div><br></div>
<div>cdef class PyBlah:</div>
<div>&nbsp; &nbsp; ...</div>
<div>&nbsp; &nbsp; def demo(self, int index):</div>
<div>

&nbsp; &nbsp; &nbsp; &nbsp; return deref(self.thisptr)[index]</div>
<div><br></div>
<div>...</div>
<div><br></div>
<div>
<div>/* "Blah.pyx":107</div>
<div>&nbsp;* &nbsp; &nbsp; def demo(self, int index):<br>
</div>
<div>&nbsp;* &nbsp; &nbsp; &nbsp; &nbsp; return deref(self.thisptr)[index] &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; # &lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;</div>

<div>&nbsp;*/<br>
</div>
<div>&nbsp; __Pyx_XDECREF(__pyx_r);</div>
<div>&nbsp; __pyx_t_1 = __Pyx_PyInt_From_int(((*__pyx_v_self-&gt;thisptr)[__pyx_v_index])); if (unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 107; __pyx_clineno = __LINE__; goto __pyx_L1_error;}</div>

<div>&nbsp; __Pyx_GOTREF(__pyx_t_1);</div>
<div>&nbsp; __pyx_r = __pyx_t_1;</div>
<div>&nbsp; __pyx_t_1 = 0;</div>
<div>&nbsp; goto __pyx_L0;</div>
<div><br></div>
<div><br></div>
<div>Please let me know if I'm doing anything wrong. Thanks!</div>

</div>
</div>
</div></div>
Ian Henriksen | 4 Aug 21:11 2014
Picon

[Cython] aritmetic with arrays in Cython

Hi all,
I noticed that some time ago there was a pull request (https://github.com/cython/cython/pull/144) open that was trying to implement basic arithmetic operations with arrays. This seems to have also been proposed in CEP 518 (https://github.com/cython/cython/wiki/enhancements-simd). What is the status on this?

This is a feature I'd really like to see in Cython. I'd be happy to help with it, though I don't currently know enough about the internals of Cython to do it without some guidance. What would be the best way to do this? I've had three ideas thus far that might allow something like this to work, but I'd really like to get some second opinions before I get too far with any of them.

1. Make these array operations available as a part of Cython's memory view objects.

2. Write a Cython wrapper around a NumPy-like C++ library. https://github.com/ContinuumIO/libdynd seems to be a good candidate, though I'm not as familiar with it. (This doesn't strike me as something that would be a part of the main Cython project, but it could be a good way to make this sort of functionality available. It seems like a good way to avoid a lot of duplicate work though.)

3. Write a Fortran backend for Cython that uses Fortran's arrays wherever memory views are used.
(This could be a bit crazy, but with the iso_c_binding module, it might be possible. It could also allow better optimizations for arrays to be applied by the compiler.)

What would be the best way to go about this?

Thanks!

-Ian Henriksen
<div><div dir="ltr">Hi all,<div>I noticed that some time ago there was a pull request (<a href="https://github.com/cython/cython/pull/144">https://github.com/cython/cython/pull/144</a>) open that was trying to implement basic arithmetic operations with arrays. This seems to have also been proposed in CEP 518 (<a href="https://github.com/cython/cython/wiki/enhancements-simd">https://github.com/cython/cython/wiki/enhancements-simd</a>). What is the status on this?</div>
<div><br></div>
<div>This is a feature I'd really like to see in Cython. I'd be happy to help with it, though I don't currently know enough about the internals of Cython to do it without some guidance. What would be the best way to do this? I've had three ideas thus far that might allow something like this to work, but I'd really like to get some second opinions before I get too far with any of them.</div>
<div><br></div>
<div>1. Make these array operations available as a part of Cython's memory view objects.</div>
<div><br></div>
<div>2. Write a Cython wrapper around a NumPy-like C++ library.&nbsp;<a href="https://github.com/ContinuumIO/libdynd">https://github.com/ContinuumIO/libdynd</a> seems to be a good candidate, though I'm not as familiar with it. (This doesn't strike me as something that would be a part of the main Cython project, but it could be a good way to make this sort of functionality available. It seems like a good way to avoid a lot of duplicate work though.)</div>
<div><br></div>
<div>3. Write a Fortran backend for Cython that uses Fortran's arrays wherever memory views are used.</div>
<div>(This could be a bit crazy, but with the iso_c_binding module, it might be possible. It could also allow better optimizations for arrays to be applied by the compiler.)</div>
<div><br></div>
<div>What would be the best way to go about this?</div>
<div><br></div>
<div>Thanks!</div>
<div><br></div>
<div>-Ian Henriksen</div>
</div></div>
Ian Bell | 2 Aug 12:56 2014
Picon

[Cython] Test/example for cpdef enum

Are there any tests/docs/examples showing how to use the cpdef enum?  This is a feature I have been looking for for a long time and I am hoping to use it very soon in my own project.

I have a windows box that I can run the alpha test suite on with the full complement of python versions.  Let me know if there is interest and I can run the tests.  Alternatively, if you are still doing a buildbot (the link at https://github.com/cython/cython/wiki/HackerGuide#buildbot is broken) , we could configure a nightly slave to run the tests.
<div><div dir="ltr">Are there any tests/docs/examples showing how to use the cpdef enum?&nbsp; 
This is a feature I have been looking for for a long time and I am 
hoping to use it very soon in my own project.<br><br>I have a windows 
box that I can run the alpha test suite on with the full complement of 
python versions.&nbsp; Let me know if there is interest and I can run the 
tests.&nbsp; Alternatively, if you are still doing a buildbot (the link at <a href="https://github.com/cython/cython/wiki/HackerGuide#buildbot" target="_blank">https://github.com/cython/cython/wiki/HackerGuide#buildbot</a> is broken) , we could configure a nightly slave to run the tests.</div></div>
Yaroslav Halchenko | 22 Jul 18:48 2014

[Cython] buffmt.__test__.wrongsize gets stuck after common_include_dir?

while trying to re-test Debian package build, in an interactive shell, I
get cython test stuck at:

...
Doctest: buffmt.__test__.wrongsize ... ok
runTest (__main__.EndToEndTest)
End-to-end basic_cythonize ... ok
runTest (__main__.EndToEndTest)
End-to-end basic_distutils ... ok
runTest (__main__.EndToEndTest)
End-to-end build_dir ... ok
runTest (__main__.EndToEndTest)
End-to-end common_include_dir ...

tests were executed using

runtests.py --no-refnanny -v -v --exclude="parallel" --work-dir=build/work-dir

any clues?  is it multiprocessing to blame again?

/bin/bash /tmp/hooks/C10shell
  /bin/bash
    /usr/bin/make -f debian/rules binary
      /usr/bin/perl -w /usr/bin/dh binary --with python2,python3 --buildsystem python_distutils
        /usr/bin/make -f debian/rules override_dh_auto_test
          /bin/sh -c set -e; for P in 2.7 3.4; do \          echo "PYTHON: $P"; \  export; \  PYTHONPATH=`/bin/ls -d
/tmp/buildd/cython-0.20.2/build/lib.*-$P` \   /usr/bin/python$P runtests.py --no-refnanny -v -v
--exclude="parallel" --work-dir=build/work-dir; \ done
            /usr/bin/python2.7 runtests.py --no-refnanny -v -v --exclude=parallel --work-dir=build/work-dir
              /bin/sh -c /usr/bin/python2.7 setup.py build_ext --inplace
                /usr/bin/python2.7 setup.py build_ext --inplace
                  /usr/bin/python2.7 setup.py build_ext --inplace
                  /usr/bin/python2.7 setup.py build_ext --inplace

--

-- 
Yaroslav O. Halchenko, Ph.D.
http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org
Research Scientist,            Psychological and Brain Sciences Dept.
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik        
Kurt Smith | 16 Jul 19:01 2014
Picon

[Cython] Automatic conversion with fixed-size C arrays

Hi devs,

Being able to convert between simple C structs and Python dictionaries is great, but it doesn't work if there is a fixed-size array field in the struct.  It seems like the following struct should be convertible safely:

cdef struct state_t:
    int i, j, k
    float x[3]
    float v[3]

Along with that, I'd expect that converting between fixed-sized C arrays and Python iterables should work as well:

def auto_convert(list ll):
    cdef int arr[10] = ll
    # ...
    return arr

If this is deemed something worth doing and if someone is willing to give pointers when I get stuck, I'm offering to put in the development time to implement this.

Thoughts?

Kurt
<div><div dir="ltr">Hi devs,<div><br></div>
<div>Being able to convert between simple C structs and Python dictionaries is great, but it doesn't work if there is a fixed-size array field in the struct. &nbsp;It seems like the following struct should be convertible safely:</div>
<div><br></div>
<div>cdef struct state_t:</div>
<div>&nbsp; &nbsp; int i, j, k</div>
<div>&nbsp; &nbsp; float x[3]</div>
<div>&nbsp; &nbsp; float v[3]</div>
<div><br></div>
<div>Along with that, I'd expect that converting between fixed-sized C arrays and Python iterables should work as well:</div>
<div><br></div>
<div>def auto_convert(list ll):</div>
<div>&nbsp; &nbsp; cdef int arr[10] = ll</div>
<div>&nbsp; &nbsp; # ...</div>
<div>&nbsp; &nbsp; return arr</div>
<div><br></div>
<div>If this is deemed something worth doing and if someone is willing to give pointers when I get stuck, I'm offering to put in the development time to implement this.</div>
<div><br></div>
<div>Thoughts?</div>
<div><br></div>
<div>Kurt</div>
</div></div>
Benjamin Lerman | 15 Jul 11:12 2014

[Cython] License information on each individual files

 Hi,

 I'm planning to use cython in chromium.

 To be able to use cython as a third-party library, we need a license
header on every files in the project (because we are creating source
packages, and need to have clear licensing).

 Would cython accept to add such a copyright header on its files?

 Thanks in advance,

--

-- 
        Benjamin

Gmane