Jeroen Demeyer | 20 Jun 15:08 2016
Picon

[Cython] Virtual cpdef methods

Hello,

I would like to have a "virtual" cpdef method: something which allows a 
fast Cython call *when implemented* but without a default implementation.

The most simple implementation would be a "cpdef" method but without an 
entry for the method in the Python method table.

Of course, the "cdef" part needs to be implemented by the user (you need 
to put something in the vtab). For my purposes, something like "raise 
AttributeError" would be sufficient.

1. Can this be accomplished currently in Cython?

2. Is this something that makes sense to add to Cython?

3. What do you think of the following syntax?

cdef class MyClass(object):
     cpdef virtual_method(self):
         # implementation of "cdef" part ...

     del virtual_method  # don't put virtual_method in the method table
Nathan Goldbaum | 10 Jun 18:18 2016
Picon

[Cython] Static checker for cython extension dependencies?

Hi all,

I'm working on a pretty large python/cython codebase (yt) that has many interdependent C extensions written in cython.

I've found it to be pretty hit or miss to depend on contributors to manually update cython dependency information in our setup.py file. The dependency information seems to only be used by setuptools to trigger recompilation, but critically setuptools will happily compile a C extension for the first time if the depends information is incomplete. This means that during development, if I update a cython routine, there's no way to ensure that other cython routines that cimport the one I modified will be recompiled unless I manually ensure the depends information is updated whenever cython code gains or loses a cimport.

To make that more concrete, here's a pull request I just made to yt that adds missing dependencies for a cython header. Without this pull request, setuptools fails to recompile these routines when selection_routines.pxd changes, causing a build failure.


I think it should be possible to write a test for this by defining the dependency information outside of setup.py and parsing grep and looking for all cython files that cimport other cython files defined inside yt. However, before I do that, I'm curious whether anyone has done something similar, or if there is some other way of forcing the dependency information to be complete on the first compilation, rather than just for subsequent incremental recompilations during development.

Thanks for your help!

-Nathan
<div><div dir="ltr">Hi all,<div><br></div>
<div>I'm working on a pretty large python/cython codebase (yt) that has many interdependent C extensions written in cython.</div>
<div><br></div>
<div>I've found it to be pretty hit or miss to depend on contributors to manually update cython dependency information in our setup.py file. The dependency information seems to only be used by setuptools to trigger recompilation, but critically setuptools will happily compile a C extension for the first time if the depends information is incomplete. This means that during development, if I update a cython routine, there's no way to ensure that other cython routines that cimport the one I modified will be recompiled unless I manually ensure the depends information is updated whenever cython code gains or loses a cimport.</div>
<div><br></div>
<div>To make that more concrete, here's a pull request I just made to yt that adds missing dependencies for a cython header. Without this pull request, setuptools fails to recompile these routines when selection_routines.pxd changes, causing a build failure.</div>
<div><br></div>
<div>
<a href="https://bitbucket.org/yt_analysis/yt/pull-requests/2220">https://bitbucket.org/yt_analysis/yt/pull-requests/2220</a><br>
</div>
<div><br></div>
<div>I think it should be possible to write a test for this by defining the dependency information outside of setup.py and parsing grep and looking for all cython files that cimport other cython files defined inside yt. However, before I do that, I'm curious whether anyone has done something similar, or if there is some other way of forcing the dependency information to be complete on the first compilation, rather than just for subsequent incremental recompilations during development.</div>
<div><br></div>
<div>Thanks for your help!</div>
<div><br></div>
<div>-Nathan</div>
</div></div>
Erik Bray | 9 Jun 09:36 2016
Picon
Gravatar

Re: [Cython] Suggested action for "Python.h not found"

On Thu, Jun 9, 2016 at 7:37 AM, Elizabeth A. Fischer
<elizabeth.fischer@...> wrote:
>> Or maybe by default it can just point the user to an Installation Page
>> which explains in detail what needs to be done to get those headers for
>> various systems ?
>
>
> I think having this information on an installation instruction page is
> definitely a good idea.  I don't think it makes sense to put in every single
> error message that users should consult the installation instructions; that
> should be a given.

I'm inclined to agree that putting instructions for several different
systems in the message is overkill, and is a slippery slope, but a
friendlier error message with links to documentation (and maybe
general, non-Cython-specific docs on building Python extension modules
too).

> It is my belief that non-developers should not be manually building
> software.  Nor should developers be doing that; if you're saying
> "./configure" yourself, it's like programming assembly code.
>
> There are many ways to install Cython without manually building:

Unfortunately sometimes, especially with Python software, the line
between "developer" and "non-developer" is blurry.  This is especially
true of scientific software where Cython finds its most use.  It's not
uncommon, for example, for Cython to be pulled in as a build
dependency of something else.  Of course as pointed out downthread
this is less of an issue on most platforms now that binary wheels are
available for so many (including now manylinux!).

So I guess point taken that when it comes to just building Cython
itself this is less of an issue than it used to be.

> * If you're on Linux (and maybe Mac too), install with Spack.  This can be
> as easy as typing:
>          spack install py-cython
>
> https://github.com/llnl/spack
>
> Spack gives you complete control over your compiler.  This is important if
> you're  building Python extensions, especially C++-based extensions, which
> must be built with the same compilers use to build Python.
>
> I use Spack to build my complete software stack: the application software I
> work on requires about 50 dependencies to build, and another ~20 Python
> dependencies to analyze the output.  That includes the full Numpy/Scipy
> stack.  Spack makes it easy for me to use consistent compiler and libraries
> for Python and my libraries.  I've had success making Spack-based install
> instructions that others (non-developers) can follow and get my software
> stack installed on their systems:
>     https://github.com/citibeth/icebin

This one's new to me.  Will have to investigate--this could be
relevant to my interests.

Thanks,
Erik

> On Wed, Jun 8, 2016 at 1:25 PM, Robert Bradshaw <robertwb@...> wrote:
>>
>> +1, want to submit a pull request?
>>
>> On Tue, Jun 7, 2016 at 11:28 PM, Abdeali Kothari
>> <abdealikothari@...> wrote:
>> > Hi,
>> >
>> > Currently, when a user installs cython, if the Python headers are not
>> > found,
>> > an error message saying "Python.h: no such file or directory" is shown
>> > (Example: https://justpaste.it/v0gz). Would it be possible to suggest an
>> > action to install the headers ?
>> >
>> > I found http://trac.cython.org/ticket/247 which is quite old, and makes
>> > the
>> > error message easier to find, but doesn't recommend a solution.
>> >
>> > `sys.platform`, `platform.linux_distribution` or `distro`[1] can be used
>> > to
>> > find which OS, platform, etc is and use that to suggest a command like:
>> >
>> >     if sys.version_info < (3, ) and sys.platform.startswith("linux") and
>> > platform.linux_distribution()[0] in ("ubuntu", "debian"):
>> >         print("Python headers were not found. On Debian/Ubuntu, `sudo
>> > apt-get install python-dev` should install the Python headers.")
>> >
>> >     elif sys.version_info < (3, ) and sys.platform.startswith("linux")
>> > and
>> > platform.linux_distribution()[0] in ("ubuntu", "debian"):
>> >         print("Python headers were not found. On Debian/Ubuntu, `sudo
>> > apt-get install python3-dev` should install the Python3 headers.")
>> >
>> > This would help so that non-developers can use cython based packages
>> > easily
>> > as it provides helpful instructions on what to do. Currently, you
>> > essentially search for the error message and go to one of the
>> > forums/blogs
>> > explaining what can be done.
>> >
>> > Or maybe by default it can just point the user to an Installation Page
>> > which
>> > explains in detail what needs to be done to get those headers for
>> > various
>> > systems ?
>> >
>> > [1] - https://pypi.python.org/pypi/distro
>> >
>> > _______________________________________________
>> > cython-devel mailing list
>> > cython-devel@...
>> > https://mail.python.org/mailman/listinfo/cython-devel
>> >
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel@...
>> https://mail.python.org/mailman/listinfo/cython-devel
>
>
>
> _______________________________________________
> cython-devel mailing list
> cython-devel@...
> https://mail.python.org/mailman/listinfo/cython-devel
>
Nathaniel Smith | 9 Jun 08:49 2016

Re: [Cython] Suggested action for "Python.h not found"

On Wed, Jun 8, 2016 at 10:37 PM, Elizabeth A. Fischer
<elizabeth.fischer@...> wrote:
> Spack gives you complete control over your compiler.  This is important if
> you're  building Python extensions, especially C++-based extensions, which
> must be built with the same compilers use to build Python.

Just to hopefully avoid confusing people -- you mostly only need to
worry about this on Windows. On OS X and Linux, I can't think of any
situation where using a different compiler for your extension and for
Python will cause a problem.

The one thing to watch out for these days is that if you have two
different C++ libraries that are being linked together (so this
doesn't apply to CPython itself -- it's not a C++ library), and the
interface between the libraries uses std::string or std::list, and
you're on Linux, then you need to be careful about the GCC 5 / C++11
ABI transition. The short version is that if you consistently use
versions of GCC <5 OR consistently use versions of GCC >=5, then you
should be fine; the long version can be found by googling :-).

-n

--

-- 
Nathaniel J. Smith -- https://vorpus.org
Abdeali Kothari | 8 Jun 08:28 2016
Picon
Gravatar

[Cython] Suggested action for "Python.h not found"

Hi,

Currently, when a user installs cython, if the Python headers are not found, an error message saying "Python.h: no such file or directory" is shown (Example: https://justpaste.it/v0gz). Would it be possible to suggest an action to install the headers ?

I found http://trac.cython.org/ticket/247 which is quite old, and makes the error message easier to find, but doesn't recommend a solution.

`sys.platform`, `platform.linux_distribution` or `distro`[1] can be used to find which OS, platform, etc is and use that to suggest a command like:

    if sys.version_info < (3, ) and sys.platform.startswith("linux") and platform.linux_distribution()[0] in ("ubuntu", "debian"):
        print("Python headers were not found. On Debian/Ubuntu, `sudo apt-get install python-dev` should install the Python headers.")
    
    elif sys.version_info < (3, ) and sys.platform.startswith("linux") and platform.linux_distribution()[0] in ("ubuntu", "debian"):
        print("Python headers were not found. On Debian/Ubuntu, `sudo apt-get install python3-dev` should install the Python3 headers.")

This would help so that non-developers can use cython based packages easily as it provides helpful instructions on what to do. Currently, you essentially search for the error message and go to one of the forums/blogs explaining what can be done.

Or maybe by default it can just point the user to an Installation Page which explains in detail what needs to be done to get those headers for various systems ?

<div><div dir="ltr">Hi,<div><br></div>
<div>Currently, when a user installs cython, if the Python headers are not found, an error message saying "Python.h: no such file or directory" is shown (Example: <a href="https://justpaste.it/v0gz">https://justpaste.it/v0gz</a>). Would it be possible to suggest an action to install the headers ?</div>
<div><br></div>
<div>I found&nbsp;<a href="http://trac.cython.org/ticket/247">http://trac.cython.org/ticket/247</a> which is quite old, and makes the error message easier to find, but doesn't recommend a solution.</div>
<div><br></div>
<div>`sys.platform`, `platform.linux_distribution` or `distro`[1] can be used to find which OS, platform, etc is and use that to suggest a command like:</div>
<div><br></div>
<div>&nbsp; &nbsp; if sys.version_info &lt; (3, ) and sys.platform.startswith("linux") and platform.linux_distribution()[0] in ("ubuntu", "debian"):</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; print("Python headers were not found. On Debian/Ubuntu, `sudo apt-get install python-dev` should install the Python headers.")</div>
<div>
<div>&nbsp; &nbsp;&nbsp;</div>
<div>&nbsp; &nbsp; elif sys.version_info &lt; (3, ) and sys.platform.startswith("linux") and platform.linux_distribution()[0] in ("ubuntu", "debian"):</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; print("Python headers were not found. On Debian/Ubuntu, `sudo apt-get install python3-dev` should install the Python3 headers.")</div>
</div>
<div><br></div>
<div>This would help so that non-developers can use cython based packages easily as it provides helpful instructions on what to do. Currently, you essentially search for the error message and go to one of the forums/blogs explaining what can be done.</div>
<div><br></div>
<div>Or maybe by default it can just point the user to an Installation Page which explains in detail what needs to be done to get those headers for various systems ?</div>
<div><br></div>
<div>[1] -&nbsp;<a href="https://pypi.python.org/pypi/distro">https://pypi.python.org/pypi/distro</a>
</div>
</div></div>
William Stein | 6 Jun 05:27 2016
Picon
Gravatar

[Cython] cython.org

Hi Cython devs,

Unfortunately, cython.org is down, since it was hosted at UW, and
pretty much all UW math computing infrastructure I have is dead.
I don't know why, though I suspect an attack of some sort.   I don't
know whether there are any backups of cython.org as it was hosted on
hardware that I didn't manage at all.    Sorry.  There is no ETA for
recovery.

 -- William

--

-- 
William (http://wstein.org)
Jeroen Demeyer | 31 May 12:28 2016
Picon

[Cython] Supporting cython.operator in C

The following code

cimport cython.operator
cdef long foo(long x, long y):
     return cython.operator.comma(x, y)

gives AttributeError: 'CBinopNode' object has no attribute 
'analyse_c_operation'

Is there any fundamental reason why such operators are only implemented 
for C++ and not pure C? What would be needed to support such operators in C?

Jeroen.
Yury Selivanov | 14 May 23:31 2016
Picon

[Cython] segfault in 'async def' coroutines

Hi,

Under some circumstances, in asyncio code that runs in uvloop [1],
cython code segfaults in cython/Cython/Utility/Coroutine.c:

   static PyObject *
   __Pyx_Coroutine_get_qualname(__pyx_CoroutineObject *self)
   {
     Py_INCREF(self->gi_qualname);
     return self->gi_qualname;
   }

"self->gi_qualname" can be NULL.  The correct code is probably:

   __Pyx_Coroutine_get_qualname(__pyx_CoroutineObject *self)
   {
     if (self->gi_qualname == NULL) { return __pyx_empty_unicode; }
     Py_INCREF(self->gi_qualname);
     return self->gi_qualname;
   }

Thanks,
Yury
Yury Selivanov | 15 May 00:28 2016
Picon

[Cython] async def coroutines miss __module__

In CPython:

 >>> async def foo(): pass
 >>> foo.__module__
'__main__'

In Cython, async def coroutines lack __module__ attribute

Yury
Jeroen Demeyer | 12 May 10:29 2016
Picon

[Cython] Illegal C code generated with numpy complex type

Current Cython git master breaks SageMath because Cython generates calls 
to non-existing functions like fabs_npy_float64(). Since this is a 
regression since Cython 0.24, I made a blocker ticket
http://trac.cython.org/ticket/879

Jeroen.
Jeroen Demeyer | 11 May 22:28 2016
Picon

[Cython] <at> staticmethod decorator applied twice

When investigating some issue with decorators, I noticed the following 
stange thing: it seems that the  <at> staticmethod decorator is actually 
applied twice in some cases. For example, the testsuite file 
tests/run/static_methods.pyx contains the following code:

cdef class A:
      <at> staticmethod
     def static_def(int x):
         ...

The relevant part of the Cython-generated code:

   __pyx_t_1 = 
PyCFunction_NewEx(&__pyx_mdef_14static_methods_1A_1static_def, NULL, 
__pyx_n_s_static_methods);
...
PyTuple_SET_ITEM(__pyx_t_2, 0, __pyx_t_1);
   __pyx_t_1 = __Pyx_PyObject_Call(__pyx_builtin_staticmethod, 
__pyx_t_2, NULL);
...
   if (PyDict_SetItem((PyObject 
*)__pyx_ptype_14static_methods_A->tp_dict, __pyx_n_s_static_def, 
__pyx_t_1) < 0)
...
   __pyx_t_1 = __Pyx_GetNameInClass((PyObject 
*)__pyx_ptype_14static_methods_A, __pyx_n_s_static_def);
...
PyTuple_SET_ITEM(__pyx_t_2, 0, __pyx_t_1);
   __pyx_t_1 = __Pyx_PyObject_Call(__pyx_builtin_staticmethod, 
__pyx_t_2, NULL);
...
   if (PyDict_SetItem((PyObject 
*)__pyx_ptype_14static_methods_A->tp_dict, __pyx_n_s_static_def, 
__pyx_t_1) < 0)

I might be missing something, but this really looks like Cython is 
applying the  <at> staticmethod decorator twice.

Gmane