Yury V. Zaytsev | 25 Jul 15:27 2016

[Cython] (Possible) bug: module level global vars end up in upper scope?

Hi,

I've hit the following deviation from Python 2: when a Cython module has a 
global variable, somehow, upon importing the module, the global variables 
end up in the *current* (importing) module scope, rather than *imported* 
module scope, and, while normally relatively harmless, in the case of 
namedtuples, for instance, this has caused pickling errors in our 
production code.

test1.py / test2.pyx

     import collections

     Test = collections.namedtuple("Test", ["test"])

$ ipython
Python 2.7.6 (default, Jun 22 2015, 17:58:13)
IPython 1.2.1 -- An enhanced Interactive Python.

In [1]: import test1

In [2]: test1.Test
Out[2]: test1.Test

In [3]: import test2

In [4]: test2.Test
Out[4]: __main__.Test

I would appreciate any hints in the case that I'm deeply confused and 
(Continue reading)

Robert Bradshaw | 20 Jul 08:01 2016
Picon

[Cython] Cython infrastructure

Since the inception of the Cython project, William Stein's generously
allowed us to share Sage's hardware to host Cython's infrastructure,
which has been a great help. However, I think the time has come to
re-evaluate our options, which have actually improved a lot over the
last couple of years.

Note that this is not simply an issue of finding hardware, or raw VMs,
as we want to cut down on the personal maintenance costs as well (both
ongoing and emergency, neither of which are suited to a small number
of volunteers).

Several years ago we moved the main repositories over to github, and
the wiki was moved a while ago as well (to solve the spam problem).
However, the web site, trac (for bugs), and jenkins (for continuous
testing) are still on UW hardware. I propose we address them as
follows:

cython.org
Currently cython.org consists of a single landing page, a pile of
release tarballs, and all the documentation (which is 100% generated
from the repository). I propose we rely on github pages for the
(small) site, pypi for tarball archiving, and
http://cython.readthedocs.io (I recently got admin permission for) for
the documentation.

trac.cython.org
This is probably the most controversial, but I think it makes sense to
migrate to github issues. While clearly not as powerful, featureful,
or customizable as trac, it seems it would fulfill our modest needs. I
am happy to do the migration if no one objects.
(Continue reading)

swapnil jariwala | 15 Jul 14:11 2016
Picon

[Cython] bug report

Hello all,
trac.cython.org does not open for me, I've come across an error while trying the tutorial, I've elaborated the same in below question


any help would be highly appreciated.

Thanks
Swapnil
<div><div dir="ltr">Hello all,<div>
<a href="http://trac.cython.org">trac.cython.org</a> does not open for me, I've come across an error while trying the tutorial, I've elaborated the same in below question</div>
<div><br></div>
<div>
<a href="http://stackoverflow.com/questions/38386722/how-to-resolve-error-error-compiling-cython-file-error">http://stackoverflow.com/questions/38386722/how-to-resolve-error-error-compiling-cython-file-error</a><br>
</div>
<div><br></div>
<div>any help would be highly appreciated.</div>
<div><br></div>
<div>Thanks</div>
<div>Swapnil</div>
</div></div>
Jeroen Demeyer | 20 Jun 15:08 2016
Picon

[Cython] Virtual cpdef methods

Hello,

I would like to have a "virtual" cpdef method: something which allows a 
fast Cython call *when implemented* but without a default implementation.

The most simple implementation would be a "cpdef" method but without an 
entry for the method in the Python method table.

Of course, the "cdef" part needs to be implemented by the user (you need 
to put something in the vtab). For my purposes, something like "raise 
AttributeError" would be sufficient.

1. Can this be accomplished currently in Cython?

2. Is this something that makes sense to add to Cython?

3. What do you think of the following syntax?

cdef class MyClass(object):
     cpdef virtual_method(self):
         # implementation of "cdef" part ...

     del virtual_method  # don't put virtual_method in the method table
Nathan Goldbaum | 10 Jun 18:18 2016
Picon

[Cython] Static checker for cython extension dependencies?

Hi all,

I'm working on a pretty large python/cython codebase (yt) that has many interdependent C extensions written in cython.

I've found it to be pretty hit or miss to depend on contributors to manually update cython dependency information in our setup.py file. The dependency information seems to only be used by setuptools to trigger recompilation, but critically setuptools will happily compile a C extension for the first time if the depends information is incomplete. This means that during development, if I update a cython routine, there's no way to ensure that other cython routines that cimport the one I modified will be recompiled unless I manually ensure the depends information is updated whenever cython code gains or loses a cimport.

To make that more concrete, here's a pull request I just made to yt that adds missing dependencies for a cython header. Without this pull request, setuptools fails to recompile these routines when selection_routines.pxd changes, causing a build failure.


I think it should be possible to write a test for this by defining the dependency information outside of setup.py and parsing grep and looking for all cython files that cimport other cython files defined inside yt. However, before I do that, I'm curious whether anyone has done something similar, or if there is some other way of forcing the dependency information to be complete on the first compilation, rather than just for subsequent incremental recompilations during development.

Thanks for your help!

-Nathan
<div><div dir="ltr">Hi all,<div><br></div>
<div>I'm working on a pretty large python/cython codebase (yt) that has many interdependent C extensions written in cython.</div>
<div><br></div>
<div>I've found it to be pretty hit or miss to depend on contributors to manually update cython dependency information in our setup.py file. The dependency information seems to only be used by setuptools to trigger recompilation, but critically setuptools will happily compile a C extension for the first time if the depends information is incomplete. This means that during development, if I update a cython routine, there's no way to ensure that other cython routines that cimport the one I modified will be recompiled unless I manually ensure the depends information is updated whenever cython code gains or loses a cimport.</div>
<div><br></div>
<div>To make that more concrete, here's a pull request I just made to yt that adds missing dependencies for a cython header. Without this pull request, setuptools fails to recompile these routines when selection_routines.pxd changes, causing a build failure.</div>
<div><br></div>
<div>
<a href="https://bitbucket.org/yt_analysis/yt/pull-requests/2220">https://bitbucket.org/yt_analysis/yt/pull-requests/2220</a><br>
</div>
<div><br></div>
<div>I think it should be possible to write a test for this by defining the dependency information outside of setup.py and parsing grep and looking for all cython files that cimport other cython files defined inside yt. However, before I do that, I'm curious whether anyone has done something similar, or if there is some other way of forcing the dependency information to be complete on the first compilation, rather than just for subsequent incremental recompilations during development.</div>
<div><br></div>
<div>Thanks for your help!</div>
<div><br></div>
<div>-Nathan</div>
</div></div>
Erik Bray | 9 Jun 09:36 2016
Picon
Gravatar

Re: [Cython] Suggested action for "Python.h not found"

On Thu, Jun 9, 2016 at 7:37 AM, Elizabeth A. Fischer
<elizabeth.fischer@...> wrote:
>> Or maybe by default it can just point the user to an Installation Page
>> which explains in detail what needs to be done to get those headers for
>> various systems ?
>
>
> I think having this information on an installation instruction page is
> definitely a good idea.  I don't think it makes sense to put in every single
> error message that users should consult the installation instructions; that
> should be a given.

I'm inclined to agree that putting instructions for several different
systems in the message is overkill, and is a slippery slope, but a
friendlier error message with links to documentation (and maybe
general, non-Cython-specific docs on building Python extension modules
too).

> It is my belief that non-developers should not be manually building
> software.  Nor should developers be doing that; if you're saying
> "./configure" yourself, it's like programming assembly code.
>
> There are many ways to install Cython without manually building:

Unfortunately sometimes, especially with Python software, the line
between "developer" and "non-developer" is blurry.  This is especially
true of scientific software where Cython finds its most use.  It's not
uncommon, for example, for Cython to be pulled in as a build
dependency of something else.  Of course as pointed out downthread
this is less of an issue on most platforms now that binary wheels are
available for so many (including now manylinux!).

So I guess point taken that when it comes to just building Cython
itself this is less of an issue than it used to be.

> * If you're on Linux (and maybe Mac too), install with Spack.  This can be
> as easy as typing:
>          spack install py-cython
>
> https://github.com/llnl/spack
>
> Spack gives you complete control over your compiler.  This is important if
> you're  building Python extensions, especially C++-based extensions, which
> must be built with the same compilers use to build Python.
>
> I use Spack to build my complete software stack: the application software I
> work on requires about 50 dependencies to build, and another ~20 Python
> dependencies to analyze the output.  That includes the full Numpy/Scipy
> stack.  Spack makes it easy for me to use consistent compiler and libraries
> for Python and my libraries.  I've had success making Spack-based install
> instructions that others (non-developers) can follow and get my software
> stack installed on their systems:
>     https://github.com/citibeth/icebin

This one's new to me.  Will have to investigate--this could be
relevant to my interests.

Thanks,
Erik

> On Wed, Jun 8, 2016 at 1:25 PM, Robert Bradshaw <robertwb@...> wrote:
>>
>> +1, want to submit a pull request?
>>
>> On Tue, Jun 7, 2016 at 11:28 PM, Abdeali Kothari
>> <abdealikothari@...> wrote:
>> > Hi,
>> >
>> > Currently, when a user installs cython, if the Python headers are not
>> > found,
>> > an error message saying "Python.h: no such file or directory" is shown
>> > (Example: https://justpaste.it/v0gz). Would it be possible to suggest an
>> > action to install the headers ?
>> >
>> > I found http://trac.cython.org/ticket/247 which is quite old, and makes
>> > the
>> > error message easier to find, but doesn't recommend a solution.
>> >
>> > `sys.platform`, `platform.linux_distribution` or `distro`[1] can be used
>> > to
>> > find which OS, platform, etc is and use that to suggest a command like:
>> >
>> >     if sys.version_info < (3, ) and sys.platform.startswith("linux") and
>> > platform.linux_distribution()[0] in ("ubuntu", "debian"):
>> >         print("Python headers were not found. On Debian/Ubuntu, `sudo
>> > apt-get install python-dev` should install the Python headers.")
>> >
>> >     elif sys.version_info < (3, ) and sys.platform.startswith("linux")
>> > and
>> > platform.linux_distribution()[0] in ("ubuntu", "debian"):
>> >         print("Python headers were not found. On Debian/Ubuntu, `sudo
>> > apt-get install python3-dev` should install the Python3 headers.")
>> >
>> > This would help so that non-developers can use cython based packages
>> > easily
>> > as it provides helpful instructions on what to do. Currently, you
>> > essentially search for the error message and go to one of the
>> > forums/blogs
>> > explaining what can be done.
>> >
>> > Or maybe by default it can just point the user to an Installation Page
>> > which
>> > explains in detail what needs to be done to get those headers for
>> > various
>> > systems ?
>> >
>> > [1] - https://pypi.python.org/pypi/distro
>> >
>> > _______________________________________________
>> > cython-devel mailing list
>> > cython-devel@...
>> > https://mail.python.org/mailman/listinfo/cython-devel
>> >
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel@...
>> https://mail.python.org/mailman/listinfo/cython-devel
>
>
>
> _______________________________________________
> cython-devel mailing list
> cython-devel@...
> https://mail.python.org/mailman/listinfo/cython-devel
>
Nathaniel Smith | 9 Jun 08:49 2016

Re: [Cython] Suggested action for "Python.h not found"

On Wed, Jun 8, 2016 at 10:37 PM, Elizabeth A. Fischer
<elizabeth.fischer@...> wrote:
> Spack gives you complete control over your compiler.  This is important if
> you're  building Python extensions, especially C++-based extensions, which
> must be built with the same compilers use to build Python.

Just to hopefully avoid confusing people -- you mostly only need to
worry about this on Windows. On OS X and Linux, I can't think of any
situation where using a different compiler for your extension and for
Python will cause a problem.

The one thing to watch out for these days is that if you have two
different C++ libraries that are being linked together (so this
doesn't apply to CPython itself -- it's not a C++ library), and the
interface between the libraries uses std::string or std::list, and
you're on Linux, then you need to be careful about the GCC 5 / C++11
ABI transition. The short version is that if you consistently use
versions of GCC <5 OR consistently use versions of GCC >=5, then you
should be fine; the long version can be found by googling :-).

-n

--

-- 
Nathaniel J. Smith -- https://vorpus.org
Abdeali Kothari | 8 Jun 08:28 2016
Picon
Gravatar

[Cython] Suggested action for "Python.h not found"

Hi,

Currently, when a user installs cython, if the Python headers are not found, an error message saying "Python.h: no such file or directory" is shown (Example: https://justpaste.it/v0gz). Would it be possible to suggest an action to install the headers ?

I found http://trac.cython.org/ticket/247 which is quite old, and makes the error message easier to find, but doesn't recommend a solution.

`sys.platform`, `platform.linux_distribution` or `distro`[1] can be used to find which OS, platform, etc is and use that to suggest a command like:

    if sys.version_info < (3, ) and sys.platform.startswith("linux") and platform.linux_distribution()[0] in ("ubuntu", "debian"):
        print("Python headers were not found. On Debian/Ubuntu, `sudo apt-get install python-dev` should install the Python headers.")
    
    elif sys.version_info < (3, ) and sys.platform.startswith("linux") and platform.linux_distribution()[0] in ("ubuntu", "debian"):
        print("Python headers were not found. On Debian/Ubuntu, `sudo apt-get install python3-dev` should install the Python3 headers.")

This would help so that non-developers can use cython based packages easily as it provides helpful instructions on what to do. Currently, you essentially search for the error message and go to one of the forums/blogs explaining what can be done.

Or maybe by default it can just point the user to an Installation Page which explains in detail what needs to be done to get those headers for various systems ?

<div><div dir="ltr">Hi,<div><br></div>
<div>Currently, when a user installs cython, if the Python headers are not found, an error message saying "Python.h: no such file or directory" is shown (Example: <a href="https://justpaste.it/v0gz">https://justpaste.it/v0gz</a>). Would it be possible to suggest an action to install the headers ?</div>
<div><br></div>
<div>I found&nbsp;<a href="http://trac.cython.org/ticket/247">http://trac.cython.org/ticket/247</a> which is quite old, and makes the error message easier to find, but doesn't recommend a solution.</div>
<div><br></div>
<div>`sys.platform`, `platform.linux_distribution` or `distro`[1] can be used to find which OS, platform, etc is and use that to suggest a command like:</div>
<div><br></div>
<div>&nbsp; &nbsp; if sys.version_info &lt; (3, ) and sys.platform.startswith("linux") and platform.linux_distribution()[0] in ("ubuntu", "debian"):</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; print("Python headers were not found. On Debian/Ubuntu, `sudo apt-get install python-dev` should install the Python headers.")</div>
<div>
<div>&nbsp; &nbsp;&nbsp;</div>
<div>&nbsp; &nbsp; elif sys.version_info &lt; (3, ) and sys.platform.startswith("linux") and platform.linux_distribution()[0] in ("ubuntu", "debian"):</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; print("Python headers were not found. On Debian/Ubuntu, `sudo apt-get install python3-dev` should install the Python3 headers.")</div>
</div>
<div><br></div>
<div>This would help so that non-developers can use cython based packages easily as it provides helpful instructions on what to do. Currently, you essentially search for the error message and go to one of the forums/blogs explaining what can be done.</div>
<div><br></div>
<div>Or maybe by default it can just point the user to an Installation Page which explains in detail what needs to be done to get those headers for various systems ?</div>
<div><br></div>
<div>[1] -&nbsp;<a href="https://pypi.python.org/pypi/distro">https://pypi.python.org/pypi/distro</a>
</div>
</div></div>
William Stein | 6 Jun 05:27 2016
Picon
Gravatar

[Cython] cython.org

Hi Cython devs,

Unfortunately, cython.org is down, since it was hosted at UW, and
pretty much all UW math computing infrastructure I have is dead.
I don't know why, though I suspect an attack of some sort.   I don't
know whether there are any backups of cython.org as it was hosted on
hardware that I didn't manage at all.    Sorry.  There is no ETA for
recovery.

 -- William

--

-- 
William (http://wstein.org)
Jeroen Demeyer | 31 May 12:28 2016
Picon

[Cython] Supporting cython.operator in C

The following code

cimport cython.operator
cdef long foo(long x, long y):
     return cython.operator.comma(x, y)

gives AttributeError: 'CBinopNode' object has no attribute 
'analyse_c_operation'

Is there any fundamental reason why such operators are only implemented 
for C++ and not pure C? What would be needed to support such operators in C?

Jeroen.
Yury Selivanov | 14 May 23:31 2016
Picon

[Cython] segfault in 'async def' coroutines

Hi,

Under some circumstances, in asyncio code that runs in uvloop [1],
cython code segfaults in cython/Cython/Utility/Coroutine.c:

   static PyObject *
   __Pyx_Coroutine_get_qualname(__pyx_CoroutineObject *self)
   {
     Py_INCREF(self->gi_qualname);
     return self->gi_qualname;
   }

"self->gi_qualname" can be NULL.  The correct code is probably:

   __Pyx_Coroutine_get_qualname(__pyx_CoroutineObject *self)
   {
     if (self->gi_qualname == NULL) { return __pyx_empty_unicode; }
     Py_INCREF(self->gi_qualname);
     return self->gi_qualname;
   }

Thanks,
Yury

Gmane