General discussion about the Cython source code generator for Python extensions

Ryan Pessa | 8 May 2013 20:09

[Cython] nogil doesn't seem to work when defined on extern cpp functions

Hello,

I ran into an interesting problem today. It doesn't seem like Cython respects the `nogil` statement on extern cpp functions. I am trying to use a blocking I/O function, and am running it in secondary thread so I can use another library function to cancel it.

I have tried it both on the `extern` line:

cdef extern from "digitalpersona/dpfp_api.h" nogil:

uint32_t DPFPGetEvent(dp_uid_t* pDevUID, dp_device_event_t** ppEvent, uint32_t uTimeoutMsec)

and on the function itself:

cdef extern from "digitalpersona/dpfp_api.h" nogil:

uint32_t DPFPGetEvent(dp_uid_t* pDevUID, dp_device_event_t** ppEvent, uint32_t uTimeoutMsec) nogil

Either way, this statement still blocks other threads:

res = dpfp_api.DPFPGetEvent(pIdDev, &pEvent, dpfp_api.DP_TIMEOUT_INFINITE)

While this statement does not:

with nogil:

res = dpfp_api.DPFPGetEvent(pIdDev, &pEvent, dpfp_api.DP_TIMEOUT_INFINITE)

All variables used (res, pIdDev, pEvent) are C variables.

Obviously this is not a huge issue, as I can bypass the GIL using `with nogil`. But I figured it would be a good idea to report it anyway.

Ryan Pessa
AerisPOS Project Manager / Developer
Essential Elements | AerisPOS

<div><div dir="ltr">Hello,<div><br></div>
<div>I ran into an interesting problem today. It doesn't seem like Cython respects the `nogil` statement on extern cpp functions. I am trying to use a blocking I/O function, and am running it in secondary thread so I can use another library function to cancel it.</div>

<div><br></div>
<div>I have tried it both on the `extern` line:</div>
<div>&nbsp; &nbsp; cdef extern from "digitalpersona/dpfp_api.h" nogil:</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; uint32_t DPFPGetEvent(dp_uid_t* pDevUID, dp_device_event_t** ppEvent, uint32_t uTimeoutMsec)</div>

<div>and on the function itself:</div>
<div>
<div>&nbsp; &nbsp; cdef extern from "digitalpersona/dpfp_api.h" nogil:</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; uint32_t DPFPGetEvent(dp_uid_t* pDevUID, dp_device_event_t** ppEvent, uint32_t uTimeoutMsec) nogil</div>

</div>
<div><br></div>
<div>Either way, this statement still blocks other threads:</div>
<div>&nbsp; &nbsp; res = dpfp_api.DPFPGetEvent(pIdDev, &amp;pEvent, dpfp_api.DP_TIMEOUT_INFINITE)</div>
<div>While this statement does not:</div>
<div>
&nbsp; &nbsp; with nogil:</div>
<div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; res = dpfp_api.DPFPGetEvent(pIdDev, &amp;pEvent, dpfp_api.DP_TIMEOUT_INFINITE)</div>
<div>All variables used (res, pIdDev, pEvent) are C variables.</div>
<div><br></div>
<div>

Obviously this is not a huge issue, as I can bypass the GIL using `with nogil`. But I figured it would be a good idea to report it anyway.</div>
<div>
<p><span>Ryan Pessa</span><br><span>AerisPOS Project Manager / Developer</span><br><a href="http://www.essential-elements.net/" target="_blank"><span>Essential Elements</span></a> | <a href="http://www.aerispos.com/" target="_blank"><span>AerisPOS</span></a></p>

</div>
</div>
</div></div>

headers

Stefan Behnel | 6 May 2013 21:26

Re: [Cython] [cython] Order cdef classes by inheritance before generating the code. (#223)

Nikita Nemkin, 05.05.2013 16:13:
> cdef classes must be (topologically) ordered by inheritance to satisfy the following dependencies:
> 1) dealloc function definition: derived classes call base class dealloc;
> 2) vtable pointer initialization in module init function.
> Wrong order causes C/C++ compilation failure for 1) and segfault on import for 2).
> 
> Normally, source order is the correct order, but forward declarations can change this. See included test
case for example.
> I hit this problem because I forward declare all my classes in alphabetic order (and then include class
definitions from .pxi files, one class per file).
> 
> The simple (although not obvious) workaround is to reorder forward delarations, but I believe it&#39;s
worth fixing permanently.
> You can merge this Pull Request by running:
> 
>   git pull https://github.com/nnemkin/cython cdef_class_order
> 
> Or you can view, comment on it, or merge it online at:
> 
>   https://github.com/cython/cython/pull/223

Luckily, this broke the Sage build, by running into an infinite loop while
sorting.

https://sage.math.washington.edu:8091/hudson/job/sage-build/1778/

We clearly need more tests for extension type hierarchies spread across
multiple modules.

Stefan

headers

Stefan Behnel | 5 May 2013 13:03

Re: [Cython] [cython] Hide Cython utility classes (like memoryview) from Python level module scope. (#222)

Nikita Nemkin, 04.05.2013 19:52:
> Two changes included:
> 
> 1) cdef classes in utility code can have compiler directives attached to them. This is not used anywhere
ATM, but memoryviews may benefit from `` <at> cython.final``.
> 
> 2) All utility classes are excluded from module dictionary by *implicitly* marking them with
`` <at> cython.internal`` . This fixes [#775](http://trac.cython.org/cython_trac/ticket/775), test
is included.
> 
> I don&#39;t quite understand what CythonScope is and how utility classes are *supposed* to be hidden, but
as it is now, utility code scope is merged into main module scope and there is nothing special about its classes.
> 
> BTW if a user declares his own class with the same name as utility class (for example, ``memoryview``),
everything breaks down.
> You can merge this Pull Request by running:
> 
>   git pull https://github.com/nnemkin/cython cy_utility_decorators
> 
> Or you can view, comment on it, or merge it online at:
> 
>   https://github.com/cython/cython/pull/222

I wonder why utility classes actually need a Python name. Can't they just
live with None as "name"? All they should really need is their cname and a
properly analysed entry stored in the right places, so deleting their
Python visible name when merging them into the main module should fix this.

Stefan

headers

Stefan Behnel | 3 May 2013 09:29

[Cython] any more fixes for 0.19.1 ?

Hi,

since there's an unpleasant regression in 0.19, I'd like to release 0.19.1
soon. If you have any more fixes that should go in, you may want to apply
them now or at least speak up to delay the release.

I'm aware of ticket 810, but it doesn't seem like a regression and I don't
currently see an obvious simple way to reproduce it. It looks like a
classical "last reference dies while still in use" kind of problem to me.
Maybe even a bug in Sage rather than Cython that just accidentally worked
in older releases, although I'm not saying that there is nothing Cython
could do about it.

http://trac.cython.org/cython_trac/ticket/810

Ticket 814 (compiler crash during C++ code analysis) would also be worth
closing before a release as it might be an easy fix, even though it's not a
regression (crashes for me also in 0.18).

http://trac.cython.org/cython_trac/ticket/814

I also think that Ticket 775 ("memoryview" name leaks into module
namespace) should eventually be fixed as it's annoying and unexpected,
although not necessarily right in this release (it's been like this forever).

http://trac.cython.org/cython_trac/ticket/775

Anything else?

Stefan

headers

Lisandro Dalcin | 1 May 2013 09:14

[Cython] Regresion in 0.19

See the example below, the last hunk fails with Cython 0.19. It seems
Cython is confusing attribute access with constness.

CYTHON=cython

echo 'cdef import from *:' > defs.pxd
echo '    enum:SIZE'      >> defs.pxd

echo 'from defs cimport *'   > code1.pyx
echo 'cdef char buf[SIZE]'  >> code1.pyx
$CYTHON code1.pyx

echo 'cimport defs'              > code2.pyx
echo 'cdef char buf[defs.SIZE]' >> code2.pyx
$CYTHON code2.pyx

--
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

headers

eliswilson | 1 May 2013 00:06

[Cython] Biggest Fake Conference in Computer Science

Biggest Fake Conference in Computer Science

We are researchers from different parts of the world and conducted a study on the world’s biggest
bogus computer science conference WORLDCOMP http://sites.google.com/site/worlddump1
organized by Prof. Hamid Arabnia from University of Georgia, USA.

We submitted a fake paper to WORLDCOMP 2011 and again (the same paper with a modified title) to
WORLDCOMP 2012. This paper had numerous fundamental mistakes. Sample statements from that
paper include:

(1). Binary logic is fuzzy logic and vice versa
(2). Pascal developed fuzzy logic
(3). Object oriented languages do not exhibit any polymorphism or inheritance
(4). TCP and IP are synonyms and are part of OSI model
(5). Distributed systems deal with only one computer
(6). Laptop is an example for a super computer
(7). Operating system is an example for computer hardware

Also, our paper did not express any conceptual meaning. However, it was accepted both the times
without any modifications (and without any reviews) and we were invited to submit the final paper
and a payment of $500+ fee to present the paper. We decided to use the fee for better purposes than
making Prof. Hamid Arabnia richer. After that, we received few reminders from WORLDCOMP to pay
the fee but we never responded. This fake paper is different from the two fake papers already published
(see https://sites.google.com/site/worlddump4 for details) in WORLDCOMP.

We MUST say that you should look at the above website if you have any thoughts of participating in
WORLDCOMP. DBLP and other indexing agencies have stopped indexing WORLDCOMP’s proceedings
since 2011 due to its fakeness. See
http://www.informatik.uni-trier.de/~ley/db/conf/icai/index.html for
of one of the conferences of WORLDCOMP and notice that there is no listing after 2010. See Section 2 of
http://sites.google.com/site/dumpconf for comments from well-known researchers about
WORLDCOMP.

The status of your WORLDCOMP papers can be changed from scientific to other (i.e., junk or
non-technical) at any time. Better not to have a paper than having it in WORLDCOMP and spoil the
resume and peace of mind forever!

Our study revealed that WORLDCOMP is money making business, using University of Georgia mask, for
Prof. Hamid Arabnia. He is throwing out a small chunk of that money (around 20 dollars per paper
published in WORLDCOMP’s proceedings) to his puppet (Mr. Ashu Solo or A.M.G. Solo) who publicizes
WORLDCOMP and also defends it at various forums, using fake/anonymous names. The puppet uses
fake names and defames other conferences to divert traffic to WORLDCOMP. He also makes anonymous
phone calls and threatens the critiques of WORLDCOMP (See Item 7 of Section 5 of above website). That
is, the puppet does all his best to get a maximum number of papers published at WORLDCOMP to get
more money into his (and Prof. Hamid Arabnia’s) pockets. Prof. Hamid Arabnia makes a lot of tricks. For
example, he appeared in a newspaper to fool the public, claiming him a victim of cyber-attack (see Item
8 in Section 5 of above website).

Monte Carlo Resort (the venue of WORLDCOMP for more than 10 years, until 2012) has refused to
provide the venue for WORLDCOMP’13 because of the fears of their image being tarnished due to
WORLDCOMP’s fraudulent activities. That is why WORLDCOMP’13 is taking place at a different resort.
WORLDCOMP will not be held after 2013.

The draft paper submission deadline is over but still there are no committee members, no reviewers,
and there is no conference Chairman. The only contact details available on WORLDCOMP’s website is
just an email address!

We ask Prof. Hamid Arabnia to publish all reviews for all the papers (after blocking identifiable details)
since 2000 conference. Reveal the names and affiliations of all the reviewers (for each year) and how
many papers each reviewer had reviewed on average. We also ask him to look at the Open Challenge
(Section 6) at https://sites.google.com/site/moneycomp1 and respond if he has any professional values.

Sorry for posting to multiple lists. Spreading the word is the only way to stop this bogus conference.
Please forward this message to other mailing lists and people.

We are shocked with Prof. Hamid Arabnia and his puppet’s activities at
http://worldcomp-fake-bogus.blogspot.com Search Google using the keyword worldcomp fake for
additional links.

_______________________________________________
cython-devel mailing list
cython-devel <at> python.org
http://mail.python.org/mailman/listinfo/cython-devel

headers

David Hirschfeld | 29 Apr 2013 11:58

Re: [Cython] -O3 causes MinGW to segfault python

Forwarded because attachments were too large. The source files and generated .c file can now be viewed at:

https://gist.github.com/dhirschfeld/5480711

Is this a bug or is the recommendation to not build with -O3?

Build log, segfault and version info below.

Input files and generated .c file attached.

C:\temp> python setup.py build_ext --inplace --force

running build_ext

cythoning test_o3.pyx to test_o3.c

building 'test_o3' extension

C:\dev\bin\MinGW32\bin\gcc.exe -march=native -mdll -O3 -Wall

-IC:\dev\bin\Python27\include -IC:\dev\bin\Python27\PC

-c test_o3.c -o build\temp.win32-2.7\Release\test_o3.o

test_o3.c: In function '__Pyx_RaiseArgtupleInvalid':

test_o3.c:1002:18: warning: unknown conversion type character 'z' in format [-Wformat]

test_o3.c:1002:18: warning: format '%s' expects argument of type 'char *',

but argument 5 has type 'Py_ssize_t' [-Wformat]

test_o3.c:1002:18: warning: unknown conversion type character 'z' in format [-Wformat]

test_o3.c:1002:18: warning: too many arguments for format [-Wformat-extra-args]

writing build\temp.win32-2.7\Release\test_o3.def

C:\dev\bin\MinGW32\bin\gcc.exe -march=native -shared -s

build\temp.win32-2.7\Release\test_o3.o build\temp.win32-2.7\Release\test_o3.def

-LC:\dev\bin\Python27\libs -LC:\dev\bin\Python27\PCbuild -lpython27 -lmsvcr90 -o

C:\temp\test_o3.pyd

C:\temp> python

Python 2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit (Intel)] on win32

Type "help", "copyright", "credits" or "license" for more information.

>>> import Cython; Cython.__version__

'0.19'

>>> import faulthandler; faulthandler.enable()

>>> from test_o3 import f

>>> f(1., 2., 3., 4.)

Fatal Python error: Segmentation fault

Current thread 0x00001c38:

File "<stdin>", line 1 in <module>

C:\temp> gcc --version

gcc.exe (tdm-1) 4.7.1

Copyright (C) 2012 Free Software Foundation, Inc.

This is free software; see the source for copying conditions. There is NO

warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Thanks,

Dave

<div><div dir="ltr">
<div>Forwarded because attachments were too large. The source files and generated .c file can now be viewed at:</div>
<div>
<a href="https://gist.github.com/dhirschfeld/5480711">https://gist.github.com/dhirschfeld/5480711</a><br>
</div>
<div><br></div>
<br><div class="gmail_extra">
<div class="gmail_quote"><blockquote class="gmail_quote">
<div dir="ltr">
<div>Is this a bug or is the recommendation to not build with -O3?</div>
<div><br></div>
<div>Build log, segfault and version info below.</div>
<div>Input files and generated .c file attached.</div>
<div><br></div>
<div>
<div>C:\temp&gt; python setup.py build_ext --inplace --force</div>
<div>running build_ext</div>
<div>cythoning test_o3.pyx to test_o3.c</div>
<div>building 'test_o3' extension</div>
<div>C:\dev\bin\MinGW32\bin\gcc.exe -march=native -mdll -O3 -Wall&nbsp;</div>

<div>&nbsp; &nbsp; -IC:\dev\bin\Python27\include -IC:\dev\bin\Python27\PC&nbsp;</div>
<div>&nbsp; &nbsp; -c test_o3.c -o build\temp.win32-2.7\Release\test_o3.o</div>
<div>test_o3.c: In function '__Pyx_RaiseArgtupleInvalid':</div>
<div>test_o3.c:1002:18: warning: unknown conversion type character 'z' in format [-Wformat]</div>

<div>test_o3.c:1002:18: warning: format '%s' expects argument of type 'char *',&nbsp;</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; but argument 5 has type 'Py_ssize_t' [-Wformat]</div>
<div>test_o3.c:1002:18: warning: unknown conversion type character 'z' in format [-Wformat]</div>

<div>test_o3.c:1002:18: warning: too many arguments for format [-Wformat-extra-args]</div>
<div>writing build\temp.win32-2.7\Release\test_o3.def</div>
<div>C:\dev\bin\MinGW32\bin\gcc.exe -march=native -shared -s&nbsp;</div>
<div>

&nbsp; &nbsp; build\temp.win32-2.7\Release\test_o3.o build\temp.win32-2.7\Release\test_o3.def&nbsp;</div>
<div>&nbsp; &nbsp; -LC:\dev\bin\Python27\libs -LC:\dev\bin\Python27\PCbuild -lpython27 -lmsvcr90 -o&nbsp;</div>
<div>&nbsp; &nbsp; C:\temp\test_o3.pyd</div>
</div>
<div><br></div>
<div>
<div>C:\temp&gt; python</div>
<div>Python 2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit (Intel)] on win32</div>
<div>Type "help", "copyright", "credits" or "license" for more information.</div>

<div>
<div>&gt;&gt;&gt; import Cython; Cython.__version__<br>
</div>
<div>'0.19'</div>
</div>
<div>&gt;&gt;&gt; import faulthandler; faulthandler.enable()</div>
<div>&gt;&gt;&gt; from test_o3 import f</div>
<div>&gt;&gt;&gt; f(1., 2., 3., 4.)</div>

<div>Fatal Python error: Segmentation fault</div>
<div><br></div>
<div>Current thread 0x00001c38:</div>
<div>&nbsp; File "&lt;stdin&gt;", line 1 in &lt;module&gt;</div>
<div><br></div>
<div>C:\temp&gt; gcc --version</div>

<div>gcc.exe (tdm-1) 4.7.1</div>
<div>Copyright (C) 2012 Free Software Foundation, Inc.</div>
<div>This is free software; see the source for copying conditions. &nbsp;There is NO</div>
<div>warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.</div>

<div><br></div>
<div>Thanks,</div>
<div>Dave</div>
<div><br></div>
</div>
</div>
</blockquote></div>
<br>
</div>
</div></div>

headers

Nikita Nemkin | 26 Apr 2013 17:53

[Cython] A little bugtracker cleanup

Hi,

While browsing Cython's bugtracker I've found a few
issues that should be closed as fixed:

http://trac.cython.org/cython_trac/ticket/42
http://trac.cython.org/cython_trac/ticket/94
http://trac.cython.org/cython_trac/ticket/113
http://trac.cython.org/cython_trac/ticket/246
http://trac.cython.org/cython_trac/ticket/358

Best regards,
Nikita Nemkin

headers

Stefan Behnel | 26 Apr 2013 11:50

[Cython] Fwd: Re: Version of Cython packages in Ubuntu PPA

[forwarding]

-------- Original-Message --------
Subject: Re: Version of Cython packages in Ubuntu PPA
Date: Fri, 26 Apr 2013 07:24:48 +0000
From: Thomas Pietrowski <thopiekar...@...>

Hey Stefan,

I wished it would work. Talked about that with the maintainers of the
launchpad.net site. There is a option using tags as version number, but as
we are importing code from github this feature does not work 

So as the version changes it needs to be changed in the recipes:
https://code.launchpad.net/~thopiekar/+recipe/cython-py2+3-daily
https://code.launchpad.net/~thopiekar/+recipe/cython-py2-daily

You can also use the packaging files for doing that, but I decided to
overwrite it in the recipes, which might me easier and faster for you
changing it online.

Fell free to ask me, if any questions are still left 

Regards,
Thomas

Am 26.04.2013 08:34 schrieb Stefan Behnel:
> Hi,
>
> the packages in the Ubuntu PPA still have 0.18.0 in their name, although
> they actually build the current master, which is at 0.19.
>
> Is there a way to adapt that automatically?
>
> Stefan
>

headers

Stefan Behnel | 26 Apr 2013 08:34

[Cython] Version of Cython packages in Ubuntu PPA

Hi,

the packages in the Ubuntu PPA still have 0.18.0 in their name, although
they actually build the current master, which is at 0.19.

Is there a way to adapt that automatically?

Stefan

headers

Stefan Behnel | 26 Apr 2013 07:07

[Cython] regarding ticket #810 (segfault in Sage)

Hi,

looking at this ticket:

http://trac.cython.org/cython_trac/ticket/810

and at the underlying Sage ticket:

http://trac.sagemath.org/sage_trac/ticket/14452

ISTM that this crash is not actually related to cdef kwargs but simply to
the fact that the C code is now calling directly through the C layer and no
longer through Python. The major difference being that Python puts
arguments into a tuple that is kept alive for the whole timespan of the
call, whereas C just passes in the reference and forgets about it.

Since the method in question replaces references internally, it's quite
possible that it somehow deletes the last reference to the object that was
passed in. I wouldn't really know why, as we specifically guard against
these things since somewhere around 0.17 (and comment #7 in the Sage ticket
shows an example of that), but maybe this is just a corner case that we
failed to handle so far, or maybe the Sage code is doing something
'unexpected' here.

Stefan

23	May 2013
129	April 2013
87	March 2013
75	February 2013
31	January 2013
30	December 2012
33	November 2012
45	October 2012
56	September 2012
154	August 2012
128	July 2012
124	June 2012
279	May 2012
235	April 2012
158	March 2012
146	February 2012
125	January 2012
20	December 2011
43	November 2011
196	October 2011
116	September 2011
184	August 2011
245	July 2011
72	June 2011
220	May 2011
294	April 2011
217	March 2011
105	February 2011
200	January 2011
237	December 2010
377	November 2010
62	October 2010
124	September 2010
232	August 2010
288	July 2010
88	June 2010
135	May 2010
230	April 2010
222	March 2010
181	February 2010
111	January 2010
253	December 2009
245	November 2009
607	October 2009
360	September 2009
219	August 2009
206	July 2009
357	June 2009
639	May 2009
527	April 2009
678	March 2009
406	February 2009
181	January 2009
274	December 2008
291	November 2008
418	October 2008
256	September 2008
320	August 2008
215	July 2008
284	June 2008
705	May 2008
398	April 2008
223	March 2008
65	February 2008
133	January 2008
137	December 2007
82	November 2007
49	October 2007
151	September 2007
32	August 2007

Mon	Tue	Wed	Thu	Fri	Sat	Sun

		1	2	3	4	5
6	7	8	9	10	11	12
13	14	15	16	17	18	19
20	21	22	23	24	25	26
27	28	29	30	31