headers
Nikita Nemkin | 26 May 2013 19:04
Picon

[Cython] Exception check optimization

Hi,

I wonder why is __pyx_filename (in exception check blocks)
tracked dynamically? AFAIK it's impossible to split function
body between multiple files (include only works at the top level),
which makes filename a compile time constant for any given function.

If the above is correct, __pyx_filename variable can be eliminated,
saving at least 5 bytes per exception check (more on x64).

On a related note, why is c_lineno enabled by default? So far I
have only found one pretty rare case when it is useful - debugging
autogenerated code in module init.
I suspect that most people don't bother turning it off and get
suboptimal code.

Best regards,
Nikita Nemkin
Permalink | Reply | 3 comments |
headers
Vitja Makarov | 24 May 2013 07:02
Picon
Gravatar

[Cython] How should be C-clobal deletion handled?

Hi!

Recently I've found that the following code causes segmentation fault:

cdef object f
del f
print f

So the question is: how should that work?

global objects are implicitly initialized to None and no CF and no cf analysis is performed for it.

So I see three options here:

1. prohibit cglobal deletion
2. set it back to None
3. check for a null value at every reference and assignment

--
vitja.
<div><div dir="ltr">Hi!<div><br></div>
<div>Recently I've found that the following code causes segmentation fault:</div>
<div><br></div>
<div>cdef object f</div>
<div>del f</div>
<div>print f</div>
<div>
<br>
</div>
<div>So the question is: how should that work?</div>
<div><br></div>
<div>global objects are implicitly initialized to None and no CF and no cf analysis is performed for it.</div>
<div><br></div>
<div>So I see three options here:</div>
<div><br></div>
<div>1. prohibit cglobal deletion</div>
<div>2. set it back to None</div>
<div>3. check for a null value at every reference and assignment</div>
<div><br></div>
<div>-- <br>vitja.<br>
</div>
</div></div>
Permalink | Reply | 7 comments |
headers
Nikita Nemkin | 22 May 2013 11:13
Picon

[Cython] Compiler crash fix

Hi,

Someone please apply this patch (too simple for pull request):

diff --git a/Cython/Compiler/ExprNodes.py b/Cython/Compiler/ExprNodes.py
index 0406fad..640d002 100755
--- a/Cython/Compiler/ExprNodes.py
+++ b/Cython/Compiler/ExprNodes.py
 <at>  <at>  -4993,7 +4993,7  <at>  <at>  class AttributeNode(ExprNode):
            # creates a corresponding NameNode and returns it, otherwise
            # returns None.
            type = self.obj.analyse_as_extension_type(env)
-        if type:
+        if type and type.scope:
                entry = type.scope.lookup_here(self.attribute)
                if entry and entry.is_cmethod:
                    if type.is_builtin_type:

It fixes CompilerCrash (None does not have "lookup_here" method)
that I have observed on two occasions:
1) cdef class Name; cimport Name; Name.attr
2) from X cimport Name; Name.attr  # cimport_from_pyx is active, Name is a  
class with errors

Makes me wonder if ErrorScope should be introduced to avoid None scope  
checks.

Best regards,
Nikita Nemkin
Permalink | Reply | 2 comments |
headers
Vitja Makarov | 21 May 2013 12:26
Picon
Gravatar

[Cython] CF based type inference

Hi!

Recently I've started work on new type inference engine. Now it's almost ready and I want to discuss it.

It works like this: first infer type for each assignment then for whole entry. It has some advantages over previous algorithm:
 - it handles assignment cycles, see test_swap() for example
 - it can infer type using info about assignments on the whole entry:

a = 1
b = a
a = "str"

a is python object and b is integer.

Here are testcases that show some new cases it can solve:


--
vitja.
<div><div dir="ltr">
<div>Hi!</div>
<div><br></div>
<div>Recently I've started work on new type inference engine. Now it's almost ready and I want to discuss it.</div>
<div><br></div>
<div>It works like this: first infer type for each assignment then for whole entry. It has some advantages over previous algorithm:<br>
</div>
<div>&nbsp;- it handles assignment cycles, see test_swap() for example</div>
<div>&nbsp;- it can infer type using info about assignments on the whole entry:</div>
<div><br></div>
<div>a = 1</div>
<div>
b = a</div>
<div>a = "str"</div>
<div><br></div>
<div>a is python object and b is integer.</div>
<div><br></div>
<div>Here are <span class="">testcases</span> that show some new cases it can solve:<br>
</div>
<div>
<a href="https://github.com/vitek/cython/blob/_type_inference_new/tests/run/type_inference_new.pyx">https://<span class="">github</span>.com/<span class="">vitek</span>/<span class="">cython</span>/blob/_type_inference_new/tests/run/type_inference_new.pyx</a><br>
</div>
<div><br></div>
<div>Here is branch for it&nbsp;<a href="https://github.com/vitek/cython/tree/_type_inference_new">https://<span class="">github</span>.com/<span class="">vitek</span>/<span class="">cython</span>/tree/_type_inference_new</a><br>
</div>
<div><br></div>-- <br><span class="">vitja</span>.<br>
</div></div>
Permalink | Reply | 9 comments |
headers
Ryan Pessa | 8 May 2013 20:09
Gravatar

[Cython] nogil doesn't seem to work when defined on extern cpp functions

Hello,

I ran into an interesting problem today. It doesn't seem like Cython respects the `nogil` statement on extern cpp functions. I am trying to use a blocking I/O function, and am running it in secondary thread so I can use another library function to cancel it.

I have tried it both on the `extern` line:
    cdef extern from "digitalpersona/dpfp_api.h" nogil:
        uint32_t DPFPGetEvent(dp_uid_t* pDevUID, dp_device_event_t** ppEvent, uint32_t uTimeoutMsec)
and on the function itself:
    cdef extern from "digitalpersona/dpfp_api.h" nogil:
        uint32_t DPFPGetEvent(dp_uid_t* pDevUID, dp_device_event_t** ppEvent, uint32_t uTimeoutMsec) nogil

Either way, this statement still blocks other threads:
    res = dpfp_api.DPFPGetEvent(pIdDev, &pEvent, dpfp_api.DP_TIMEOUT_INFINITE)
While this statement does not:
    with nogil:
        res = dpfp_api.DPFPGetEvent(pIdDev, &pEvent, dpfp_api.DP_TIMEOUT_INFINITE)
All variables used (res, pIdDev, pEvent) are C variables.

Obviously this is not a huge issue, as I can bypass the GIL using `with nogil`. But I figured it would be a good idea to report it anyway.

Ryan Pessa
AerisPOS Project Manager / Developer
Essential Elements | AerisPOS

<div><div dir="ltr">Hello,<div><br></div>
<div>I ran into an interesting problem today. It doesn't seem like Cython respects the `nogil` statement on extern cpp functions. I am trying to use a blocking I/O function, and am running it in secondary thread so I can use another library function to cancel it.</div>

<div><br></div>
<div>I have tried it both on the `extern` line:</div>
<div>&nbsp; &nbsp; cdef extern from "digitalpersona/dpfp_api.h" nogil:</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; uint32_t DPFPGetEvent(dp_uid_t* pDevUID, dp_device_event_t** ppEvent, uint32_t uTimeoutMsec)</div>

<div>and on the function itself:</div>
<div>
<div>&nbsp; &nbsp; cdef extern from "digitalpersona/dpfp_api.h" nogil:</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; uint32_t DPFPGetEvent(dp_uid_t* pDevUID, dp_device_event_t** ppEvent, uint32_t uTimeoutMsec) nogil</div>

</div>
<div><br></div>
<div>Either way, this statement still blocks other threads:</div>
<div>&nbsp; &nbsp; res = dpfp_api.DPFPGetEvent(pIdDev, &amp;pEvent, dpfp_api.DP_TIMEOUT_INFINITE)</div>
<div>While this statement does not:</div>
<div>
&nbsp; &nbsp; with nogil:</div>
<div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; res = dpfp_api.DPFPGetEvent(pIdDev, &amp;pEvent, dpfp_api.DP_TIMEOUT_INFINITE)</div>
<div>All variables used (res, pIdDev, pEvent) are C variables.</div>
<div><br></div>
<div>

Obviously this is not a huge issue, as I can bypass the GIL using `with nogil`. But I figured it would be a good idea to report it anyway.</div>
<div>
<p><span>Ryan Pessa</span><br><span>AerisPOS Project Manager / Developer</span><br><a href="http://www.essential-elements.net/" target="_blank"><span>Essential Elements</span></a> | <a href="http://www.aerispos.com/" target="_blank"><span>AerisPOS</span></a></p>

</div>
</div>
</div></div>
Permalink | Reply | 2 comments |
headers
Stefan Behnel | 6 May 2013 21:26
Picon
Favicon

Re: [Cython] [cython] Order cdef classes by inheritance before generating the code. (#223)

Nikita Nemkin, 05.05.2013 16:13:
> cdef classes must be (topologically) ordered by inheritance to satisfy the following dependencies:
> 1) dealloc function definition: derived classes call base class dealloc;
> 2) vtable pointer initialization in module init function.
> Wrong order causes C/C++ compilation failure for 1) and segfault on import for 2).
> 
> Normally, source order is the correct order, but forward declarations can change this. See included test
case for example.
> I hit this problem because I forward declare all my classes in alphabetic order (and then include class
definitions from .pxi files, one class per file).
> 
> The simple (although not obvious) workaround is to reorder forward delarations, but I believe it&#39;s
worth fixing permanently.
> You can merge this Pull Request by running:
> 
>   git pull https://github.com/nnemkin/cython cdef_class_order
> 
> Or you can view, comment on it, or merge it online at:
> 
>   https://github.com/cython/cython/pull/223

Luckily, this broke the Sage build, by running into an infinite loop while
sorting.

https://sage.math.washington.edu:8091/hudson/job/sage-build/1778/

We clearly need more tests for extension type hierarchies spread across
multiple modules.

Stefan
Permalink | Reply | 0 comments |
headers
Stefan Behnel | 5 May 2013 13:03
Picon
Favicon

Re: [Cython] [cython] Hide Cython utility classes (like memoryview) from Python level module scope. (#222)

Nikita Nemkin, 04.05.2013 19:52:
> Two changes included:
> 
> 1) cdef classes in utility code can have compiler directives attached to them. This is not used anywhere
ATM, but memoryviews may benefit from `` <at> cython.final``.
> 
> 2) All utility classes are excluded from module dictionary by *implicitly* marking them with
`` <at> cython.internal`` . This fixes [#775](http://trac.cython.org/cython_trac/ticket/775), test
is included.
> 
> I don&#39;t quite understand what CythonScope is and how utility classes are *supposed* to be hidden, but
as it is now, utility code scope is merged into main module scope and there is nothing special about its classes.
> 
> BTW if a user declares his own class with the same name as utility class (for example, ``memoryview``),
everything breaks down.
> You can merge this Pull Request by running:
> 
>   git pull https://github.com/nnemkin/cython cy_utility_decorators
> 
> Or you can view, comment on it, or merge it online at:
> 
>   https://github.com/cython/cython/pull/222

I wonder why utility classes actually need a Python name. Can't they just
live with None as "name"? All they should really need is their cname and a
properly analysed entry stored in the right places, so deleting their
Python visible name when merging them into the main module should fix this.

Stefan
Permalink | Reply | 2 comments |
headers
Stefan Behnel | 3 May 2013 09:29
Picon
Favicon

[Cython] any more fixes for 0.19.1 ?

Hi,

since there's an unpleasant regression in 0.19, I'd like to release 0.19.1
soon. If you have any more fixes that should go in, you may want to apply
them now or at least speak up to delay the release.

I'm aware of ticket 810, but it doesn't seem like a regression and I don't
currently see an obvious simple way to reproduce it. It looks like a
classical "last reference dies while still in use" kind of problem to me.
Maybe even a bug in Sage rather than Cython that just accidentally worked
in older releases, although I'm not saying that there is nothing Cython
could do about it.

http://trac.cython.org/cython_trac/ticket/810

Ticket 814 (compiler crash during C++ code analysis) would also be worth
closing before a release as it might be an easy fix, even though it's not a
regression (crashes for me also in 0.18).

http://trac.cython.org/cython_trac/ticket/814

I also think that Ticket 775 ("memoryview" name leaks into module
namespace) should eventually be fixed as it's annoying and unexpected,
although not necessarily right in this release (it's been like this forever).

http://trac.cython.org/cython_trac/ticket/775

Anything else?

Stefan
Permalink | Reply | 3 comments |
headers
Lisandro Dalcin | 1 May 2013 09:14
Picon
Gravatar

[Cython] Regresion in 0.19

See the example below, the last hunk fails with Cython 0.19. It seems
Cython is confusing attribute access with constness.

CYTHON=cython

echo 'cdef import from *:' > defs.pxd
echo '    enum:SIZE'      >> defs.pxd

echo 'from defs cimport *'   > code1.pyx
echo 'cdef char buf[SIZE]'  >> code1.pyx
$CYTHON code1.pyx

echo 'cimport defs'              > code2.pyx
echo 'cdef char buf[defs.SIZE]' >> code2.pyx
$CYTHON code2.pyx

--
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169
Permalink | Reply | 4 comments |
headers
eliswilson | 1 May 2013 00:06
Favicon

[Cython] Biggest Fake Conference in Computer Science

Biggest Fake Conference in Computer Science

We are researchers from different parts of the world and conducted a study on the world’s biggest 
bogus computer science conference WORLDCOMP  http://sites.google.com/site/worlddump1 
organized by Prof. Hamid Arabnia from University of Georgia, USA.

We submitted a fake paper to WORLDCOMP 2011 and again (the same paper with a modified title) to 
WORLDCOMP 2012. This paper had numerous fundamental mistakes. Sample statements from that 
paper include: 

(1). Binary logic is fuzzy logic and vice versa
(2). Pascal developed fuzzy logic
(3). Object oriented languages do not exhibit any polymorphism or inheritance
(4). TCP and IP are synonyms and are part of OSI model 
(5). Distributed systems deal with only one computer
(6). Laptop is an example for a super computer
(7). Operating system is an example for computer hardware

Also, our paper did not express any conceptual meaning.  However, it was accepted both the times 
without any modifications (and without any reviews) and we were invited to submit the final paper 
and a payment of $500+ fee to present the paper. We decided to use the fee for better purposes than 
making Prof. Hamid Arabnia richer. After that, we received few reminders from WORLDCOMP to pay 
the fee but we never responded. This fake paper is different from the two fake papers already published 
(see https://sites.google.com/site/worlddump4 for details) in WORLDCOMP.

We MUST say that you should look at the above website if you have any thoughts of participating in
WORLDCOMP.  DBLP and other indexing agencies have stopped indexing WORLDCOMP’s proceedings 
since 2011 due to its fakeness. See
http://www.informatik.uni-trier.de/~ley/db/conf/icai/index.html for
of one of the conferences of WORLDCOMP and notice that there is no listing after 2010. See Section 2 of
http://sites.google.com/site/dumpconf for comments from well-known researchers about 
WORLDCOMP. 

The status of your WORLDCOMP papers can be changed from scientific to other (i.e., junk or 
non-technical) at any time. Better not to have a paper than having it in WORLDCOMP and spoil the 
resume and peace of mind forever!

Our study revealed that WORLDCOMP is money making business, using University of Georgia mask, for 
Prof. Hamid Arabnia. He is throwing out a small chunk of that money (around 20 dollars per paper 
published in WORLDCOMP’s proceedings) to his puppet (Mr. Ashu Solo or A.M.G. Solo) who publicizes 
WORLDCOMP and also defends it at various forums, using fake/anonymous names. The puppet uses 
fake names and defames other conferences to divert traffic to WORLDCOMP. He also makes anonymous 
phone calls and threatens the critiques of WORLDCOMP (See Item 7 of Section 5 of above website). That 
is, the puppet does all his best to get a maximum number of papers published at WORLDCOMP to get 
more money into his (and Prof. Hamid Arabnia’s) pockets. Prof. Hamid Arabnia makes a lot of tricks. For 
example, he appeared in a newspaper to fool the public, claiming him a victim of cyber-attack (see Item 
8 in Section 5 of above website).

Monte Carlo Resort (the venue of WORLDCOMP for more than 10 years, until 2012) has refused to 
provide the venue for WORLDCOMP’13 because of the fears of their image being tarnished due to 
WORLDCOMP’s fraudulent activities. That is why WORLDCOMP’13 is taking place at a different resort. 
WORLDCOMP will not be held after 2013. 

The draft paper submission deadline is over but still there are no committee members, no reviewers, 
and there is no conference Chairman. The only contact details available on WORLDCOMP’s website is 
just an email address! 

We ask Prof. Hamid Arabnia to publish all reviews for all the papers (after blocking identifiable details) 
since 2000 conference. Reveal the names and affiliations of all the reviewers (for each year) and how 
many papers each reviewer had reviewed on average. We also ask him to look at the Open Challenge 
(Section 6) at https://sites.google.com/site/moneycomp1 and respond if he has any professional values.

Sorry for posting to multiple lists. Spreading the word is the only way to stop this bogus conference. 
Please forward this message to other mailing lists and people. 

We are shocked with Prof. Hamid Arabnia and his puppet’s activities at
http://worldcomp-fake-bogus.blogspot.com   Search Google using the keyword worldcomp fake for 
additional links.

_______________________________________________
cython-devel mailing list
cython-devel <at> python.org
http://mail.python.org/mailman/listinfo/cython-devel
Permalink | Reply | 0 comments |
headers
David Hirschfeld | 29 Apr 2013 11:58
Picon

Re: [Cython] -O3 causes MinGW to segfault python

Forwarded because attachments were too large. The source files and generated .c file can now be viewed at:


Is this a bug or is the recommendation to not build with -O3?

Build log, segfault and version info below.
Input files and generated .c file attached.

C:\temp> python setup.py build_ext --inplace --force
running build_ext
cythoning test_o3.pyx to test_o3.c
building 'test_o3' extension
C:\dev\bin\MinGW32\bin\gcc.exe -march=native -mdll -O3 -Wall 
    -IC:\dev\bin\Python27\include -IC:\dev\bin\Python27\PC 
    -c test_o3.c -o build\temp.win32-2.7\Release\test_o3.o
test_o3.c: In function '__Pyx_RaiseArgtupleInvalid':
test_o3.c:1002:18: warning: unknown conversion type character 'z' in format [-Wformat]
test_o3.c:1002:18: warning: format '%s' expects argument of type 'char *', 
                            but argument 5 has type 'Py_ssize_t' [-Wformat]
test_o3.c:1002:18: warning: unknown conversion type character 'z' in format [-Wformat]
test_o3.c:1002:18: warning: too many arguments for format [-Wformat-extra-args]
writing build\temp.win32-2.7\Release\test_o3.def
C:\dev\bin\MinGW32\bin\gcc.exe -march=native -shared -s 
    build\temp.win32-2.7\Release\test_o3.o build\temp.win32-2.7\Release\test_o3.def 
    -LC:\dev\bin\Python27\libs -LC:\dev\bin\Python27\PCbuild -lpython27 -lmsvcr90 -o 
    C:\temp\test_o3.pyd

C:\temp> python
Python 2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import Cython; Cython.__version__
'0.19'
>>> import faulthandler; faulthandler.enable()
>>> from test_o3 import f
>>> f(1., 2., 3., 4.)
Fatal Python error: Segmentation fault

Current thread 0x00001c38:
  File "<stdin>", line 1 in <module>

C:\temp> gcc --version
gcc.exe (tdm-1) 4.7.1
Copyright (C) 2012 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Thanks,
Dave


<div><div dir="ltr">
<div>Forwarded because attachments were too large. The source files and generated .c file can now be viewed at:</div>
<div>
<a href="https://gist.github.com/dhirschfeld/5480711">https://gist.github.com/dhirschfeld/5480711</a><br>
</div>
<div><br></div>
<br><div class="gmail_extra">
<div class="gmail_quote"><blockquote class="gmail_quote">
<div dir="ltr">
<div>Is this a bug or is the recommendation to not build with -O3?</div>
<div><br></div>
<div>Build log, segfault and version info below.</div>
<div>Input files and generated .c file attached.</div>
<div><br></div>
<div>
<div>C:\temp&gt; python setup.py build_ext --inplace --force</div>
<div>running build_ext</div>
<div>cythoning test_o3.pyx to test_o3.c</div>
<div>building 'test_o3' extension</div>
<div>C:\dev\bin\MinGW32\bin\gcc.exe -march=native -mdll -O3 -Wall&nbsp;</div>

<div>&nbsp; &nbsp; -IC:\dev\bin\Python27\include -IC:\dev\bin\Python27\PC&nbsp;</div>
<div>&nbsp; &nbsp; -c test_o3.c -o build\temp.win32-2.7\Release\test_o3.o</div>
<div>test_o3.c: In function '__Pyx_RaiseArgtupleInvalid':</div>
<div>test_o3.c:1002:18: warning: unknown conversion type character 'z' in format [-Wformat]</div>

<div>test_o3.c:1002:18: warning: format '%s' expects argument of type 'char *',&nbsp;</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; but argument 5 has type 'Py_ssize_t' [-Wformat]</div>
<div>test_o3.c:1002:18: warning: unknown conversion type character 'z' in format [-Wformat]</div>

<div>test_o3.c:1002:18: warning: too many arguments for format [-Wformat-extra-args]</div>
<div>writing build\temp.win32-2.7\Release\test_o3.def</div>
<div>C:\dev\bin\MinGW32\bin\gcc.exe -march=native -shared -s&nbsp;</div>
<div>

&nbsp; &nbsp; build\temp.win32-2.7\Release\test_o3.o build\temp.win32-2.7\Release\test_o3.def&nbsp;</div>
<div>&nbsp; &nbsp; -LC:\dev\bin\Python27\libs -LC:\dev\bin\Python27\PCbuild -lpython27 -lmsvcr90 -o&nbsp;</div>
<div>&nbsp; &nbsp; C:\temp\test_o3.pyd</div>
</div>
<div><br></div>
<div>
<div>C:\temp&gt; python</div>
<div>Python 2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit (Intel)] on win32</div>
<div>Type "help", "copyright", "credits" or "license" for more information.</div>

<div>
<div>&gt;&gt;&gt; import Cython; Cython.__version__<br>
</div>
<div>'0.19'</div>
</div>
<div>&gt;&gt;&gt; import faulthandler; faulthandler.enable()</div>
<div>&gt;&gt;&gt; from test_o3 import f</div>
<div>&gt;&gt;&gt; f(1., 2., 3., 4.)</div>

<div>Fatal Python error: Segmentation fault</div>
<div><br></div>
<div>Current thread 0x00001c38:</div>
<div>&nbsp; File "&lt;stdin&gt;", line 1 in &lt;module&gt;</div>
<div><br></div>
<div>C:\temp&gt; gcc --version</div>

<div>gcc.exe (tdm-1) 4.7.1</div>
<div>Copyright (C) 2012 Free Software Foundation, Inc.</div>
<div>This is free software; see the source for copying conditions. &nbsp;There is NO</div>
<div>warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.</div>

<div><br></div>
<div>Thanks,</div>
<div>Dave</div>
<div><br></div>
</div>
</div>
</blockquote></div>
<br>
</div>
</div></div>
Permalink | Reply | 2 comments |

Gmane