Erik Bray | 26 Apr 16:58 2016
Picon
Gravatar

[Cython] Cygwin: Handling missing C99 long double functions

Hi again,

Sorry if I'm spamming the list too much, but I've encountered another
pretty serious and unfortunate issue with Cython on Cygwin.

The problem now is that many of libm long double functions like sqrtl,
tanl, etc. are missing on Cygwin (missing from newlib to be specific).
I think this is a previously known issue, but nothing's ever really
been done about it.  Well, to be clear, sometimes they're present, but
only when sizeof(double) == sizeof(long double).  However on 64-bit
Cygwin sizeof(long double) == 16, so the functions are simply not
defined.

This seems to be due to lack of interest / effort:
https://www.cygwin.com/ml/cygwin/2011-04/msg00231.html  That post is 5
years old, but I can't find any evidence that this has changed.

There are quite a few tests in Cygwin's test suite that test long
double support.  I guess what I'm asking is what would be the best
thing to do about it.

I could just skip those tests on Cygwin, though I'm not sure the best
way to go about skipping an entire test for a given platform.

More generally though, outside the context of testing, this means
Cygwin will sometimes generate code that cannot be compiled on this
platform, and a question arises as to what to do about that.  I have
some thoughts, but am not sure if it's worth discussing any further or
not.

(Continue reading)

Erik Bray | 25 Apr 17:18 2016
Picon
Gravatar

[Cython] Minor issue with running tests and -Wlong-long

Hello,

As some of you already know I've been doing some work on Cython on
Cygwin [in the process I I'm constantly mixing the two up in speech,
but maybe in writing I'll do better :)].

There are several issues with the tests on Cygwin, and that's one
thing I'll work on.  But a major annoyance I've encountered when
running any tests is a huge number of warnings from gcc such as:

embray <at> PC-pret-47 ~/src/cython
$ CFLAGS="-O0" ./runtests.py -vv --no-cpp addloop
Python 2.7.10 (default, Jun  1 2015, 18:05:38)
[GCC 4.9.2]

Running tests against Cython 0.24 f68b5bd0fa620d0dc26166bffe5fe42d94068720
Backends: c

runTest (__main__.CythonRunTestCase)
compiling (c) and running addloop ...
=== C/C++ compiler error output: ===
In file included from /usr/include/python2.7/Python.h:58:0,
                 from addloop.c:4:
/usr/include/python2.7/pyport.h:69:27: warning: ISO C90 does not
support ‘long long’ [-Wlong-long]
 #define PY_LONG_LONG long long
                           ^
/usr/include/python2.7/pyport.h:793:34: note: in definition of macro
‘PyAPI_FUNC’
 #       define PyAPI_FUNC(RTYPE) RTYPE
(Continue reading)

Nathaniel Smith | 24 Apr 10:15 2016

[Cython] bug report on cython-mode.el: freezes when using which-function-mode

Hi all,

Bug report here -- trying to edit some cython code in emacs just now,
emacs was repeatedly freezing until I'd hit C-g repeatedly. Made
things literally unusable -- I couldn't type characters into the
buffer. M-x toggle-debug-on-quit gives the backtrace:

Debugger entered--Lisp error: (quit)
  syntax-ppss()
  python-nav-beginning-of-statement()
  cython-beginning-of-block()
  cython-current-defun()
  run-hook-with-args-until-success(cython-current-defun)
  which-function()
  which-func-update-1(#<window 3 on _http_parser.pyx>)
  which-func-update()
  apply(which-func-update nil)
  timer-event-handler([t 0 0 500000 t which-func-update nil idle 0])

Which strongly suggests that the problem has something to do with my
having which-function-mode enabled, and likely that something is wrong
with cython-current-defun. (which-function-mode is a minor mode built
into emacs.) Toggling which-function-mode off seems tentatively to
have fixed the problem. So there's a workaround, but really
cython-mode + which-function-mode shouldn't cause freezes :-).

Possible contributing factor: this emacs is built from a git snapshot
of master ("GNU Emacs 25.1.50.1 (x86_64-pc-linux-gnu, GTK+ Version
3.18.9) of 2016-04-22"), so it has the git versions of python-mode and
which-function-mode. (I'm just using the python.el that ships with
(Continue reading)

Isuru Fernando | 22 Apr 11:14 2016
Picon
Gravatar

[Cython] Cython compiler crash in 0.24

Hi,

When cythonizing a .pyx I get an error in Cython 0.24 which was not there in Cython 0.23.3


where a None object is passed and the following line calls the None object.


Let me know if you need more information.

Thanks,

Isuru Fernando



[ 33%] Cythonizing symengine_wrapper.pyx

Error compiling Cython file:
------------------------------------------------------------
...
    cdef double complex[::1] cmplx_view
    if real:
        try:
            real_view = iterable
        except (ValueError, TypeError):
            real_view = cython.view.array(shape=(_size(iterable),),
                                                     ^
------------------------------------------------------------

symengine_wrapper.pyx:2464:54: Compiler crash in TransformBuiltinMethods

ModuleNode.body = StatListNode(symengine_wrapper.pyx:1:0)
StatListNode.stats[163] = StatListNode(symengine_wrapper.pyx:2455:0)
StatListNode.stats[0] = DefNode(symengine_wrapper.pyx:2455:0,
    doc = ' if iterable supports the buffer interface: return iterable,\n        if not, return a cython.view.array object (which does) ',
    modifiers = [...]/0,
    name = 'with_buffer',
    num_required_args = 1,
    py_wrapper_required = True,
    reqd_kw_flags_cname = '0')
DefNode.body = StatListNode(symengine_wrapper.pyx:2456:4)
StatListNode.stats[0] = IfStatNode(symengine_wrapper.pyx:2460:4)
IfStatNode.if_clauses[0] = IfClauseNode(symengine_wrapper.pyx:2460:7)
IfClauseNode.body = StatListNode(symengine_wrapper.pyx:2461:8)
StatListNode.stats[0] = TryExceptStatNode(symengine_wrapper.pyx:2461:8)
TryExceptStatNode.except_clauses[0] = ExceptClauseNode(symengine_wrapper.pyx:2463:8)
ExceptClauseNode.body = StatListNode(symengine_wrapper.pyx:2464:12,
    is_terminator = True)
StatListNode.stats[0] = SingleAssignmentNode(symengine_wrapper.pyx:2464:41)
SingleAssignmentNode.rhs = GeneralCallNode(symengine_wrapper.pyx:2464:41,
    result_is_used = True,
    use_managed_ref = True)
File 'ExprNodes.py', line 8035, in compile_time_value: DictNode(symengine_wrapper.pyx:2464:47,
    is_dict_literal = True,
    is_temp = 1,
    obj_conversion_errors = [...]/0,
    reject_duplicates = True,
    result_is_used = True,
    use_managed_ref = True)
File 'ExprNodes.py', line 7334, in compile_time_value: TupleNode(symengine_wrapper.pyx:2464:49,
    is_sequence_constructor = 1,
    result_is_used = True,
    use_managed_ref = True)
File 'ExprNodes.py', line 6730, in compile_time_value_list: TupleNode(symengine_wrapper.pyx:2464:49,
    is_sequence_constructor = 1,
    result_is_used = True,
    use_managed_ref = True)
File 'ExprNodes.py', line 4981, in compile_time_value: SimpleCallNode(symengine_wrapper.pyx:2464:54,
    result_is_used = True,
    use_managed_ref = True)
File 'ExprNodes.py', line 1783, in compile_time_value: NameNode(symengine_wrapper.pyx:2464:54,
    cf_maybe_null = True,
    is_name = True,
    name = '_size',
    result_is_used = True,
    use_managed_ref = True)

Compiler crash traceback from this point on:
  File "/home/isuru/miniconda3/envs/test-cython/lib/python3.5/site-packages/Cython/Compiler/ExprNodes.py", line 1781, in compile_time_value
    return denv.lookup(self.name)
AttributeError: 'NoneType' object has no attribute 'lookup'


<div><div dir="ltr">Hi,<br><br>When cythonizing a .pyx I get an error in Cython 0.24 which was not there in Cython 0.23.3<div><br></div>
<div>After printing the stacktrace, this seems to be because of this line&nbsp;<a href="https://github.com/cython/cython/commit/6d55fd189f6ee9d4374d00b8c9c320bd04332bab#diff-28c66ef9e2ff564619ef82aa9d72ee7dR2762" target="_blank">https://github.com/cython/cython/commit/6d55fd189f6ee9d4374d00b8c9c320bd04332bab#diff-28c66ef9e2ff564619ef82aa9d72ee7dR2762</a>
</div>
<div><br></div>
<div>where a None object is passed and the following line calls the None object.<br>
</div>
<div><br></div>
<div>
<a href="https://github.com/cython/cython/blob/6d55fd189f6ee9d4374d00b8c9c320bd04332bab/Cython/Compiler/ExprNodes.py#L1780" target="_blank">https://github.com/cython/cython/blob/6d55fd189f6ee9d4374d00b8c9c320bd04332bab/Cython/Compiler/ExprNodes.py#L1780</a><br>
</div>
<div><br></div>
<div>Let me know if you need more information.</div>
<div>
<br>Thanks,</div>
<div><br></div>
<div>Isuru Fernando</div>
<div><br></div>
<div><br></div>
<div><br></div>
<div>
<div>[ 33%] Cythonizing symengine_wrapper.pyx</div>
<div><br></div>
<div>Error compiling Cython file:</div>
<div>------------------------------------------------------------</div>
<div>...</div>
<div>&nbsp; &nbsp; cdef double complex[::1] cmplx_view</div>
<div>&nbsp; &nbsp; if real:</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; try:</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; real_view = iterable</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; except (ValueError, TypeError):</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; real_view = cython.view.array(shape=(_size(iterable),),</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;^</div>
<div>------------------------------------------------------------</div>
<div><br></div>
<div>symengine_wrapper.pyx:2464:54: Compiler crash in TransformBuiltinMethods</div>
<div><br></div>
<div>ModuleNode.body = StatListNode(symengine_wrapper.pyx:1:0)</div>
<div>StatListNode.stats[163] = StatListNode(symengine_wrapper.pyx:2455:0)</div>
<div>StatListNode.stats[0] = DefNode(symengine_wrapper.pyx:2455:0,</div>
<div>&nbsp; &nbsp; doc = ' if iterable supports the buffer interface: return iterable,\n &nbsp; &nbsp; &nbsp; &nbsp;if not, return a cython.view.array object (which does) ',</div>
<div>&nbsp; &nbsp; modifiers = [...]/0,</div>
<div>&nbsp; &nbsp; name = 'with_buffer',</div>
<div>&nbsp; &nbsp; num_required_args = 1,</div>
<div>&nbsp; &nbsp; py_wrapper_required = True,</div>
<div>&nbsp; &nbsp; reqd_kw_flags_cname = '0')</div>
<div>DefNode.body = StatListNode(symengine_wrapper.pyx:2456:4)</div>
<div>StatListNode.stats[0] = IfStatNode(symengine_wrapper.pyx:2460:4)</div>
<div>IfStatNode.if_clauses[0] = IfClauseNode(symengine_wrapper.pyx:2460:7)</div>
<div>IfClauseNode.body = StatListNode(symengine_wrapper.pyx:2461:8)</div>
<div>StatListNode.stats[0] = TryExceptStatNode(symengine_wrapper.pyx:2461:8)</div>
<div>TryExceptStatNode.except_clauses[0] = ExceptClauseNode(symengine_wrapper.pyx:2463:8)</div>
<div>ExceptClauseNode.body = StatListNode(symengine_wrapper.pyx:2464:12,</div>
<div>&nbsp; &nbsp; is_terminator = True)</div>
<div>StatListNode.stats[0] = SingleAssignmentNode(symengine_wrapper.pyx:2464:41)</div>
<div>SingleAssignmentNode.rhs = GeneralCallNode(symengine_wrapper.pyx:2464:41,</div>
<div>&nbsp; &nbsp; result_is_used = True,</div>
<div>&nbsp; &nbsp; use_managed_ref = True)</div>
<div>File 'ExprNodes.py', line 8035, in compile_time_value: DictNode(symengine_wrapper.pyx:2464:47,</div>
<div>&nbsp; &nbsp; is_dict_literal = True,</div>
<div>&nbsp; &nbsp; is_temp = 1,</div>
<div>&nbsp; &nbsp; obj_conversion_errors = [...]/0,</div>
<div>&nbsp; &nbsp; reject_duplicates = True,</div>
<div>&nbsp; &nbsp; result_is_used = True,</div>
<div>&nbsp; &nbsp; use_managed_ref = True)</div>
<div>File 'ExprNodes.py', line 7334, in compile_time_value: TupleNode(symengine_wrapper.pyx:2464:49,</div>
<div>&nbsp; &nbsp; is_sequence_constructor = 1,</div>
<div>&nbsp; &nbsp; result_is_used = True,</div>
<div>&nbsp; &nbsp; use_managed_ref = True)</div>
<div>File 'ExprNodes.py', line 6730, in compile_time_value_list: TupleNode(symengine_wrapper.pyx:2464:49,</div>
<div>&nbsp; &nbsp; is_sequence_constructor = 1,</div>
<div>&nbsp; &nbsp; result_is_used = True,</div>
<div>&nbsp; &nbsp; use_managed_ref = True)</div>
<div>File 'ExprNodes.py', line 4981, in compile_time_value: SimpleCallNode(symengine_wrapper.pyx:2464:54,</div>
<div>&nbsp; &nbsp; result_is_used = True,</div>
<div>&nbsp; &nbsp; use_managed_ref = True)</div>
<div>File 'ExprNodes.py', line 1783, in compile_time_value: NameNode(symengine_wrapper.pyx:2464:54,</div>
<div>&nbsp; &nbsp; cf_maybe_null = True,</div>
<div>&nbsp; &nbsp; is_name = True,</div>
<div>&nbsp; &nbsp; name = '_size',</div>
<div>&nbsp; &nbsp; result_is_used = True,</div>
<div>&nbsp; &nbsp; use_managed_ref = True)</div>
<div><br></div>
<div>Compiler crash traceback from this point on:</div>
<div>&nbsp; File "/home/isuru/miniconda3/envs/test-cython/lib/python3.5/site-packages/Cython/Compiler/ExprNodes.py", line 1781, in compile_time_value</div>
<div>&nbsp; &nbsp; return denv.lookup(<a href="http://self.name" target="_blank">self.name</a>)</div>
<div>AttributeError: 'NoneType' object has no attribute 'lookup'<br><br><br>
</div>
</div>
</div></div>
Xuancong Wang | 19 Apr 11:13 2016
Picon

[Cython] unsupported meta-programming-related features

Dear Cython developers,

Python supports meta-programming, in which a variable with name
specified in a string can be created at run-time. One built-in library
which make use of this is argparse.

For example:

parser.add_argument('-N', '--max_threads', help='maximum number of
concurrent decoding threads', type=int, default=16)

in this case, the variable max_threads is created from the string
argument. And then Cython will generate an incorrect C program with
the following error:

smt.py:78:88: undeclared name not builtin: headtail
smt.c:1:2: error: #error Do not use this file, it is the result of a
failed Cython compilation.

In comparison, I found that nuitka can convert this kind of Python
programs sucessfully. I hope Cython can be improved. Thanks!

Cheers,
Xuancong
Stefan Behnel | 14 Apr 18:15 2016
Picon

[Cython] PEP 509: detect dict modification by version tag

Hi!

This new PEP seems interesting for Cython optimisations, too:

https://www.python.org/dev/peps/pep-0509/

Essentially, it adds a 64 bit modification counter to dicts that allows
detecting unmodified dicts, e.g. during lookups of methods or globals.

It's currently proposed for Py3.6.

Stefan
Manuel Nuno Melo | 13 Apr 21:35 2016
Picon

[Cython] Cannot cythonize subclasses of setuptools.extension._Extension

Hello devs,

I'm developing the setup.py for a scientific package, MDAnalysis (see PR #799). We depend on distutils and setuptool. Namely, we use setuptools.extension.Extension class for our extensions.

Some older versions of setuptools (<18.0) do filename cythonization themselves upon initialization of the Extension object.

Because we want to control name cythonization ourselves I try to directly use distutils.extension.Extension, which has none of setuptools' cythonization. However, this doesn't work because setuptools patches distutils, so that distutils.extension.Extension effectively becomes setuptools.extension.Extension.

setuptools does provide a copy of the unpatched Extension class, via setuptools.extension._Extension. Yet, if I attempt to pass instances of this class to Cython.cythonize it fails because my extension parent no longer conforms to one of the expected classes:

  File "./setup.py", line 283, in cythonize
    new_ext_modules = Cython.cythonize(self.ext_modules)
  File "/scratch/virtualenv/lib/python2.7/site-packages/Cython-0.24-py2.7-linux-x86_64.egg/Cython/Build/Dependencies.py", line 796, in cythonize
    aliases=aliases)
  File "/scratch/virtualenv/lib/python2.7/site-packages/Cython-0.24-py2.7-linux-x86_64.egg/Cython/Build/Dependencies.py", line 686, in create_extension_list
    raise TypeError(msg)
TypeError: pattern is not of type str nor subclass of Extension (<class setuptools.extension.Extension at 0x7f66be4e2a78>) but of type <type 'instance'> and class distutils.extension.Extension

I believe cythonize should be more permissive with the extension parent classes it accepts, or at least have the setuptools _Extension one in the whitelist, since it's the only way to access the original distutils class. What do you think?

In the meantime I'll solve this by subclassing setuptools.extension.Extension and making sure it doesn't cythonize anything.

Cheers,
Manuel
<div><div dir="ltr">
<div class="gmail_default">Hello devs,<br><br>
</div>
<div class="gmail_default">I'm developing the setup.py for a scientific package, <a href="https://github.com/MDAnalysis/mdanalysis/">MDAnalysis</a> (see <a href="https://github.com/MDAnalysis/mdanalysis/pull/799">PR #799</a>). We depend on distutils and setuptool. Namely, we use setuptools.extension.Extension class for our extensions.<br><br>Some older versions of setuptools (&lt;18.0) do filename cythonization themselves upon initialization of the Extension object. <br><br>Because we want to control name cythonization ourselves I try to directly use distutils.extension.Extension, which has none of setuptools' cythonization. However, this doesn't work because setuptools patches distutils, so that distutils.extension.Extension effectively becomes setuptools.extension.Extension.<br><br>
</div>
<div class="gmail_default">setuptools does provide a copy of the unpatched Extension class, via setuptools.extension._Extension. Yet, if I attempt to pass instances of this class to Cython.cythonize it fails because my extension parent no longer conforms to one of the expected classes:<br><br>
</div>
<div class="gmail_default">&nbsp; File "./setup.py", line 283, in cythonize<br>&nbsp;&nbsp;&nbsp; new_ext_modules = Cython.cythonize(self.ext_modules)<br>&nbsp; File "/scratch/virtualenv/lib/python2.7/site-packages/Cython-0.24-py2.7-linux-x86_64.egg/Cython/Build/Dependencies.py", line 796, in cythonize<br>&nbsp;&nbsp;&nbsp; aliases=aliases)<br>&nbsp; File "/scratch/virtualenv/lib/python2.7/site-packages/Cython-0.24-py2.7-linux-x86_64.egg/Cython/Build/Dependencies.py", line 686, in create_extension_list<br>&nbsp;&nbsp;&nbsp; raise TypeError(msg)<br>TypeError: pattern is not of type str nor subclass of Extension (&lt;class setuptools.extension.Extension at 0x7f66be4e2a78&gt;) but of type &lt;type 'instance'&gt; and class distutils.extension.Extension<br><br>
</div>
<div class="gmail_default">I believe cythonize should be more permissive with the extension parent classes it accepts, or at least have the setuptools _Extension one in the whitelist, since it's the only way to access the original distutils class. What do you think?<br><br>
</div>
<div class="gmail_default">In the meantime I'll solve this by subclassing setuptools.extension.Extension and making sure it doesn't cythonize anything.<br>
</div>
<div class="gmail_default"><br></div>
<div class="gmail_default">Cheers,<br>
</div>
<div class="gmail_default">Manuel<br clear="all">
</div>
</div></div>
Jeroen Demeyer | 12 Apr 10:36 2016
Picon

[Cython] cdef public declarations

(this thread is related to the thread "Question about how best require 
compiler options for C sources")

I have a question for Cython users and developers: are you sure that 
"cdef public" actually works as documented at 
http://docs.cython.org/src/userguide/external_C_code.html#public-declarations

I couldn't find it tested in the Cython testsuite, although it's not 
easy to know what to grep for. I hightly doubt that it works on OS X 
because of the difference between shared libraries (with .dylib 
extension) and loadable modules/bundles (with .so extension).
Nathaniel Smith | 11 Apr 14:51 2016

Re: [Cython] Fwd: Question about how best require compiler options for C sources

On Apr 11, 2016 04:18, "Erik Bray" <erik.m.bray-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
> On Fri, Apr 8, 2016 at 5:49 PM, Nathaniel Smith <njs-DvPbftppVdQdnm+yROfE0A@public.gmane.org> wrote:
> > Can you give a tiny concrete example? My questions are basic enough that I
> > feel like I'm missing something fundamental :-)
>
> Yes, I think you might be missing something, but I'm not sure exactly
> where.  In the issue I provided a tiny example which you can see here:
>
> https://gist.github.com/embray/12a67edb82b213217e31f408007898e6
>
> The C code generated by this example currently does not compile on
> Windows, because of how Cython uses DL_IMPORT incorrectly.  Regardless
> of what it *does* do, Cython should not be using the DL_IMPORT macro
> at all actually since that no longer even exists in Python.

Sure.

For that example, the correct thing to do is to *not* export the function.

Backing up, to make sure we're on the same page: There are three levels of symbol visibility in C: file-internal, shared library internal (different .c files that make up the library can see it, but users of the shared library can't see it), and shared library exported (everyone can see it; can also carry other consequences, e.g. on Linux then internal calls will become noticeably slower, and it becomes possible for weird symbol interposition issues to occur). So the rule of thumb is to make everything as private as you can get away with.

Making this more interesting:
- vanilla C only includes 2 ways to mark symbol visibility, which is not enough to make a 3-way distinction. Hence the need for an extra attribute thingummy.
- everyone agrees that symbols marked 'static' should be file-internal, but different platforms disagree about what should happen if the extra attribute thingummy is missing.

So on Windows, the convention is:
'static' -> file internal
no marking -> shared library internal
'dllexport' -> public

And on Linux it's:
'static' -> file internal
'visibility (hidden)' -> shared library internal
no marking -> public

It's generally agreed that Linux got this wrong and that you should always use '-fvisibility=hidden' to switch it to the windows style, but cython doesn't control compiler options and thus should probably generate code that works correctly regardless of such compiler settings. Fortunately, Linux does provide some markings to explicitly make things public: you can mark a symbol 'visibility (default)' (which means public), or you can use the dllexport syntax, just like Windows, because gcc is helpful like that.

OTOH, windows is annoying because of this dllimport thing that started this whole thread: on other systems just marking symbols as extern is enough to handle both shared-object-internal and shared-library-exported symbols, and the linker will sort it out. On Windows, you have to explicitly distinguish between these. (And annoyingly, if you accidentally leave out the dllimport making on functions then it will use some fallback hack that works but silently degrades performance; on other symbols it just doesn't work, and ditto for if you use it when it isn't needed.)

So final conclusion: for non-static symbols, cython should first decide whether they are supposed to be shared-library-internal or actually exported from the shared library.

For shared-library-internal symbols: their definition should be marked 'visibility(hidden)' on Linux, and unmarked on Windows. This is easy using some preprocessor gunk. (Or maybe simplest is: marked that everywhere except if using msvc, because I think everyone else will understand 'visibility (hidden)' even if it's a no op.) Their declaration in the header file should just be 'extern' everywhere.

For shared-library-exported symbols: I am dubious about whether cython should even support these at all. But if it does, then the definitions should be marked 'dllexport' (no macro trickery needed, because everyone understands this syntax), and their declaration in the header file needs some extra hack that is the subject of this thread.

Now, back to your example: Here the caller and callee are both compiled into the same shared library, so you don't want dllexport/dllimport at all, you just want a shared-library-internal symbol, which as we see is much easier.

NumPy also ran into some problems with this in our experiments with using cython internally. Our temporary solution was to use the preprocessor to monkeypatch DL_IMPORT into expanding to the appropriate shared-library-internal thing :-).

> > My first question is why you even need this, since AFAIK there are no cases
> > where it is correct to have a cython module dllexporting symbols that appear
> > in header files. This is what cimport is for, right?
>
> I don't think this has anything to do with cimport.  Could you explain
> what you mean?

We only need to solve the dllimport issue if we want to support shared-library-exported symbols from cython extensions. This is only useful if you have different extensions that are directly linking to each other using the platform linker. But this simply doesn't work in almost any cases, because platform linkers have all kinds of quirks that don't play well with python packages -- to start with, your runtime linker probably does not follow Python's rules for which directories to search to find a shared library... To solve these problems, the Cython devs invented the cimport mechanism, which is basically a portable, python-centric way of doing shared-library-exports while avoiding all the problems caused by using the platform linker. So my question was, what's your use case that's better served by linking directly against an extension module and using the platform's shared-library-export functionality, instead of cython's?

> > My second question is why you would want to do this via the command line,
> > when compiling the dll means that you are compiling some cython-generated
> > .c, which means that you can put the #define directly in the source code,
> > no?
>
> Not necessarily. As you can see in my example the file fooutil.c is
> hand-written C code that was not generated by Cython, but which uses a
> function in the Cython-generated code.  It includes "foo.h".  In
> principle you're right--the hand-written C code could set the proper
> #defines but it would have to do so *before* including "foo.h".  It's
> not very obvious that this would be needed.  Most other code I've seen
> that addresses this issue--including cpython itself--do so by passing
> an appropriate define to the compiler via the command-line, and that
> seems the clearest to me.

I see, right. I guess my suggestion would be that if a symbol really does need to be marked for shared-library-export *and simultaneously* used by different files within the same shared library -- which is the only case where this arises -- then possibly the simplest and most robust thing is to set up the header file so that external users just do #include "foo.h", and the internal users do

  #define FOO_INTERNAL
  #include "foo.h"

(This is how numpy's include files work, actually, though for reasons unrelated to symbol visibility.)

-- bottom line --

I think the obvious thing is that cython should provide a natural way to mark a function as being shared-library-internal; that covers 99% of real use cases *and* just works without any further macros needed. Probably the existing "public" annotation should be changed to mean this. (Obviously it wasn't quite fully thought through in the first place and has few if any users, since it got this very wrong without anyone noticing. So fixing it seems viable to me.)

And then *maybe* there should also be a way to make a symbol shared-library-exported, if that really is useful, but as a non-default experts-only kind of thing, and as such it would be OK to require these rare expert users to be a bit more careful about how they #include the resulting header within their own project.

-n

<div>
<p dir="ltr">On Apr 11, 2016 04:18, "Erik Bray" &lt;<a href="mailto:erik.m.bray@...">erik.m.bray@...</a>&gt; wrote:<br>
&gt;<br>
&gt; On Fri, Apr 8, 2016 at 5:49 PM, Nathaniel Smith &lt;<a href="mailto:njs@...">njs@...</a>&gt; wrote:<br>
&gt; &gt; Can you give a tiny concrete example? My questions are basic enough that I<br>
&gt; &gt; feel like I'm missing something fundamental :-)<br>
&gt;<br>
&gt; Yes, I think you might be missing something, but I'm not sure exactly<br>
&gt; where.&nbsp; In the issue I provided a tiny example which you can see here:<br>
&gt;<br>
&gt; <a href="https://gist.github.com/embray/12a67edb82b213217e31f408007898e6">https://gist.github.com/embray/12a67edb82b213217e31f408007898e6</a><br>
&gt;<br>
&gt; The C code generated by this example currently does not compile on<br>
&gt; Windows, because of how Cython uses DL_IMPORT incorrectly.&nbsp; Regardless<br>
&gt; of what it *does* do, Cython should not be using the DL_IMPORT macro<br>
&gt; at all actually since that no longer even exists in Python.</p>
<p dir="ltr">Sure.</p>
<p dir="ltr">For that example, the correct thing to do is to *not* export the function.</p>
<p dir="ltr">Backing up, to make sure we're on the same page: There are three levels of symbol visibility in C: file-internal, shared library internal (different .c files that make up the library can see it, but users of the shared library can't see it), and shared library exported (everyone can see it; can also carry other consequences, e.g. on Linux then internal calls will become noticeably slower, and it becomes possible for weird symbol interposition issues to occur). So the rule of thumb is to make everything as private as you can get away with.</p>
<p dir="ltr">Making this more interesting:<br>
- vanilla C only includes 2 ways to mark symbol visibility, which is not enough to make a 3-way distinction. Hence the need for an extra attribute thingummy.<br>
- everyone agrees that symbols marked 'static' should be file-internal, but different platforms disagree about what should happen if the extra attribute thingummy is missing.</p>
<p dir="ltr">So on Windows, the convention is:<br>
'static' -&gt; file internal<br>
no marking -&gt; shared library internal<br>
'dllexport' -&gt; public</p>
<p dir="ltr">And on Linux it's:<br>
'static' -&gt; file internal<br>
'visibility (hidden)' -&gt; shared library internal<br>
no marking -&gt; public</p>
<p dir="ltr">It's generally agreed that Linux got this wrong and that you should always use '-fvisibility=hidden' to switch it to the windows style, but cython doesn't control compiler options and thus should probably generate code that works correctly regardless of such compiler settings. Fortunately, Linux does provide some markings to explicitly make things public: you can mark a symbol 'visibility (default)' (which means public), or you can use the dllexport syntax, just like Windows, because gcc is helpful like that.</p>
<p dir="ltr">OTOH, windows is annoying because of this dllimport thing that started this whole thread: on other systems just marking symbols as extern is enough to handle both shared-object-internal and shared-library-exported symbols, and the linker will sort it out. On Windows, you have to explicitly distinguish between these. (And annoyingly, if you accidentally leave out the dllimport making on functions then it will use some fallback hack that works but silently degrades performance; on other symbols it just doesn't work, and ditto for if you use it when it isn't needed.)</p>
<p dir="ltr">So final conclusion: for non-static symbols, cython should first decide whether they are supposed to be shared-library-internal or actually exported from the shared library.</p>
<p dir="ltr">For shared-library-internal symbols: their definition should be marked 'visibility(hidden)' on Linux, and unmarked on Windows. This is easy using some preprocessor gunk. (Or maybe simplest is: marked that everywhere except if using msvc, because I think everyone else will understand 'visibility (hidden)' even if it's a no op.) Their declaration in the header file should just be 'extern' everywhere.</p>
<p dir="ltr">For shared-library-exported symbols: I am dubious about whether cython should even support these at all. But if it does, then the definitions should be marked 'dllexport' (no macro trickery needed, because everyone understands this syntax), and their declaration in the header file needs some extra hack that is the subject of this thread.</p>
<p dir="ltr">Now, back to your example: Here the caller and callee are both compiled into the same shared library, so you don't want dllexport/dllimport at all, you just want a shared-library-internal symbol, which as we see is much easier.</p>
<p dir="ltr">NumPy also ran into some problems with this in our experiments with using cython internally. Our temporary solution was to use the preprocessor to monkeypatch DL_IMPORT into expanding to the appropriate shared-library-internal thing :-).</p>
<p dir="ltr">&gt; &gt; My first question is why you even need this, since AFAIK there are no cases<br>
&gt; &gt; where it is correct to have a cython module dllexporting symbols that appear<br>
&gt; &gt; in header files. This is what cimport is for, right?<br>
&gt;<br>
&gt; I don't think this has anything to do with cimport.&nbsp; Could you explain<br>
&gt; what you mean?</p>
<p dir="ltr">We only need to solve the dllimport issue if we want to support shared-library-exported symbols from cython extensions. This is only useful if you have different extensions that are directly linking to each other using the platform linker. But this simply doesn't work in almost any cases, because platform linkers have all kinds of quirks that don't play well with python packages -- to start with, your runtime linker probably does not follow Python's rules for which directories to search to find a shared library... To solve these problems, the Cython devs invented the cimport mechanism, which is basically a portable, python-centric way of doing shared-library-exports while avoiding all the problems caused by using the platform linker. So my question was, what's your use case that's better served by linking directly against an extension module and using the platform's shared-library-export functionality, instead of cython's?</p>
<p dir="ltr">&gt; &gt; My second question is why you would want to do this via the command line,<br>
&gt; &gt; when compiling the dll means that you are compiling some cython-generated<br>
&gt; &gt; .c, which means that you can put the #define directly in the source code,<br>
&gt; &gt; no?<br>
&gt;<br>
&gt; Not necessarily. As you can see in my example the file fooutil.c is<br>
&gt; hand-written C code that was not generated by Cython, but which uses a<br>
&gt; function in the Cython-generated code.&nbsp; It includes "foo.h".&nbsp; In<br>
&gt; principle you're right--the hand-written C code could set the proper<br>
&gt; #defines but it would have to do so *before* including "foo.h".&nbsp; It's<br>
&gt; not very obvious that this would be needed.&nbsp; Most other code I've seen<br>
&gt; that addresses this issue--including cpython itself--do so by passing<br>
&gt; an appropriate define to the compiler via the command-line, and that<br>
&gt; seems the clearest to me.</p>
<p dir="ltr">I see, right. I guess my suggestion would be that if a symbol really does need to be marked for shared-library-export *and simultaneously* used by different files within the same shared library -- which is the only case where this arises -- then possibly the simplest and most robust thing is to set up the header file so that external users just do #include "foo.h", and the internal users do</p>
<p dir="ltr">&nbsp; #define FOO_INTERNAL<br>
&nbsp; #include "foo.h"</p>
<p dir="ltr">(This is how numpy's include files work, actually, though for reasons unrelated to symbol visibility.)</p>
<p dir="ltr">-- bottom line --</p>
<p dir="ltr">I think the obvious thing is that cython should provide a natural way to mark a function as being shared-library-internal; that covers 99% of real use cases *and* just works without any further macros needed. Probably the existing "public" annotation should be changed to mean this. (Obviously it wasn't quite fully thought through in the first place and has few if any users, since it got this very wrong without anyone noticing. So fixing it seems viable to me.)</p>
<p dir="ltr">And then *maybe* there should also be a way to make a symbol shared-library-exported, if that really is useful, but as a non-default experts-only kind of thing, and as such it would be OK to require these rare expert users to be a bit more careful about how they #include the resulting header within their own project.</p>
<p dir="ltr">-n</p>
</div>
Erik Bray | 8 Apr 13:01 2016
Picon
Gravatar

[Cython] Fwd: Question about how best require compiler options for C sources

Hi all,

I'd like to call attention to an issue I've been looking into for the
past couple days:

https://github.com/cython/cython/pull/360

To summarize the discussion, when building DLLs for Windows, functions
that should be exported by that DLL must be marked
__declspec(dllexport) in their declaration.  However, when using the
same header file in a project that links to that DLL the same function
must be declared __declspec(dllimport).

It's common practice to have a macro for this, whose value is
controlled by whether or not a macro (often called something like
"DLL_EXPORT").  When compiling the DLL we would define -DDLL_EXPORT to
output __declspec(dllexport).  Otherwise it outputs
__declspec(dllimport).

Cython currently handles this with such a macro called DL_IMPORT which
comes from Python. However, this macro was deprecated some time ago,
and is removed in current Python 3 versions.  So Cython must replace
it with its own.

I'm working on a patch for this--to reduce confusion the macro is
named specifically for the Cython module it's associated with.  For
example, for a Cython module named "foo.bar" there are two macros
DLL_EXPORT__foo__bar and EXPORT__foo__bar.  If the latter is defined
then the former outputs dllexport, otherwise it outputs dllimport.
I've attached the patch in progress.

I'm open to comment on this, but where I'm stuck now is that in order
for the "foo.bar" module to be compiled correctly it needs
EXPORT__foo__bar to be defined at compile time.  And it's not clear to
me what the best way is for the Cython compiler to pass down options
that are passed to the C/C++ compiler.  In general the best way would
be to attach this to the define_macros attribute of the associated
distutils/setuptools Extension object and let the distutils compiler
class generate the right compiler options.  But it's not clear to me
from Cython's internals where the best place to do that would be.

Thanks,
Erik
Attachment (dll_export.patch): application/octet-stream, 5620 bytes
Hi all,

I'd like to call attention to an issue I've been looking into for the
past couple days:

https://github.com/cython/cython/pull/360

To summarize the discussion, when building DLLs for Windows, functions
that should be exported by that DLL must be marked
__declspec(dllexport) in their declaration.  However, when using the
same header file in a project that links to that DLL the same function
must be declared __declspec(dllimport).

It's common practice to have a macro for this, whose value is
controlled by whether or not a macro (often called something like
"DLL_EXPORT").  When compiling the DLL we would define -DDLL_EXPORT to
output __declspec(dllexport).  Otherwise it outputs
__declspec(dllimport).

Cython currently handles this with such a macro called DL_IMPORT which
comes from Python. However, this macro was deprecated some time ago,
and is removed in current Python 3 versions.  So Cython must replace
it with its own.

I'm working on a patch for this--to reduce confusion the macro is
named specifically for the Cython module it's associated with.  For
example, for a Cython module named "foo.bar" there are two macros
DLL_EXPORT__foo__bar and EXPORT__foo__bar.  If the latter is defined
then the former outputs dllexport, otherwise it outputs dllimport.
I've attached the patch in progress.

I'm open to comment on this, but where I'm stuck now is that in order
for the "foo.bar" module to be compiled correctly it needs
EXPORT__foo__bar to be defined at compile time.  And it's not clear to
me what the best way is for the Cython compiler to pass down options
that are passed to the C/C++ compiler.  In general the best way would
be to attach this to the define_macros attribute of the associated
distutils/setuptools Extension object and let the distutils compiler
class generate the right compiler options.  But it's not clear to me
from Cython's internals where the best place to do that would be.

Thanks,
Erik
Nathaniel Smith | 22 Mar 07:45 2016

[Cython] [ANN] Python compilers workshop at SciPy this year

Hi all,

I wanted to announce a workshop I'm organizing at SciPy this year, and
invite you to attend!

What: A two-day workshop bringing together folks working on JIT/AOT
compilation in Python.

When/where: July 11-12, in Austin, Texas.

(This is co-located with SciPy 2016, at the same time as the tutorial
sessions, just before the conference proper.)

Website: https://python-compilers-workshop.github.io/

Note that I anticipate that we'll be able to get sponsorship funding
to cover travel costs for folks who can't get their employers to foot
the bill.

Cheers,
-n

--

-- 
Nathaniel J. Smith -- https://vorpus.org

Gmane