mark florisson | 2 Oct 12:38 2011
Picon

[Cython] buffer bug

Hey,

I'm unable to login in trac, but I found a bug in the buffer support:

cimport cython
cimport numpy as np

 <at> cython.boundscheck(False)
 <at> cython.wraparound(False)
cdef void func(np.ndarray[np.float32_t, ndim=2] a) nogil:
    pass

This calls __Pyx_GetBufferAndValidate, which needs the GIL.

When I get the last failing tests fixed (introduced after rebasing on
the latest master) for memoryviews, should be transform the current
buffer support to memoryviews before doing a release? The only
incompatibility I see is that readonly buffers are not supported.
On the other hand it might be a good idea to wait with that, in case
there are any bugs. We don't want to break everyone's existing code.
Opinions?

Mark
Dag Sverre Seljebotn | 2 Oct 13:04 2011
Picon
Picon

Re: [Cython] buffer bug

On 10/02/2011 12:38 PM, mark florisson wrote:
> Hey,
>
> I'm unable to login in trac, but I found a bug in the buffer support:
>
> cimport cython
> cimport numpy as np
>
>  <at> cython.boundscheck(False)
>  <at> cython.wraparound(False)
> cdef void func(np.ndarray[np.float32_t, ndim=2] a) nogil:
>      pass
>
> This calls __Pyx_GetBufferAndValidate, which needs the GIL.

Hmm. I thought buffers were disallowed as arguments to cdef functions?

> When I get the last failing tests fixed (introduced after rebasing on
> the latest master) for memoryviews, should be transform the current
> buffer support to memoryviews before doing a release? The only
> incompatibility I see is that readonly buffers are not supported.

Do you mean readonly memoryviews?

I'm not sure how much of an issue it is. NumPy arrays support being 
readonly, but it is not straightforward to make a NumPy array so.

Eventually I guess "const int[:]" should be supported; one could do so 
even without allowing const anywhere else.

(Continue reading)

mark florisson | 2 Oct 13:13 2011
Picon

Re: [Cython] buffer bug

On 2 October 2011 12:04, Dag Sverre Seljebotn
<d.s.seljebotn <at> astro.uio.no> wrote:
> On 10/02/2011 12:38 PM, mark florisson wrote:
>>
>> Hey,
>>
>> I'm unable to login in trac, but I found a bug in the buffer support:
>>
>> cimport cython
>> cimport numpy as np
>>
>>  <at> cython.boundscheck(False)
>>  <at> cython.wraparound(False)
>> cdef void func(np.ndarray[np.float32_t, ndim=2] a) nogil:
>>     pass
>>
>> This calls __Pyx_GetBufferAndValidate, which needs the GIL.
>
> Hmm. I thought buffers were disallowed as arguments to cdef functions?

Ah, perhaps they are, but I didn't get any error message.

>> When I get the last failing tests fixed (introduced after rebasing on
>> the latest master) for memoryviews, should be transform the current
>> buffer support to memoryviews before doing a release? The only
>> incompatibility I see is that readonly buffers are not supported.
>
> Do you mean readonly memoryviews?
>
> I'm not sure how much of an issue it is. NumPy arrays support being
(Continue reading)

Vitja Makarov | 2 Oct 19:52 2011
Picon

Re: [Cython] CyFunction refactoring plan

2011/9/30 mark florisson <markflorisson88@...>:
> On 30 September 2011 07:47, Vitja Makarov
<vitja.makarov@...> wrote:
>> 2011/9/30 Vitja Makarov <vitja.makarov@...>:
>>> 2011/9/30 Robert Bradshaw <robertwb@...>:
>>>> On Thu, Sep 29, 2011 at 10:43 PM, Stefan Behnel
<stefan_ml@...> wrote:
>>>>> Vitja Makarov, 30.09.2011 06:41:
>>>>>>
>>>>>> 2011/9/28 Vitja Makarov:
>>>>>>>
>>>>>>> I tried to build simple plan for ongoing cython function refactoring
>>>>>>>
>>>>>>> * Replace assignment synthesis with SingleAssignmentNode, where LHS is
>>>>>>> NameNode and RHS is PyCFunctionNode
>>>>>>> * Split function body into python wrapper and C function
>>>>>>> http://wiki.cython.org/enhancements/generators#Pythonfunctionrefactoring
>>>>>>>
>>>>>>> Then we can implement some features and optimizations:
>>>>>>>
>>>>>>> * Reduce difference between cdef and def functions
>>>>>>> * Store runtime evaluated default values inside CyFunction, ticket #674
>>>>>>> * Implement no-args super(), ticket #696
>>>>>>> * Function call inlining
>>>>>>
>>>>>> If nobody don't mind I would start with first one.
>>>>
>>>> I would love to see this happen.
>>>>
>>>>> Please go ahead. :)
(Continue reading)

mark florisson | 2 Oct 20:21 2011
Picon

Re: [Cython] CyFunction refactoring plan

On 2 October 2011 18:52, Vitja Makarov <vitja.makarov@...> wrote:
> 2011/9/30 mark florisson <markflorisson88@...>:
>> On 30 September 2011 07:47, Vitja Makarov
<vitja.makarov@...> wrote:
>>> 2011/9/30 Vitja Makarov <vitja.makarov@...>:
>>>> 2011/9/30 Robert Bradshaw <robertwb@...>:
>>>>> On Thu, Sep 29, 2011 at 10:43 PM, Stefan Behnel
<stefan_ml@...> wrote:
>>>>>> Vitja Makarov, 30.09.2011 06:41:
>>>>>>>
>>>>>>> 2011/9/28 Vitja Makarov:
>>>>>>>>
>>>>>>>> I tried to build simple plan for ongoing cython function refactoring
>>>>>>>>
>>>>>>>> * Replace assignment synthesis with SingleAssignmentNode, where LHS is
>>>>>>>> NameNode and RHS is PyCFunctionNode
>>>>>>>> * Split function body into python wrapper and C function
>>>>>>>> http://wiki.cython.org/enhancements/generators#Pythonfunctionrefactoring
>>>>>>>>
>>>>>>>> Then we can implement some features and optimizations:
>>>>>>>>
>>>>>>>> * Reduce difference between cdef and def functions
>>>>>>>> * Store runtime evaluated default values inside CyFunction, ticket #674
>>>>>>>> * Implement no-args super(), ticket #696
>>>>>>>> * Function call inlining
>>>>>>>
>>>>>>> If nobody don't mind I would start with first one.
>>>>>
>>>>> I would love to see this happen.
>>>>>
(Continue reading)

Vitja Makarov | 2 Oct 20:44 2011
Picon

Re: [Cython] CyFunction refactoring plan

2011/10/2 mark florisson <markflorisson88@...>:
> On 2 October 2011 18:52, Vitja Makarov <vitja.makarov@...> wrote:
>> 2011/9/30 mark florisson <markflorisson88@...>:
>>> On 30 September 2011 07:47, Vitja Makarov
<vitja.makarov@...> wrote:
>>>> 2011/9/30 Vitja Makarov <vitja.makarov@...>:
>>>>> 2011/9/30 Robert Bradshaw <robertwb@...>:
>>>>>> On Thu, Sep 29, 2011 at 10:43 PM, Stefan Behnel
<stefan_ml@...> wrote:
>>>>>>> Vitja Makarov, 30.09.2011 06:41:
>>>>>>>>
>>>>>>>> 2011/9/28 Vitja Makarov:
>>>>>>>>>
>>>>>>>>> I tried to build simple plan for ongoing cython function refactoring
>>>>>>>>>
>>>>>>>>> * Replace assignment synthesis with SingleAssignmentNode, where LHS is
>>>>>>>>> NameNode and RHS is PyCFunctionNode
>>>>>>>>> * Split function body into python wrapper and C function
>>>>>>>>> http://wiki.cython.org/enhancements/generators#Pythonfunctionrefactoring
>>>>>>>>>
>>>>>>>>> Then we can implement some features and optimizations:
>>>>>>>>>
>>>>>>>>> * Reduce difference between cdef and def functions
>>>>>>>>> * Store runtime evaluated default values inside CyFunction, ticket #674
>>>>>>>>> * Implement no-args super(), ticket #696
>>>>>>>>> * Function call inlining
>>>>>>>>
>>>>>>>> If nobody don't mind I would start with first one.
>>>>>>
>>>>>> I would love to see this happen.
(Continue reading)

mark florisson | 2 Oct 20:57 2011
Picon

Re: [Cython] CyFunction refactoring plan

On 2 October 2011 19:44, Vitja Makarov <vitja.makarov@...> wrote:
> 2011/10/2 mark florisson <markflorisson88@...>:
>> On 2 October 2011 18:52, Vitja Makarov <vitja.makarov@...> wrote:
>>> 2011/9/30 mark florisson <markflorisson88@...>:
>>>> On 30 September 2011 07:47, Vitja Makarov
<vitja.makarov@...> wrote:
>>>>> 2011/9/30 Vitja Makarov <vitja.makarov@...>:
>>>>>> 2011/9/30 Robert Bradshaw <robertwb@...>:
>>>>>>> On Thu, Sep 29, 2011 at 10:43 PM, Stefan Behnel
<stefan_ml@...> wrote:
>>>>>>>> Vitja Makarov, 30.09.2011 06:41:
>>>>>>>>>
>>>>>>>>> 2011/9/28 Vitja Makarov:
>>>>>>>>>>
>>>>>>>>>> I tried to build simple plan for ongoing cython function refactoring
>>>>>>>>>>
>>>>>>>>>> * Replace assignment synthesis with SingleAssignmentNode, where LHS is
>>>>>>>>>> NameNode and RHS is PyCFunctionNode
>>>>>>>>>> * Split function body into python wrapper and C function
>>>>>>>>>> http://wiki.cython.org/enhancements/generators#Pythonfunctionrefactoring
>>>>>>>>>>
>>>>>>>>>> Then we can implement some features and optimizations:
>>>>>>>>>>
>>>>>>>>>> * Reduce difference between cdef and def functions
>>>>>>>>>> * Store runtime evaluated default values inside CyFunction, ticket #674
>>>>>>>>>> * Implement no-args super(), ticket #696
>>>>>>>>>> * Function call inlining
>>>>>>>>>
>>>>>>>>> If nobody don't mind I would start with first one.
>>>>>>>
(Continue reading)

mark florisson | 2 Oct 23:39 2011
Picon

Re: [Cython] CyFunction refactoring plan

On 2 October 2011 19:44, Vitja Makarov <vitja.makarov@...> wrote:
> 2011/10/2 mark florisson <markflorisson88@...>:
>> On 2 October 2011 18:52, Vitja Makarov <vitja.makarov@...> wrote:
>>> 2011/9/30 mark florisson <markflorisson88@...>:
>>>> On 30 September 2011 07:47, Vitja Makarov
<vitja.makarov@...> wrote:
>>>>> 2011/9/30 Vitja Makarov <vitja.makarov@...>:
>>>>>> 2011/9/30 Robert Bradshaw <robertwb@...>:
>>>>>>> On Thu, Sep 29, 2011 at 10:43 PM, Stefan Behnel
<stefan_ml@...> wrote:
>>>>>>>> Vitja Makarov, 30.09.2011 06:41:
>>>>>>>>>
>>>>>>>>> 2011/9/28 Vitja Makarov:
>>>>>>>>>>
>>>>>>>>>> I tried to build simple plan for ongoing cython function refactoring
>>>>>>>>>>
>>>>>>>>>> * Replace assignment synthesis with SingleAssignmentNode, where LHS is
>>>>>>>>>> NameNode and RHS is PyCFunctionNode
>>>>>>>>>> * Split function body into python wrapper and C function
>>>>>>>>>> http://wiki.cython.org/enhancements/generators#Pythonfunctionrefactoring
>>>>>>>>>>
>>>>>>>>>> Then we can implement some features and optimizations:
>>>>>>>>>>
>>>>>>>>>> * Reduce difference between cdef and def functions
>>>>>>>>>> * Store runtime evaluated default values inside CyFunction, ticket #674
>>>>>>>>>> * Implement no-args super(), ticket #696
>>>>>>>>>> * Function call inlining
>>>>>>>>>
>>>>>>>>> If nobody don't mind I would start with first one.
>>>>>>>
(Continue reading)

Vitja Makarov | 2 Oct 23:52 2011
Picon

Re: [Cython] CyFunction refactoring plan

2011/10/3 mark florisson <markflorisson88@...>:
> On 2 October 2011 19:44, Vitja Makarov <vitja.makarov@...> wrote:
>> 2011/10/2 mark florisson <markflorisson88@...>:
>>> On 2 October 2011 18:52, Vitja Makarov <vitja.makarov@...> wrote:
>>>> 2011/9/30 mark florisson <markflorisson88@...>:
>>>>> On 30 September 2011 07:47, Vitja Makarov
<vitja.makarov@...> wrote:
>>>>>> 2011/9/30 Vitja Makarov <vitja.makarov@...>:
>>>>>>> 2011/9/30 Robert Bradshaw <robertwb@...>:
>>>>>>>> On Thu, Sep 29, 2011 at 10:43 PM, Stefan Behnel
<stefan_ml@...> wrote:
>>>>>>>>> Vitja Makarov, 30.09.2011 06:41:
>>>>>>>>>>
>>>>>>>>>> 2011/9/28 Vitja Makarov:
>>>>>>>>>>>
>>>>>>>>>>> I tried to build simple plan for ongoing cython function refactoring
>>>>>>>>>>>
>>>>>>>>>>> * Replace assignment synthesis with SingleAssignmentNode, where LHS is
>>>>>>>>>>> NameNode and RHS is PyCFunctionNode
>>>>>>>>>>> * Split function body into python wrapper and C function
>>>>>>>>>>> http://wiki.cython.org/enhancements/generators#Pythonfunctionrefactoring
>>>>>>>>>>>
>>>>>>>>>>> Then we can implement some features and optimizations:
>>>>>>>>>>>
>>>>>>>>>>> * Reduce difference between cdef and def functions
>>>>>>>>>>> * Store runtime evaluated default values inside CyFunction, ticket #674
>>>>>>>>>>> * Implement no-args super(), ticket #696
>>>>>>>>>>> * Function call inlining
>>>>>>>>>>
>>>>>>>>>> If nobody don't mind I would start with first one.
(Continue reading)

mark florisson | 4 Oct 23:19 2011
Picon

[Cython] Utilities, cython.h, libcython

Hey,

I briefly mentioned something about this in a pull request, but maybe
it deserves some actual discussion on the ML.

So I propose that after fused types gets merged we try to move as many
utility codes as possible to their utility code files (unless they are
used in pending pull requests or other branches). Preferably this will
be done in one or a few commits. How should we split up the work, any
volunteers? Perhaps people who wrote certain utilities also want to
move them? In that case, we should start a new branch and then merge
that into master when it's done.
We could actually move things before fused types get merged, as long
as we don't touch binding_cfunc_utility_code.

Before we go there, Stefan, do we still want to implement the header
.ini style which can list dependencies and such? I personally don't
care very much about it, but memoryviews and the utility loaders are
merged so if someone wants to take up that job, it'd be good to do
before moving the utilities.

Another issue is that Cython compile time is increasing with the
addition of control flow and cython utilities. If you use fused types
you're also going to combinatorially add more compile time. I'm sure
this came up earlier, but I really think we should have a libcython
and a cython.h. libcython (a shared library) should contain any common
Cython-specific code not meant to be inlined, and cython.h any types,
macros and inline functions etc. This will decrease Cython and C
compile time, and will also make executables smaller. This could be
enabled using a command line option to Cython, as well as with
(Continue reading)


Gmane