Robert Bradshaw | 2 Sep 07:55 2011

Re: [Cython] ComprehensionNode problem

On Sun, Aug 28, 2011 at 1:19 PM, Vitja Makarov
<vitja.makarov@...> wrote:
> I've started #715 ticket investigation.
>
>
> Here is minimal test case:
>
> # cython: language_level=3
> def foo(target):
>    return [(e for e in t) for t in target]
>
> Crash in the ticket is related to GeneratorExpressionScope (name is
> not correct, actually ScopedExprScope or NestedScope)
>
> If you set languange_level to 2 you will see next error:
> ...
> Compiler crash traceback from this point on:
>  File "/home/vitja/work/cython-vitek-git/Cython/Compiler/ExprNodes.py",
> line 7732, in __init__
>    if not result_type.create_from_py_utility_code(env):
> AttributeError: 'UnspecifiedType' object has no attribute
> 'create_from_py_utility_code'
>
>
> Since ComprehensionNode.append and ComprehensionNode.loop.body is the
> same generator body is created twice in MarkClosureVisitor
>
> So the issue could be solved in two ways:
>
>  - Write special handler for ComprehensionNode in MarkClosureVisitor
(Continue reading)

Vitja Makarov | 2 Sep 08:06 2011
Picon

Re: [Cython] ComprehensionNode problem

2011/9/2 Robert Bradshaw <robertwb@...>:
> On Sun, Aug 28, 2011 at 1:19 PM, Vitja Makarov
<vitja.makarov@...> wrote:
>> I've started #715 ticket investigation.
>>
>>
>> Here is minimal test case:
>>
>> # cython: language_level=3
>> def foo(target):
>>    return [(e for e in t) for t in target]
>>
>> Crash in the ticket is related to GeneratorExpressionScope (name is
>> not correct, actually ScopedExprScope or NestedScope)
>>
>> If you set languange_level to 2 you will see next error:
>> ...
>> Compiler crash traceback from this point on:
>>  File "/home/vitja/work/cython-vitek-git/Cython/Compiler/ExprNodes.py",
>> line 7732, in __init__
>>    if not result_type.create_from_py_utility_code(env):
>> AttributeError: 'UnspecifiedType' object has no attribute
>> 'create_from_py_utility_code'
>>
>>
>> Since ComprehensionNode.append and ComprehensionNode.loop.body is the
>> same generator body is created twice in MarkClosureVisitor
>>
>> So the issue could be solved in two ways:
>>
(Continue reading)

Vitja Makarov | 4 Sep 19:12 2011
Picon

Re: [Cython] non-virtual methods

2011/8/30 Robert Bradshaw <robertwb@...>:
> On Tue, Aug 30, 2011 at 10:19 AM, Vitja Makarov
<vitja.makarov@...> wrote:
>> 2011/8/30 Stefan Behnel <stefan_ml@...>:
>>> Vitja Makarov, 30.08.2011 18:39:
>>>>
>>>> 2011/8/30 Stefan Behnel:
>>>>>
>>>>> Robert Bradshaw, 30.08.2011 18:18:
>>>>>>
>>>>>> On Tue, Aug 30, 2011 at 9:14 AM, Vitja Makarov wrote:
>>>>>>>
>>>>>>> What about final classes with cpdef methods?
>>>>>>>
>>>>>>>  <at> cython.final
>>>>>>> class Foo:
>>>>>>>    cpdef bar(self):
>>>>>>>        pass
>>>>>>>
>>>>>>> Should that raise an error?
>>>>>>
>>>>>> That should be perfectly fine.
>>>>>
>>>>> Well, the 'final' decorator shouldn't work on normal Python classes.
>>>>>
>>>>> Regarding extension types, CPython has a way of declaring them 'final'
>>>>> with
>>>>> a type flag, which effectively prevents them from being subclassed in
>>>>> Python. So the above works as just fine for cdef classes.
>>>>>
(Continue reading)

Adrian Martínez Vargas | 5 Sep 18:25 2011
Picon
Picon

[Cython] how to get a double** from np.ndarray[double, ndim=2]

Hi,

I would like to get a double** from a 2d numpy array in cython in order 
to call properly some function written in C++. Can some one to give a 
pure cython solution?

 From now I have a double* from the np.ndarray[double, ndim=2] .data. 
I'm getting the double array in C++ with:

     array2D_=new double* [nrows];
     for (int i=0; i<nrows; ++i)
     {
       array2D_[i]=ptr1D;
       ptr1D+=ncols;
     }

with that I'm responsible to delete the array from C++

//     delete [] this->array2D_[0];  // delete the memory pool
     delete [] this->array2D_;      // delete the row pointers

but this->array2D_[0];  is shared with the numpy array.

I think that it may be more elegant to get directly a double** from 
numpy and let to python the responsibility of managing the memory.

  Regards
Adrian Martínez Vargas
Dag Sverre Seljebotn | 5 Sep 19:28 2011
Picon
Picon

Re: [Cython] how to get a double** from np.ndarray[double, ndim=2]

NumPy does not have a double** anywhere, what you did is the correct solution (as long as you make sure the array is C-contiguous).
--
Sent from my Android phone with K-9 Mail. Please excuse my brevity.

"Adrian Martínez Vargas" <adriangeologo-mRCrAkd8dF0@public.gmane.org> wrote:
Hi, I would like to get a double** from a 2d numpy array in cython in order to call properly some function written in C++. Can some one to give a pure cython solution? From now I have a double* from the np.ndarray[double, ndim=2] .data. I'm getting the double array in C++ with: array2D_=new double* [nrows]; for (int i=0; i<nrows; ++i) { array2D_[i]=ptr1D; ptr1D+=ncols; } with that I'm responsible to delete the array from C++ // delete [] this->array2D_[0]; // delete the memory pool delete [] this->array2D_; // delete the row pointers but this->array2D_[0]; is shared with the numpy array. I think that it may be more elegant to get directly a double** from numpy and let to python the responsibility of managing the memory. Regards Adrian Martínez Vargascython-devel mailing list cython-devel-+ZN9ApsXKcEdnm+yROfE0A@public.gmane.org http://mail.python.org/mailman/listinfo/cython-devel
<div>NumPy does not have a double** anywhere, what you did is the correct solution (as long as you make sure the array is C-contiguous). <br>
-- <br>
Sent from my Android phone with K-9 Mail. Please excuse my brevity.<br><br><div class="gmail_quote">"Adrian Mart&iacute;nez Vargas" &lt;adriangeologo@...&gt; wrote:<blockquote class="gmail_quote">
<div>Hi,

I would like to get a double** from a 2d numpy array in cython in order  to call properly some function written in C++. Can some one to give a  pure cython solution?

 From now I have a double* from the np.ndarray[double, ndim=2] .data. 
I'm getting the double array in C++ with:

     array2D_=new double* [nrows];
     for (int i=0; i&lt;nrows; ++i)
     {
       array2D_[i]=ptr1D;
       ptr1D+=ncols;
     }
 with that I'm responsible to delete the array from C++

//     delete [] this-&gt;array2D_[0];  // delete the memory pool
     delete [] this-&gt;array2D_;      // delete the row pointers
 but this-&gt;array2D_[0];  is shared with the numpy array.

I think that it may be more elegant to get directly a double** from  numpy and let to python the responsibility of managing the memory.

  Regards
Adrian Mart&iacute;nez Vargascython-devel mailing list
cython-devel@...
<a href="http://mail.python.org/mailman/listinfo/cython-devel">http://mail.python.org/mailman/listinfo/cython-devel</a>
</div>
</blockquote>
</div>
</div>
Stefan Behnel | 6 Sep 22:12 2011
Picon

[Cython] Jenkins jobs refactored

Hi,

I replaced the half-a-ton of cython-devel jobs in Jenkins by three 
multi-configuration matrix jobs:

https://sage.math.washington.edu:8091/hudson/job/cython-devel-build/

https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests/

https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr/

The sdist job that does the git checkouts is unchanged:

https://sage.math.washington.edu:8091/hudson/job/cython-devel-sdist/

This setup only slightly reduces the flexibility of the overall 
configuration, but it greatly reduces the maintenance overhead for the jobs 
and makes it much easier to keep their configuration aligned.

The only downside is that it that all build jobs must terminate before the 
tests are triggered. Given that the turn-over times are quite low, I don't 
think that's a problem. Quite the contrary, if one of the PyX.Y builds 
fails, none of the test jobs will run, which I think is a good thing.

I've disabled the old jobs and will eventually remove them when this setup 
proves to be acceptable for everyone.

Stefan
Robert Bradshaw | 6 Sep 22:21 2011

Re: [Cython] Jenkins jobs refactored

On Tue, Sep 6, 2011 at 1:12 PM, Stefan Behnel <stefan_ml@...> wrote:
> Hi,
>
> I replaced the half-a-ton of cython-devel jobs in Jenkins by three
> multi-configuration matrix jobs:
>
> https://sage.math.washington.edu:8091/hudson/job/cython-devel-build/
>
> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests/
>
> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr/
>
> The sdist job that does the git checkouts is unchanged:
>
> https://sage.math.washington.edu:8091/hudson/job/cython-devel-sdist/
>
> This setup only slightly reduces the flexibility of the overall
> configuration, but it greatly reduces the maintenance overhead for the jobs
> and makes it much easier to keep their configuration aligned.

Nice.

> The only downside is that it that all build jobs must terminate before the
> tests are triggered. Given that the turn-over times are quite low, I don't
> think that's a problem. Quite the contrary, if one of the PyX.Y builds
> fails, none of the test jobs will run, which I think is a good thing.

The one drawback is this seems to make it hard to implement the idea
of testing 2.4, 2.7, and 3.2 before building and testing all the rest
for quicker feedback?

> I've disabled the old jobs and will eventually remove them when this setup
> proves to be acceptable for everyone.
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel@...
> http://mail.python.org/mailman/listinfo/cython-devel
>
Stefan Behnel | 6 Sep 22:58 2011
Picon

Re: [Cython] Jenkins jobs refactored

Robert Bradshaw, 06.09.2011 22:21:
> On Tue, Sep 6, 2011 at 1:12 PM, Stefan Behnel wrote:
>> I replaced the half-a-ton of cython-devel jobs in Jenkins by three
>> multi-configuration matrix jobs:
>>
>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-build/
>>
>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests/
>>
>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr/
>>
>> The sdist job that does the git checkouts is unchanged:
>>
>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-sdist/
>>
>> This setup only slightly reduces the flexibility of the overall
>> configuration, but it greatly reduces the maintenance overhead for the jobs
>> and makes it much easier to keep their configuration aligned.
>
> Nice.
>
>> The only downside is that it that all build jobs must terminate before the
>> tests are triggered. Given that the turn-over times are quite low, I don't
>> think that's a problem. Quite the contrary, if one of the PyX.Y builds
>> fails, none of the test jobs will run, which I think is a good thing.
>
> The one drawback is this seems to make it hard to implement the idea
> of testing 2.4, 2.7, and 3.2 before building and testing all the rest
> for quicker feedback?

Partly. All build jobs will have to terminate first, yes, but Jenkins has a 
notion of a "touchstone build", which allows to configure certain builds in 
the matrix that should run first. So, after all PyX.Y builds are ready, 
Py2.4/7 and Py3k will be tested first. That way, the feedback is a little 
slower than today, but it still prefers the important platforms.

Stefan
Robert Bradshaw | 6 Sep 23:03 2011

Re: [Cython] Jenkins jobs refactored

On Tue, Sep 6, 2011 at 1:58 PM, Stefan Behnel <stefan_ml@...> wrote:
> Robert Bradshaw, 06.09.2011 22:21:
>>
>> On Tue, Sep 6, 2011 at 1:12 PM, Stefan Behnel wrote:
>>>
>>> I replaced the half-a-ton of cython-devel jobs in Jenkins by three
>>> multi-configuration matrix jobs:
>>>
>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-build/
>>>
>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests/
>>>
>>>
>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr/
>>>
>>> The sdist job that does the git checkouts is unchanged:
>>>
>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-sdist/
>>>
>>> This setup only slightly reduces the flexibility of the overall
>>> configuration, but it greatly reduces the maintenance overhead for the
>>> jobs
>>> and makes it much easier to keep their configuration aligned.
>>
>> Nice.
>>
>>> The only downside is that it that all build jobs must terminate before
>>> the
>>> tests are triggered. Given that the turn-over times are quite low, I
>>> don't
>>> think that's a problem. Quite the contrary, if one of the PyX.Y builds
>>> fails, none of the test jobs will run, which I think is a good thing.
>>
>> The one drawback is this seems to make it hard to implement the idea
>> of testing 2.4, 2.7, and 3.2 before building and testing all the rest
>> for quicker feedback?
>
> Partly. All build jobs will have to terminate first, yes, but Jenkins has a
> notion of a "touchstone build", which allows to configure certain builds in
> the matrix that should run first. So, after all PyX.Y builds are ready,
> Py2.4/7 and Py3k will be tested first. That way, the feedback is a little
> slower than today, but it still prefers the important platforms.

That sounds find. If the build breaks, I'm fine with forcing that to
be fixed before going on to testing.

- Robert
Romain Guillebert | 8 Sep 06:18 2011
Picon

[Cython] CTypes backend for Cython Status

Hi

The Google Summer of Code has ended and I didn't give the current status
to anyone yet (I was very busy with a report I had to write for my
university).

There is still work to do on the project (there was more work than I
expected, especially because of semantic differences between Cython and
ctypes) so I'll talk about what needs to be done (even if it does not
sound good compared to talking about what has been done) from the most
important to the least important in my opinion :

- Pointer vs Array, Cython mixes the two while ctypes does not, this
  can probably be fixed by using arrays everywhere (if we can convert
  pointers into arrays)
- Take into account header files declared globally
- Macros, this is probably the biggest part but it's doable, Cython has
  the types of the arguments and the return value so it's possible to
  generate a C function corresponding to a macro
- Pointer to functions

Some of them are trivial, others just require good ideas and macros
demands a big amount of work.
I'm still working on it and if someone wants to give a hand, I'll be
happy to explain what I've done.

Thanks
Romain

Gmane