headers
Stefan Behnel | 1 Feb 08:02
Picon
Favicon

Re: Re: passing pointers into __init__

mark florisson, 31.01.2012 21:55:
> On 31 January 2012 20:28, Thomas wrote:
>> Yes, that compiled.  Thank you.
>>
>> Why did that work?  Is cdef required when defining any variables that
>> are cdef class types?
> 
> It's not required, but then Cython treats it as an arbitrary Python
> object as it doesn't know what it is. Turning on type inferencing
> would probably make it work as well.

Hmm, I had thought that we got this to work - at least, it should work
because the return type of the fast instantiation call is always known, so
the object type can be inferred. My guess is that it's an issue with the
execution order of type inference and this specific optimisation.
Basically, the step that could infer the type of the variable as an
extension type can't know that it will have that type because the call to
__new__() wasn't analysed yet. Once again, the anticipated type analysis
restructuring would fix this.

Stefan
Permalink | Reply | 0 comments |
headers
Chris Barker | 1 Feb 17:58
Picon
Favicon

Re: Re: Python's array.array

On Tue, Jan 31, 2012 at 11:00 AM, Stefan Behnel <stefan_ml <at> behnel.de> wrote:
> We can't use NumPy as a dependency in Cython, simply because it's way to
> large for that.

fair enough -- I guess I was figuring something memory-view based, then.

> In the other direction, interaction with NumPy will always be possible
> through the buffer interface or by wrapping "something" in a NumPy array.

right -- as long as that's easy, that's enough numpy integration for most of us.

Also, while being a standard part of Cython would be nice, and
external lib that requires numpy would still be useful.

--

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker <at> noaa.gov
Permalink | Reply | 0 comments |
headers
Chris Barker | 1 Feb 18:45
Picon
Favicon

Re: Re: Python's array.array

On Tue, Jan 31, 2012 at 11:14 AM, Dag Sverre Seljebotn
<d.s.seljebotn <at> astro.uio.no> wrote:
> array.array has the benefit of existing at all.

sure -- though it really doesn't add much -- a re-sizable array isn't
all that much code to write, though maybe it has features I'm not
thinking of right now.

> Really, if going in that direction one must first create something without
> Cython support. Either add support in NumPy, or create a class in Cython
> implementing everything as one would like it (including dtype support, NumPy
> arithmetic compatability in Python space etc. etc.) except that "append" is
> a tad slower than it would be with Cython language support.

Done that already-- I'll try to post it here soon.

> Getting resizeable arrays into NumPy would require you to a) get it approved
> on the numpy mailing list (easier said than done, I think this has been
> requested several times in the past), b) do the implementation.

In a way, it's likely that (b) is the bigger hang-up -- it's been
requested in the past, but I don't think that the hang-up was so much
that the idea was rejected as that non-one with the skills to do it
(as a built-in) found it compelling. And that's fair enough -- with my
pyton-based version, the standard: "accumulate in a list, convert to
an array" is actually faster for the common cases -- I think that has
to do with where and when the conversion from python types happens,
but I haven't carefully profiled.

example:
(Continue reading)

Permalink | Reply | 2 comments |
headers
Chris Barker | 1 Feb 19:30
Picon
Favicon

Re: Re: Python's array.array

On Wed, Feb 1, 2012 at 9:45 AM, Chris Barker <chris.barker <at> noaa.gov> wrote:
> Done that already-- I'll try to post it here soon.

OK - so this is what I've got -- frankly, it's not much, but it's
useful enough that I'm using it. I've also got a reasonable number of
unit tests in place, in anticipation of Cython-izing it, etc.

It's pure python, and uses a numpy array for internal storage, thus
allowing it to delagte all the hard stuff to numpy. So all it's really
doing it over-allocating storare, keeping track of how much is used,
and calling ndarray.resize() when needed.

I've tried to make it so no one will every have a view on the
underlying array, to avoid all that hassle.

As I mentioned: performance is worse that using a pyton list for the
simple cases, though it does save memory for big data sets, and gets
better performance for numpy special dtypes and extending in chunks,
rather than appending one by one.

I'm hoping a Cython-ization could allow it to match list performance,
and better still, create a way to work directly with it from Cython
code, which could provide a real benifit.

Could the internal storage be a memoryview instead of a numpy array?

By the way, where can I find docs for memory views -- I've found a
couple discussions in the Wiki, but it's not clear to me what has been
implemented.
(Continue reading)

Permalink | Reply | 1 comment |
headers
Robert Bradshaw | 1 Feb 20:00
Favicon

Re: Re: Python's array.array

On Wed, Feb 1, 2012 at 10:30 AM, Chris Barker <chris.barker <at> noaa.gov> wrote:
> On Wed, Feb 1, 2012 at 9:45 AM, Chris Barker <chris.barker <at> noaa.gov> wrote:
>> Done that already-- I'll try to post it here soon.
>
> OK - so this is what I've got -- frankly, it's not much, but it's
> useful enough that I'm using it. I've also got a reasonable number of
> unit tests in place, in anticipation of Cython-izing it, etc.
>
> It's pure python, and uses a numpy array for internal storage, thus
> allowing it to delagte all the hard stuff to numpy. So all it's really
> doing it over-allocating storare, keeping track of how much is used,
> and calling ndarray.resize() when needed.
>
> I've tried to make it so no one will every have a view on the
> underlying array, to avoid all that hassle.
>
> As I mentioned: performance is worse that using a pyton list for the
> simple cases, though it does save memory for big data sets, and gets
> better performance for numpy special dtypes and extending in chunks,
> rather than appending one by one.
>
> I'm hoping a Cython-ization could allow it to match list performance,
> and better still, create a way to work directly with it from Cython
> code, which could provide a real benifit.

Until Cython has support for templates of some sort, a fundamental
limitation of a library like this is that elements will need to be
appended/popped as Python objects, but it could still be a big win
from a memory standpoint.
(Continue reading)

Permalink | Reply | 0 comments |
headers
Nikolaus Rath | 1 Feb 20:28
Favicon

From numpy array to C array

Hello,

I have a 2d numpy array, which I would like to pass to a C function
which expects a double*.

According to
http://mail.scipy.org/pipermail/numpy-discussion/2008-September/037675.html
this requires a dirty hack:

In your_header.h:
#define GET_BUFPTR_HACK(x) __pyx_bstruct_#x.buf

In .pyx code:

cdef extern from "your_header.h":
     void* GET_BUFPTR_HACK(x)

cdef double *data     
data = GET_BUFPTR_HACK(buf)

But then, this mail is from 2008. Is there a better way now?

Thanks,

   -Nikolaus

--

-- 
 »Time flies like an arrow, fruit flies like a Banana.«

  PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8 AE4E 425C
(Continue reading)

Permalink | Reply | 38 comments |
headers
Chris Barker | 1 Feb 20:56
Picon
Favicon

Re: Re: Python's array.array

On Wed, Feb 1, 2012 at 11:00 AM, Robert Bradshaw
<robertwb <at> math.washington.edu> wrote:
>> On Wed, Feb 1, 2012 at 9:45 AM, Chris Barker <chris.barker <at> noaa.gov> wrote:

>> It's pure python, and uses a numpy array for internal storage, thus
>> allowing it to delagte all the hard stuff to numpy. So all it's really
>> doing it over-allocating storare, keeping track of how much is used,
>> and calling ndarray.resize() when needed.
>>
>> I've tried to make it so no one will every have a view on the
>> underlying array, to avoid all that hassle.
>>

>> I'm hoping a Cython-ization could allow it to match list performance,
>> and better still, create a way to work directly with it from Cython
>> code, which could provide a real benifit.

> Until Cython has support for templates of some sort, a fundamental
> limitation of a library like this is that elements will need to be
> appended/popped as Python objects,

I'm confused -- why is that so? if the user specified a dtype in their
Cython code, couldn't it all be static from there? Or is that what you
mean by "templates of some sort".

But another thought -- if you limit functionality to storage, then
internally, all you would need to know is how many bytes the dtype is
-- the compiler wouldn't need to know what type is being stored.

Not sure how we'd handle the casting in and out of the storage, though.
(Continue reading)

Permalink | Reply | 2 comments |
headers
Chris Barker | 1 Feb 21:00
Picon
Favicon

Re: From numpy array to C array

On Wed, Feb 1, 2012 at 11:28 AM, Nikolaus Rath <Nikolaus <at> rath.org> wrote:
> I have a 2d numpy array, which I would like to pass to a C function
> which expects a double*.
> But then, this mail is from 2008. Is there a better way now?

indeed there is -- this looks like a reasonable example:

http://stackoverflow.com/questions/3046305/simple-wrapping-of-c-code-with-cython

-Chris

--

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker <at> noaa.gov
Permalink | Reply | 35 comments |
headers
Robert Bradshaw | 1 Feb 21:41
Favicon

Re: Re: Python's array.array

On Wed, Feb 1, 2012 at 11:56 AM, Chris Barker <chris.barker <at> noaa.gov> wrote:
> On Wed, Feb 1, 2012 at 11:00 AM, Robert Bradshaw
> <robertwb <at> math.washington.edu> wrote:
>>> On Wed, Feb 1, 2012 at 9:45 AM, Chris Barker <chris.barker <at> noaa.gov> wrote:
>
>>> It's pure python, and uses a numpy array for internal storage, thus
>>> allowing it to delagte all the hard stuff to numpy. So all it's really
>>> doing it over-allocating storare, keeping track of how much is used,
>>> and calling ndarray.resize() when needed.
>>>
>>> I've tried to make it so no one will every have a view on the
>>> underlying array, to avoid all that hassle.
>>>
>
>>> I'm hoping a Cython-ization could allow it to match list performance,
>>> and better still, create a way to work directly with it from Cython
>>> code, which could provide a real benifit.
>
>> Until Cython has support for templates of some sort, a fundamental
>> limitation of a library like this is that elements will need to be
>> appended/popped as Python objects,
>
> I'm confused -- why is that so? if the user specified a dtype in their
> Cython code, couldn't it all be static from there? Or is that what you
> mean by "templates of some sort".

I meant short of having a specific Accumulator to each dtype, or
templating of some sort, the append(), pop(), __getitem__, etc.
methods take and return objects regardless of the underlying storage.
(Continue reading)

Permalink | Reply | 1 comment |
headers
stevenwinfield | 2 Feb 10:42
Favicon

Segfault when using multi-argument generators in cdef'd classes

(This is my second attempt at posting this - apologies if you receive
this twice)

Hi all,

We've been having a little trouble when using generators with Cython,
which we've boiled down to the code below - a cdef'd class with two
methods that are generators, one of which takes a single (non-self)
arg and works fine and another that takes two args and segfaults. None
of the arguments are used in either case. The problem seems to occur
when .next() is called on the second generator. Is this a bug, or are
we doing something we shouldn't?

Thanks,

Steve.

test_cygen.py

from __future__ import print_function

import pyximport
pyximport.install()

from cygen_test import Test

test = Test(4)

import sys
print("Python version = ", sys.version)
(Continue reading)

Permalink | Reply | 3 comments |

Gmane