headers
Joost van Evert | 1 Jul 2005 13:39
Picon

numarray problem creating large memmap

Hello,

does anyone know why it is not possible to create a memmap beyond a
certain length. For example: m =
MM.open('/tmp/test','write',len=500000000) works, while the same command
using a length of 600000000 gives the following error message:

/shannon/joost/lib/python/numarray/memmap.py in __init__(self, filename,
mode, len)
    299             else:
    300                 acc = mmap.ACCESS_WRITE
--> 301             self._mmap = mmap.mmap(file.fileno(), len,
access=acc)
    302         else:
    303             self._mmap = None

EnvironmentError: [Errno 22] Invalid argument

I am using:
- numarray version 1.3.1
- python 2.4.1
- linux

Greetings,
Joost

-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
(Continue reading)

Permalink | Reply | 1 comment |
headers
Todd Miller | 1 Jul 2005 18:50

Re: numarray problem creating large memmap

On Fri, 2005-07-01 at 07:39, Joost van Evert wrote:
> Hello,
> 
> does anyone know why it is not possible to create a memmap beyond a
> certain length. For example: m =
> MM.open('/tmp/test','write',len=500000000) works, while the same command
> using a length of 600000000 gives the following error message:
> 
> /shannon/joost/lib/python/numarray/memmap.py in __init__(self, filename,
> mode, len)
>     299             else:
>     300                 acc = mmap.ACCESS_WRITE
> --> 301             self._mmap = mmap.mmap(file.fileno(), len,
> access=acc)
>     302         else:
>     303             self._mmap = None
> 
> EnvironmentError: [Errno 22] Invalid argument
> 
> I am using:
> - numarray version 1.3.1
> - python 2.4.1
> - linux

I'm not sure it matters,  but how much RAM and swap do you have?  How
much free space do you have on /tmp? 

Regards,
Todd
(Continue reading)

Permalink | Reply | 0 comments |
headers
Ray Schumacher | 1 Jul 2005 23:00
Favicon

a Numeric.where re-coded for weave.inline: very fast...


I tried weave for executing a C routine over a Numeric array. 

The Numeric data array is c_int16 with 14 significant bits (bit 13 is the sign) and I wanted it as "normal" Int16.
So:
        t1=clock()
        m = where(greater(dataArray[:1000], 8191), 
                        dataArray[:1000]-16383, 
                        dataArray[:1000])
        t2 = clock()
        print 'where', round((t2-t1)*1000000, 1), 'us'
Usually gives: where 634.3 us

PyInline has about the same calling overhead as ctypes (I had tried a DLL and ctypes:
http://sourceforge.net/mailarchive/forum.php?thread_id=7630224&forum_id=24606 ), apparently,
about 240us for a simple C statement. It also requires some extra hoops to digest the pointer to short I need
(  void b2int(short *f, int N)   ), so I didn't fully try it.

weave.inline testing was very good, with different issues. I found I had to patch msvccompiler.py for MS C
7.1 .NET ( http://www.vrplumber.com/programming/mstoolkit/ ). After the free but massive download/install...

        import weave
        inline = weave.inline # for speed
        N = 1000
        code="int i; for(i = 0; i < N; i++){if (dataArray[i]>8191){dataArray[i]-=16383;}}"
        inline(code, ['dataArray', 'N']) # just pass Python object's names

This created  sc_019a1cf36209cb2dfc688820080541ef0.pyd in C:\Documents and Settings\rays\Local Settings\Temp\rays\python24_compiled\
Using the above code is slow; ~32000 us, as the compiler checks or runs each time. However, after much
fiddling and dir()s, I copied the long-name.pyd to C:\Python24\DLLs and just did
(Continue reading)

Permalink | Reply | 0 comments |
headers
Russell E. Owen | 2 Jul 2005 00:02
Favicon

Trouble installing numarray on Solaris

A user of my application reports that installing numarray 1.3.2 on 
Solaris using Python 2.3.1 has run into trouble:

  prompt> python
  Python 2.3.1 (#1, Sep 30 2003, 20:29:58) [C] on sunos5
  Type "help", "copyright", "credits" or "license" for more information.
  >>> import numarray
  Fatal Python error: Call to API function without first calling
  import_libnumarray() in Src/_convmodule.c
  Abort (core dumped)

I've seen a one or two recent reports of this error when running 
extension modules, but I've never heard of it happening when simply 
trying to import numarray.

Any suggestions?

-- Russell

P.S. I do realize that python 2.3.1 is old and should be updated, and I 
have already suggested it. But it seems a bit unlikely to fix this 
particular problem.

-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
Permalink | Reply | 10 comments |
headers
Nicolas Pernetty | 2 Jul 2005 12:33
Picon
Favicon

Re: Trouble installing numarray on Solaris

On Fri, 01 Jul 2005 15:02:44 -0700, "Russell E. Owen"
<rowen <at> cesmail.net> wrote :

> A user of my application reports that installing numarray 1.3.2 on 
> Solaris using Python 2.3.1 has run into trouble:
> 
>   prompt> python
>   Python 2.3.1 (#1, Sep 30 2003, 20:29:58) [C] on sunos5
>   Type "help", "copyright", "credits" or "license" for more
>   information.
>   >>> import numarray
>   Fatal Python error: Call to API function without first calling
>   import_libnumarray() in Src/_convmodule.c
>   Abort (core dumped)
> 
> I've seen a one or two recent reports of this error when running 
> extension modules, but I've never heard of it happening when simply 
> trying to import numarray.
> 
> Any suggestions?
> 
> -- Russell
> 
> P.S. I do realize that python 2.3.1 is old and should be updated, and
> I  have already suggested it. But it seems a bit unlikely to fix this 
> particular problem.
> 

Hello,
(Continue reading)

Permalink | Reply | 0 comments |
headers
Todd Miller | 2 Jul 2005 12:43

Re: Trouble installing numarray on Solaris

On Fri, 2005-07-01 at 15:02 -0700, Russell E. Owen wrote:
> A user of my application reports that installing numarray 1.3.2 on 
> Solaris using Python 2.3.1 has run into trouble:
> 
>   prompt> python
>   Python 2.3.1 (#1, Sep 30 2003, 20:29:58) [C] on sunos5
>   Type "help", "copyright", "credits" or "license" for more information.
>   >>> import numarray
>   Fatal Python error: Call to API function without first calling
>   import_libnumarray() in Src/_convmodule.c
>   Abort (core dumped)

It looks vaguely like an binary API mismatch of some kind... or one of
you numarray modules is stale or broken.  To restate the obvious,   it's
likely that numarray won't import (in C!) because import_libnumarray()
is very defintely in _convmodule.c.  Since it's there and the API
pointer was not initialized,  the libnumarray import failed.

I just tried:

Python-2.3.4 (non-debug)
numarray-1.3.2 (should be irrelevant that it's not 1.3.1)
% uname -a
SunOS <our sun> 5.8 Generic_117350-18 sun4u sparc SUNW,Ultra-5_10

It checked out fine.  I recommend finding your old numarray install and
deleting it and re-installing 1.3.2.  Please let me know if that doesn't
solve the problem... I realize it's an unsatisfying answer.

Regards,
(Continue reading)

Permalink | Reply | 8 comments |
headers
Les Schaffer | 3 Jul 2005 04:39
Favicon

subclassing RecordArrays

Is there a simple way to subclass RecordArrays?

it seems not.

suppose i want to have a class RecArrayD(RecArray) that returns a list 
of its fields, much like a Dict does: recarrayD.keys()

the problem is that the fromXXX functions in records.py know only about 
the one RecArray. and RecArrayD._copyFrom() seems to be of little help, 
since the new style RecArray has to be framed out first, so to speak.

which leaves me to write a new fromRecArray() function that uses the new 
class.

all to get a list of the fields.

is this by design, or am i missing something?

yes, i can get the keys like so: recarray._names, but i assume that the 
'_' in front of '_names' is there for a reason, like maybe don't count 
on it???

or would the designers be willing to add a keys()-like method in for the 
next release?

les schaffer

-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
(Continue reading)

Permalink | Reply | 3 comments |
headers
Russell E Owen | 3 Jul 2005 05:30

Re: Trouble installing numarray on Solaris

At 6:43 AM -0400 7/2/05, Todd Miller wrote:
>On Fri, 2005-07-01 at 15:02 -0700, Russell E. Owen wrote:
>>  A user of my application reports that installing numarray 1.3.2 on
>>  Solaris using Python 2.3.1 has run into trouble:
>>
>>    prompt> python
>>    Python 2.3.1 (#1, Sep 30 2003, 20:29:58) [C] on sunos5
>>    Type "help", "copyright", "credits" or "license" for more information.
>>    >>> import numarray
>>    Fatal Python error: Call to API function without first calling
>>    import_libnumarray() in Src/_convmodule.c
>>    Abort (core dumped)
>
>It looks vaguely like an binary API mismatch of some kind... or one of
>you numarray modules is stale or broken.  To restate the obvious,   it's
>likely that numarray won't import (in C!) because import_libnumarray()
>is very defintely in _convmodule.c.  Since it's there and the API
>pointer was not initialized,  the libnumarray import failed.

Here's what the person had to say about it. I found the solution 
rather puzzling, but perhaps it makes sense to somebody who 
understands more about such things:

Got it working; it was indeed that the shared object modules weren't 
being linked correctly at build time, so when python tried to load 
them, they were missing symbols (eg, that are defined in the C math 
library, for example).  Probably the result of compiling numarray 
w/Solaris's cc rather than gcc.  I manually linked the modules for 
now, and numarry can now be imported... It even passed all its tests.
(Continue reading)

Permalink | Reply | 3 comments |
headers
Thomas Heller | 4 Jul 2005 13:40
Favicon

Re: Trouble installing numarray on Solaris

Todd Miller <jmiller <at> stsci.edu> writes:

> On Fri, 2005-07-01 at 15:02 -0700, Russell E. Owen wrote:
>> A user of my application reports that installing numarray 1.3.2 on 
>> Solaris using Python 2.3.1 has run into trouble:
>> 
>>   prompt> python
>>   Python 2.3.1 (#1, Sep 30 2003, 20:29:58) [C] on sunos5
>>   Type "help", "copyright", "credits" or "license" for more information.
>>   >>> import numarray
>>   Fatal Python error: Call to API function without first calling
>>   import_libnumarray() in Src/_convmodule.c
>>   Abort (core dumped)
>
> It looks vaguely like an binary API mismatch of some kind... or one of
> you numarray modules is stale or broken.  To restate the obvious,   it's
> likely that numarray won't import (in C!) because import_libnumarray()
> is very defintely in _convmodule.c.  Since it's there and the API
> pointer was not initialized,  the libnumarray import failed.

To the numarray authors/maintainers: Isn't it a bit harsh to call
Py_FatalError() in the init function?

Thomas

-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
(Continue reading)

Permalink | Reply | 3 comments |
headers
Todd Miller | 4 Jul 2005 14:57

Re: Re: Trouble installing numarray on Solaris

On Mon, 2005-07-04 at 13:40 +0200, Thomas Heller wrote:
> Todd Miller <jmiller <at> stsci.edu> writes:
> 
> > On Fri, 2005-07-01 at 15:02 -0700, Russell E. Owen wrote:
> >> A user of my application reports that installing numarray 1.3.2 on 
> >> Solaris using Python 2.3.1 has run into trouble:
> >> 
> >>   prompt> python
> >>   Python 2.3.1 (#1, Sep 30 2003, 20:29:58) [C] on sunos5
> >>   Type "help", "copyright", "credits" or "license" for more information.
> >>   >>> import numarray
> >>   Fatal Python error: Call to API function without first calling
> >>   import_libnumarray() in Src/_convmodule.c
> >>   Abort (core dumped)
> >
> > It looks vaguely like an binary API mismatch of some kind... or one of
> > you numarray modules is stale or broken.  To restate the obvious,   it's
> > likely that numarray won't import (in C!) because import_libnumarray()
> > is very defintely in _convmodule.c.  Since it's there and the API
> > pointer was not initialized,  the libnumarray import failed.
> 
> To the numarray authors/maintainers: Isn't it a bit harsh to call
> Py_FatalError() in the init function?

That's an interesting observation...  the old recourse was to dump core
by calling through a NULL function pointer.  :-)

Since this error can occur in the context of any API function, possibly
called from any sub-function within an extension function,  I'm not sure
there is a soft & general way to handle it:  getting it implies the
(Continue reading)

Permalink | Reply | 2 comments |

Gmane