headers
Chris Barker | 2 Nov 2000 22:46

"formstring()" in place?

I have a narge array of type "1" (single bytes). I need to convert it to
Int32, in the manner that fromstring() would. Right now, I am doing:

Array = fromstring(Array.tostring(),'f')

This works fine, but what concerns me is that I need to do this on
potentially HUGE arrays, and if I understand this right, I am going to
create a copy with tostring, and then another copy with fromstring, that
then gets referenced to Array, at which point the first original copy
gets de-referenced, and should be deleted, and the temporary one gets
deleted at some point in this process. I don't know when stuff created
in the middle of a statement gets deleted, so I could potentially have
three copies of the data around at the same time, and at least two.
Since it is exactly the same C array, I'd like to be able to do this
without making any copies at all. Is it possible? It seems like it
should be a simple matter of changing the typecode and shape, but is
this possible?

While I'm asking questions: can I byteswap in place as well?

The greater problem:

To give a little background, and to see if anyone has a better idea of
how to do what I am doing, I thought I'd run through the task that I
really need to do.

I am reading a binary file full of a lot of data. I have some control
over the form of the file, but it needs to be compact, so I can't just
make everything the same large type. The file is essentially a whole
bunch of records, each of which is a collection of a couple of different
(Continue reading)

Permalink | Reply | 6 comments |
headers
Janko Hauser | 3 Nov 2000 01:05
Picon

Re: "formstring()" in place?

Use the numpyio module from Travis. With this it should be possible to
read the data directly and do any typecode conversion you want
with. It has fread and fwrite functions, and it can be used with any
NumPy type like Int0 in your case. It's part of the signaltools
package.

http://oliphant.netpedia.net/signaltools_0.5.2.tar.gz 

HTH, 
__Janko
Permalink | Reply | 1 comment |
headers
Sam Tannous | 3 Nov 2000 03:45
Picon
Favicon

Static linking with Python 2.0 (Setup.local)

I successfully installed the latest release with the 
distutils package. But now I need to statically link 
the Numeric package into my Python 2.0 interpreter
(it's easier for me to ship "frozen" python binaries 
for some lab machines...one file instead of hundreds).  

Could anyone please email me (or post) the section 
of your Modules/Setup.local that can do this?
Something like:

_numpy _numpymodule.c -L/users/stannous/lib -L/usr/lib -lm -I/users/stannous/include

just doesn't work...

TIA,
Sam
Permalink | Reply | 1 comment |
headers
Paul F. Dubois | 3 Nov 2000 03:57
Picon

RE: Static linking with Python 2.0 (Setup.local)

I believe you can deduce the modules being built and the sources / include /
lib directories for them by examining setup.py.

When we switched to distutils we chucked our Setup file.

-----Original Message-----
From: numpy-discussion-admin <at> lists.sourceforge.net
[mailto:numpy-discussion-admin <at> lists.sourceforge.net]On Behalf Of Sam
Tannous
Sent: Thursday, November 02, 2000 6:45 PM
To: numpy-discussion <at> sourceforge.net
Subject: [Numpy-discussion] Static linking with Python 2.0 (Setup.local)

I successfully installed the latest release with the
distutils package. But now I need to statically link
the Numeric package into my Python 2.0 interpreter
(it's easier for me to ship "frozen" python binaries
for some lab machines...one file instead of hundreds).

Could anyone please email me (or post) the section
of your Modules/Setup.local that can do this?
Something like:

_numpy
_numpymodule.c -L/users/stannous/lib -L/usr/lib -lm -I/users/stannous/includ
e

just doesn't work...

TIA,
(Continue reading)

Permalink | Reply | 0 comments |
headers
Bernd Rinn | 3 Nov 2000 15:44
Picon
Favicon

performance of real_fft with fftpacklite.c

Hello,

does anyone know why the performance of FFT.real with fftpacklite.c is
so unballanced for n=2**i and different values of i? My example is:

=======================================================================
from Numeric import array,Float
from FFT import real_fft
from time import time

i=1
while i<20:
    n = 2**long(i)  
    a=array(range(long(1),n),Float)
    anfang = time()
    b = real_fft(a)
    ende=time()
    print "i=", i, " time: ", ende-anfang
    i+=1
=======================================================================

and the result shows (on a Pentium-III-700 under Linux):

=================================================================
i= 1  time:  0.000182032585144
i= 2  time:  0.000133991241455
i= 3  time:  0.00012195110321
i= 4  time:  0.000123977661133
i= 5  time:  0.000131964683533
i= 6  time:  0.000155925750732
(Continue reading)

Permalink | Reply | 1 comment |
headers
Paul F. Dubois | 3 Nov 2000 16:32
Picon

RE: "formstring()" in place?

>>> y=array([1,2,3], '1')
>>> y
array([1, 2, 3],'1')
>>> y.astype(Int32)
array([1, 2, 3],'i')
>>>

-----Original Message-----
From: numpy-discussion-admin <at> lists.sourceforge.net
[mailto:numpy-discussion-admin <at> lists.sourceforge.net]On Behalf Of Chris
Barker
Sent: Thursday, November 02, 2000 1:46 PM
Cc: numpy-discussion <at> sourceforge.net
Subject: [Numpy-discussion] "formstring()" in place?

I have a narge array of type "1" (single bytes). I need to convert it to
Int32, in the manner that fromstring() would. Right now, I am doing:

Array = fromstring(Array.tostring(),'f')

This works fine, but what concerns me is that I need to do this on
potentially HUGE arrays, and if I understand this right, I am going to
create a copy with tostring, and then another copy with fromstring, that
then gets referenced to Array, at which point the first original copy
gets de-referenced, and should be deleted, and the temporary one gets
deleted at some point in this process. I don't know when stuff created
in the middle of a statement gets deleted, so I could potentially have
three copies of the data around at the same time, and at least two.
Since it is exactly the same C array, I'd like to be able to do this
without making any copies at all. Is it possible? It seems like it
(Continue reading)

Permalink | Reply | 3 comments |
headers
Scott Ransom | 3 Nov 2000 16:43
Picon
Favicon

Re: performance of real_fft with fftpacklite.c

Bernd Rinn wrote:
> 
> Hello,
> 
> does anyone know why the performance of FFT.real with fftpacklite.c is
> so unballanced for n=2**i and different values of i? My example is:

Hi,

The problem is that your routine gives arrays of length 2**n-1 not
2**n!  So for the arrays where the CPU time is huge, you are FFTing a
length that is a prime number (or at least not easily factorable). 
FFTPACK has to do a brute force DFT instead of a FFT!  (Ah, the beauty
of n*log(n)...)

You can see that this is the case by printing the lengths of the arrays:

-------------------------------
from Numeric import array,Float
from FFT import real_fft
from time import time

i=1
while i<20:
     n = 2**long(i)  
     a=array(range(long(1),n),Float)
     print len(a)
     i=i+1
-------------------------------
(Continue reading)

Permalink | Reply | 0 comments |
headers
Chris Barker | 3 Nov 2000 20:40

Re: "formstring()" in place?

Janko Hauser wrote:
> Use the numpyio module from Travis. With this it should be possible to
> read the data directly and do any typecode conversion you want
> with. It has fread and fwrite functions, and it can be used with any
> NumPy type like Int0 in your case. It's part of the signaltools
> package.
> 
> http://oliphant.netpedia.net/signaltools_0.5.2.tar.gz

I've downloaded it , and it looks pretty handy. It does include a
byteswap-in-place, which I need. What is not clear to me from the
minimal docs is whether I can read file set up like:
long long char long long char ....
and have it put the longs into one array, and the chars into another.

Also, It wasn't clear whether I could put use it to read a file that has
already been opened, starting at the file's current position. I am
working with a file that has a text header, so I can't just suck in the
whole thing until I've parsed out the header.

I can figure out the answer to these questions with some reding of the
source, but it wasn't obvious at first glance, so it would be great if
someone knows the answer off the top of there head. Travis?

By the way, there seem to be a few methods that produce a copy, rather
than doing things in place, where it seems more intuitive to do it in
place. byteswapped() and astype() come to mind. With byteswapped, I
imagine it's rare that you would want to keep a copy around. With astype
it would also be rare to keep a copy around, but since it changes the
size of the array, I imagine it would be a lot harder to code as an
(Continue reading)

Permalink | Reply | 0 comments |
headers
Chris Barker | 3 Nov 2000 20:28

Re: "formstring()" in place?

"Paul F. Dubois" wrote:
> >>> y=array([1,2,3], '1')
> >>> y
> array([1, 2, 3],'1')
> >>> y.astype(Int32)
> array([1, 2, 3],'i')

Actually, this is exactly NOT what I want to do. In this case, each 1
byte interger was converted to a 4byte integer, of the same VALUE. What
I want is to convert each SET of four bytes into a SINGLE 4 byte integer
as it:

>>> a = array([1,2,3,4],'1')
>>> a = fromstring(a.tostring(),Int32)
>>> a
array([67305985],'i')   

The four one byte items in a are turned into one four byte item. What I
want is to be able to do this in place, rather than have tostring()
create a copy. I think fromstring may create a copy as well, having a
possible total of three copies around at once. Does anyone know how many
copies will be around at once with this line of code?

-Chris

--

-- 
Christopher Barker,
Ph.D.                                                           
cbarker <at> jps.net                      ---           ---           ---
http://www.jps.net/cbarker          ----- <at>  <at>        ----- <at>  <at>        ----- <at>  <at>
(Continue reading)

Permalink | Reply | 2 comments |
headers
Janko Hauser | 3 Nov 2000 22:06
Picon

Re: "formstring()" in place?

Chris Barker writes:
 > "Paul F. Dubois" wrote:
 > > >>> y=array([1,2,3], '1')
 > > >>> y
 > > array([1, 2, 3],'1')
 > > >>> y.astype(Int32)
 > > array([1, 2, 3],'i')
 > 
 > Actually, this is exactly NOT what I want to do. In this case, each 1
 > byte interger was converted to a 4byte integer, of the same VALUE. What
 > I want is to convert each SET of four bytes into a SINGLE 4 byte integer
 > as it:
 > 
 > >>> a = array([1,2,3,4],'1')
 > >>> a = fromstring(a.tostring(),Int32)
 > >>> a
 > array([67305985],'i')   
 > 

A brut force way would be to do the transformation yourself :-)

>>> bits=array([1,256,256*256,256*256*256])
>>> sum(array([1,2,3,4])*bits)
67305985

So you need to reshape your array into (?,4) and multiply by bits.

And regarding your numpyio question, you can also read characters,
which are then put into an array by itself. It seems you have a very
messy file format (but the data world is never easy)
(Continue reading)

Permalink | Reply | 0 comments |

Gmane