headers
Brennan Williams | 1 Aug 04:42
Gravatar

Re: scipy.io.numpyio fwrite - appending or updating an array

I've tried replacing numpyio with both fopen and now also npfile but I'm 
getting the same problem, i.e. if I write a numpy array to the file,
everything else before that position in the file is now zero. It is as 
if it is a new file, not an existing one.

Brennan Williams wrote:
> I have an existing binary file containing numpy array data. It has been 
> created using open,fwrite & close and I can read the data using fread.
>
> I want to be able to either append a new array to the end of the file or 
> update an existing array within the file.
>
> I've tried opening the file with a mode of either 'ab+' or 'wb+' and 
> then writing the data using something like....
>
>      fd = open(vfname, 'ab+')
>      if fd:
>        filepos=(self.id-1)*self.yarray.size*4
>        fd.seek(filepos)
>        fwrite(fd, self.yarray.size, self.yarray,'f')
>        fd.close()
>
> When I use a mode of 'ab+' it looks like the data has been written to 
> the file ok (no errors reported) but when I read it back I get my 
> original data.
>
> When I use 'wb+' then my updated data gets written and read back ok. But 
> when I reload the file, everything apart from my updated data (i.e. 
> everything before it in the file) is now zero.
>
(Continue reading)

Permalink | Reply | 5 comments |
headers
Stéfan van der Walt | 1 Aug 09:04
Picon
Picon
Favicon

Re: scipy.io.numpyio fwrite - appending or updating an array

Hi Brennan

2008/8/1 Brennan Williams <brennan.williams <at> visualreservoir.com>:
> I've tried replacing numpyio with both fopen and now also npfile but I'm
> getting the same problem, i.e. if I write a numpy array to the file,
> everything else before that position in the file is now zero. It is as
> if it is a new file, not an existing one.

Have you looked at SciPy's memmap class?

Regards
Stéfan
Permalink | Reply | 0 comments |
headers
Stéfan van der Walt | 1 Aug 09:50
Picon
Picon
Favicon

Re: scipy.io.numpyio fwrite - appending or updating an array

Hi Brennan

2008/8/1 Brennan Williams <brennan.williams <at> visualreservoir.com>:
> I've tried replacing numpyio with both fopen and now also npfile but I'm
> getting the same problem, i.e. if I write a numpy array to the file,
> everything else before that position in the file is now zero. It is as
> if it is a new file, not an existing one.

Have you looked at SciPy's memmap class?

Regards
Stéfan
Permalink | Reply | 0 comments |
headers
Samuel GARCIA | 1 Aug 10:47
Picon
Favicon
Gravatar

Numpy : in place moving data strange behaviour

Hi list,
I have a strange behaviour of numpy if I try to move in place
a part of my array in an other place of the array.
For 1D or for 2D there is not the same behaviour.
The solution is of course copying the part I have move of thhe array.
See the code :

from scipy import *

# for 1D
a = arange(10)
print a # a = [0 1 2 3 4 5 6 7 8 9]
a[0:7] = a[3:10]
print a # OK : a = [3 4 5 6 7 8 9 7 8 9]

a = arange(10)
print a
a[3:10] = a[0:7]
print a # OK a = [0 1 2 0 1 2 3 4 5 6]

# for 2D
a = concatenate( (arange(10)[newaxis,:] , arange(10)[newaxis,:]) )
print a
#[[0 1 2 3 4 5 6 7 8 9]
# [0 1 2 3 4 5 6 7 8 9]]
a[:,0:7] = a[:,3:10]
print a
#OK a=
# [[3 4 5 6 7 8 9 7 8 9]
# [3 4 5 6 7 8 9 7 8 9]]
(Continue reading)

Permalink | Reply | 1 comment |
headers
Robert Kern | 1 Aug 11:06
Picon
Gravatar

Re: Numpy : in place moving data strange behaviour

On Fri, Aug 1, 2008 at 03:47, Samuel GARCIA <sgarcia <at> olfac.univ-lyon1.fr> wrote:
> Hi list,
> I have a strange behaviour of numpy if I try to move in place
> a part of my array in an other place of the array.
> For 1D or for 2D there is not the same behaviour.

Basically, you're treading into areas where we make no guarantees. The
exact order we iterate over the arrays should be considered an
implementation detail, and you should not rely on it. This should only
affect your results in cases where you are modifying an array inplace
where the source is an overlapping view of that very array.

Without diving into the source, I think what's happening is that the
1D version happens to deal with contiguous slices on both the LHS and
the RHS, so memcpy() is used or something similar which handles
overlaps specially. In the 2D case, neither is contiguous, so we
iterate manually in the naive way. For example, we can construct a 1D
non-contiguous case which demonstrates the same behavior:

>>> from numpy import *
>>> a = arange(10)
>>> a
array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
>>> a[2:9:2] = a[0:7:2]
>>> a
array([0, 1, 0, 3, 0, 5, 0, 7, 0, 9])

--

-- 
Robert Kern
(Continue reading)

Permalink | Reply | 0 comments |
headers
Brennan Williams | 1 Aug 11:14
Gravatar

Re: scipy.io.numpyio fwrite - appending or updating an array

I'll look into memmap as Stefan suggested. If Robert Kern's out there, 
do you have any comments about what I might be doing wrong.
Id on't know memmap at all yet - basically each file will have multiple 
numpy arrays written, read, appended and updated as required.

Brennan

Brennan Williams wrote:
> I've tried replacing numpyio with both fopen and now also npfile but I'm 
> getting the same problem, i.e. if I write a numpy array to the file,
> everything else before that position in the file is now zero. It is as 
> if it is a new file, not an existing one.
>
> Brennan Williams wrote:
>   
>> I have an existing binary file containing numpy array data. It has been 
>> created using open,fwrite & close and I can read the data using fread.
>>
>> I want to be able to either append a new array to the end of the file or 
>> update an existing array within the file.
>>
>> I've tried opening the file with a mode of either 'ab+' or 'wb+' and 
>> then writing the data using something like....
>>
>>      fd = open(vfname, 'ab+')
>>      if fd:
>>        filepos=(self.id-1)*self.yarray.size*4
>>        fd.seek(filepos)
>>        fwrite(fd, self.yarray.size, self.yarray,'f')
>>        fd.close()
(Continue reading)

Permalink | Reply | 2 comments |
headers
Robert Kern | 1 Aug 11:30
Picon
Gravatar

Re: scipy.io.numpyio fwrite - appending or updating an array

On Fri, Aug 1, 2008 at 04:14, Brennan Williams
<brennan.williams <at> visualreservoir.com> wrote:
> I'll look into memmap as Stefan suggested. If Robert Kern's out there,
> do you have any comments about what I might be doing wrong.

I think you want 'rb+'. 'ab+' puts you at the end of the file for
writing (because you asked to append). I believe the '+' in 'wb+' is
simply ignored, so you are getting the truncating behavior of 'wb'.
I've only tested 'rb+' with file.write(), but ultimately, both
file.write() and scipy.io.fwrite() both use C's fwrite(3) down at the
bottom.

In [40]: f = open('foo.dat', 'wb')

In [41]: f.write('Foo!' * 4)

In [42]: f.close()

In [43]: open('foo.dat', 'rb').read()
Out[43]: 'Foo!Foo!Foo!Foo!'

In [44]: f = open('foo.dat', 'rb+')

In [45]: f.tell()
Out[45]: 0L

In [46]: f.seek(4)

In [47]: f.tell()
Out[47]: 4L
(Continue reading)

Permalink | Reply | 1 comment |
headers
Brennan Williams | 1 Aug 11:46
Gravatar

Re: scipy.io.numpyio fwrite - appending or updating an array

Robert
Thanks for the help.
'rb+' seems to work.
'ab+' or 'ab' is working ok.
'wb' or 'wb+' doesn't work, i.e. everything preceding the position I'm 
writing to in the file becomes zero.
I re-read (slowly this time) the online documentation - in fopen the 
permissions are the same as for the built-in open so
I looked at that and yes, 'w+' will truncate.
However, to me, truncation means truncation "after" but it evidently 
also means truncation "before" as well which just wasn't what I was
expecting (probably due to a Fortran background many years ago).
It also seems strange to me to open a file with 'r+' when I want to 
write to it.
But there you go, it seems to be working.
I will also look at memmap - as the size of my data files gets bigger 
with larger datasets etc. it may well be very useful.

Brennan

Robert Kern wrote:
> On Fri, Aug 1, 2008 at 04:14, Brennan Williams
> <brennan.williams <at> visualreservoir.com> wrote:
>   
>> I'll look into memmap as Stefan suggested. If Robert Kern's out there,
>> do you have any comments about what I might be doing wrong.
>>     
>
> I think you want 'rb+'. 'ab+' puts you at the end of the file for
> writing (because you asked to append). I believe the '+' in 'wb+' is
(Continue reading)

Permalink | Reply | 0 comments |
headers
Mico Filós | 1 Aug 12:28
Picon

Re: Differential Algebraic Equation Solvers

Hi,

Please, correct me if I am wrong. Is PyDSTool a package which aims at
providing the functionalities of, say, XPP
(http://www.math.pitt.edu/~bard/xpp/xpp.html)? That would be great.
Although it is true that much can already be done with the existing
SciPy functions, it would be very helpful to have some wrapper or some
integrated environment (using ipython, matplotlib, NumPy/Scipy) for
the analysis of nonlinear systems. But perhaps this is not exactly
what the developers of PyDSTool have in mind. If this is the case, I
would appreciate to hear any suggestion about possible
Numpy/Scipy-based packages (or tips) that allow users to play
(plotting trajectories with different initial conditions, changing
parameters, drawing bifurcation diagrams, etc) with nonlinear systems
in an interactive and easy way.

Best,

M.
Permalink | Reply | 2 comments |
headers
David Huard | 1 Aug 15:23
Picon

Re: Is there a collection of useful functions/modules?

Hi Joshua,

There is a scipy cookbook at http://www.scipy.org/Cookbook

It's a great place to put those kinds of functions, but also to write your thoughts on scipy. Having a newcomers perspective is always useful for other folks starting with python.

Welcome aboard,

David

On Wed, Jul 30, 2008 at 10:42 AM, Joshua <shad0wfade <at> yahoo.com> wrote:
I'm very new to Python, as in only a week of programming, and was wondering if there is a page with a collection of highly useful functions/modules that are not necessarily maintained on a release-to-release basis for things like performing FFT's on data in a file.

I wrote a function to do this using Numpy, and put in options to normalize the FFT for me, and take the absolute value.  There are many other options I should add, but before I add complexity, I wouldn't mind having my code scrutinized, and made available for others to use and optimize.  That way students and researchers don't have to reinvent the wheel, unless they want to or are using some odd formatting scheme.  Just have the amplitude data in the list column wise.

exempli gratia:
0.0
0.1
0.0
-0.1
... et cetera.

This code makes no particular frequency sampling assumptions (other than the sampling was done correctly), and only deals with simple FFT processing.

I've attached the code and welcome scrutiny.

There are few things that I know I should to add/restructure to my code: better error handling, printing out the phase, multi-dimensional analysis, and |Magnitude| in 20dB.

Thanks for your help
Joshua



_______________________________________________
SciPy-user mailing list
SciPy-user <at> scipy.org
http://projects.scipy.org/mailman/listinfo/scipy-user


_______________________________________________
SciPy-user mailing list
SciPy-user <at> scipy.org
http://projects.scipy.org/mailman/listinfo/scipy-user
Permalink | Reply | 0 comments |

Gmane