headers
David Cournapeau | 1 Oct 03:25
Picon

Re: memoryError when i have plenty of available ram

On Thu, Oct 1, 2009 at 3:58 AM, Gustaf Nilsson <gustaf <at> laserpanda.com> wrote:
> Hiya
>
> I know someone just started a memory thread, but i didnt wanna hijack it..
> My image processing app that im working on seems to crash with "memoryError"
> when it hits about 1.1gb of mem usage (same on two computers; has 2/4gb ram,
> xp 32bit)

If possible, a small script which reproduces the problem would be helpful.

Keep in mind that on windows, by default, your python script cannot
use more than 2 Gb anyway, even if you have 4Gb of memory.

David
Permalink | Reply | 6 comments |
headers
Ryan May | 1 Oct 03:25
Picon
Gravatar

Re: numpy.squeeze not squeezing

On Wed, Sep 30, 2009 at 5:21 PM, Robert Kern <robert.kern <at> gmail.com> wrote:
> 2009/9/30 Bruce Ford <bruce <at> clearscienceinc.com>:
>> print type(swh1)  #gave <type 'numpy.ndarray'>
>>
>> print type(swh)  #gave <type 'netCDF4.Variable'>
>
> Ah, yes. The latter is what I meant.
>
> Yup, my diagnosis is correct. np.squeeze() is interpreting swh as a
> scalar (or rank 0 array) with dtype=object rather than an array. You
> will have to get a real ndarray from the Variable. I am not familiar
> with the netcdf4 API, so you will have to refer to its documentation
> on how to do that. It won't be as simple as np.asarray(swh), I am
> afraid.

If it's anything like the other NetCDF bindings, it's just:

   swh_arr = swh[:]

Ryan

--

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma
Sent from Norman, Oklahoma, United States
Permalink | Reply | 1 comment |
headers
David Goldsmith | 1 Oct 05:00
Picon

Re: memoryError when i have plenty of available ram

On Wed, Sep 30, 2009 at 6:25 PM, David Cournapeau <cournape <at> gmail.com> wrote:

On Thu, Oct 1, 2009 at 3:58 AM, Gustaf Nilsson <gustaf <at> laserpanda.com> wrote:
> Hiya
>
> I know someone just started a memory thread, but i didnt wanna hijack it..
> My image processing app that im working on seems to crash with "memoryError"
> when it hits about 1.1gb of mem usage (same on two computers; has 2/4gb ram,
> xp 32bit)

If possible, a small script which reproduces the problem would be helpful.

Keep in mind that on windows, by default, your python script cannot
use more than 2 Gb anyway, even if you have 4Gb of memory.

Interesting.  Is this true in Vista?  Windows 7?  You say "by default": is there a trivial "workaround" (other than dividing up your memory usage, e.g., arrays, into blocks smaller than 2Gb)?

DG

David
_______________________________________________
SciPy-User mailing list
SciPy-User <at> scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-user

_______________________________________________
SciPy-User mailing list
SciPy-User <at> scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-user
Permalink | Reply | 5 comments |
headers
David Cournapeau | 1 Oct 05:01
Picon
Picon

Re: memoryError when i have plenty of available ram

David Goldsmith wrote:
> On Wed, Sep 30, 2009 at 6:25 PM, David Cournapeau <cournape <at> gmail.com
> <mailto:cournape <at> gmail.com>> wrote:
>
>     On Thu, Oct 1, 2009 at 3:58 AM, Gustaf Nilsson
>     <gustaf <at> laserpanda.com <mailto:gustaf <at> laserpanda.com>> wrote:
>     > Hiya
>     >
>     > I know someone just started a memory thread, but i didnt wanna
>     hijack it..
>     > My image processing app that im working on seems to crash with
>     "memoryError"
>     > when it hits about 1.1gb of mem usage (same on two computers;
>     has 2/4gb ram,
>     > xp 32bit)
>
>     If possible, a small script which reproduces the problem would be
>     helpful.
>
>     Keep in mind that on windows, by default, your python script cannot
>     use more than 2 Gb anyway, even if you have 4Gb of memory.
>
>
> Interesting.  Is this true in Vista?  Windows 7? 

It is true for (at least) most OSes, actually, and a limitation of 32
bits addressing. The only workaround is to use several processes. The
origin is that a process cannot 'see' more than 4 Gb in 32 bits, and
part of it has to be reserved for the kernel - windows and linux by
default limit the virtual adressing to 2 Gb per process in the userland.
There are options to split between 3 Gb user /1Gb kernel or the contrary
in linux, and similar in windows.

There is this pretty good explanation here for linux for the gory
details: http://kerneltrap.org/node/2450 (I would be surprised if
windows kernel was fundamentally different - except for the fork thing
of course).

The true solution is to use a 64 bits OS.

cheers,

David
Permalink | Reply | 4 comments |
headers
Jeremy Conlin | 1 Oct 05:59
Picon

Anyone have an example of using arpack (scipy.sparse.linalg.eigen)?

I need to use the arpack wrapper in scipy.  Does anyone have an
example of how they used this?  This would be great to get me started
in my research.

Thanks,
Jeremy
Permalink | Reply | 2 comments |
headers
Sturla Molden | 1 Oct 09:12
Picon

optimize.leastsq does not converge with full Jacobian?

I was trying to fit some Michaelis-Menten data from Bates and Watts 
(1988) puromycin experiment to test non-linear regression in SciPy. I 
get some very strange results that I don't understand.

First the non-linear model to be fitted is:

y = Vmax * x / (x + Km)

Not using Jacobian works:

import numpy as np
import scipy
from scipy.linalg import qr, solve
from scipy.optimize import leastsq

# Bates and Watts (1988) puromycin data

data = ((0.02, 47, 76),
        (0.06, 97, 107),
        (0.11, 123, 139),
        (0.22, 152, 159),
        (0.56, 191, 201),
        (1.10, 200, 207))

data = np.array(data)

X = data[:,0:1].repeat(2,axis=1).flatten()
Y = data[:,1:].flatten()

# initial fit from Linewaver-Burk plot
y = Y**-1
x = np.vstack((np.ones(X.shape),X**-1)).T
q,r = qr(x, econ=True)
b = solve(r, (np.mat(y) * np.mat(q)).T).ravel()

# Michaelis-Menten fit from Lineweaver-Burk
Vmax = 1.0 / b[0]
Km = b[1] * Vmax

# refit with Levenberg-Marquardt method

def michaelis_menten(t, x):
    Vmax, Km = t
    return Vmax*x/(x + Km)

def residuals(t, x, y):
    return y - michaelis_menten(t, x)

(Vmax,Km),ierr = leastsq(residuals, (Vmax,Km), args=(X,Y))

This gives Vmax,Km = 2.12683559e+02, 6.41209954e-02, which is the 
"correct" answer.

However, when I use the full Jacobian,

def jacobian(t, x, y):
    j = np.zeros((2,x.shape[0]))
    Vmax, Km = t
    j[0,:] = x/(Km + x)
    j[1,:] = -Vmax*x/((Km + x)**2)
    return j

(Vmax,Km),ierr = leastsq(residuals, (Vmax,Km), args=(X,Y), 
Dfun=jacobian, col_deriv=1)

I always get the start value of (Vmax,Km) returned, which is (195.8027, 
0.0484065).

What on earth is going on?

Is there a bug in SciPy or am I being incredibly stupid?

Sturla Molden
Permalink | Reply | 4 comments |
headers
Pauli Virtanen | 1 Oct 09:41
Picon
Picon
Favicon

Re: optimize.leastsq does not converge with full Jacobian?

Thu, 01 Oct 2009 09:12:19 +0200, Sturla Molden wrote:
[clip]
> def residuals(t, x, y):
>     return y - michaelis_menten(t, x)
[clip]
> def jacobian(t, x, y):
>     j = np.zeros((2,x.shape[0]))
>     Vmax, Km = t
>     j[0,:] = x/(Km + x)
>     j[1,:] = -Vmax*x/((Km + x)**2)
>     return j

Sign error.

--

-- 
Pauli Virtanen
Permalink | Reply | 2 comments |
headers
Sebastian Walter | 1 Oct 10:18
Picon
Gravatar

Re: optimize.leastsq does not converge with full Jacobian?

is it ok with you if I use your code in the unit test of the automatic
differentiation tool pyadolc?
(http://github.com/b45ch1/pyadolc)

regards,
Sebastian

On Thu, Oct 1, 2009 at 9:12 AM, Sturla Molden <sturla <at> molden.no> wrote:
> I was trying to fit some Michaelis-Menten data from Bates and Watts
> (1988) puromycin experiment to test non-linear regression in SciPy. I
> get some very strange results that I don't understand.
>
> First the non-linear model to be fitted is:
>
> y = Vmax * x / (x + Km)
>
> Not using Jacobian works:
>
> import numpy as np
> import scipy
> from scipy.linalg import qr, solve
> from scipy.optimize import leastsq
>
>
> # Bates and Watts (1988) puromycin data
>
> data = ((0.02, 47, 76),
>        (0.06, 97, 107),
>        (0.11, 123, 139),
>        (0.22, 152, 159),
>        (0.56, 191, 201),
>        (1.10, 200, 207))
>
> data = np.array(data)
>
> X = data[:,0:1].repeat(2,axis=1).flatten()
> Y = data[:,1:].flatten()
>
> # initial fit from Linewaver-Burk plot
> y = Y**-1
> x = np.vstack((np.ones(X.shape),X**-1)).T
> q,r = qr(x, econ=True)
> b = solve(r, (np.mat(y) * np.mat(q)).T).ravel()
>
> # Michaelis-Menten fit from Lineweaver-Burk
> Vmax = 1.0 / b[0]
> Km = b[1] * Vmax
>
> # refit with Levenberg-Marquardt method
>
> def michaelis_menten(t, x):
>    Vmax, Km = t
>    return Vmax*x/(x + Km)
>
> def residuals(t, x, y):
>    return y - michaelis_menten(t, x)
>
> (Vmax,Km),ierr = leastsq(residuals, (Vmax,Km), args=(X,Y))
>
> This gives Vmax,Km = 2.12683559e+02, 6.41209954e-02, which is the
> "correct" answer.
>
> However, when I use the full Jacobian,
>
> def jacobian(t, x, y):
>    j = np.zeros((2,x.shape[0]))
>    Vmax, Km = t
>    j[0,:] = x/(Km + x)
>    j[1,:] = -Vmax*x/((Km + x)**2)
>    return j
>
> (Vmax,Km),ierr = leastsq(residuals, (Vmax,Km), args=(X,Y),
> Dfun=jacobian, col_deriv=1)
>
> I always get the start value of (Vmax,Km) returned, which is (195.8027,
> 0.0484065).
>
> What on earth is going on?
>
> Is there a bug in SciPy or am I being incredibly stupid?
>
>
> Sturla Molden
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User <at> scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
Permalink | Reply | 0 comments |
headers
Gustaf Nilsson | 1 Oct 13:08
Picon

Re: memoryError when i have plenty of available ram

yeah im aware of the 2 gb limit, and i expect to upgrade to 64bit (w7) next month...
still, my program doesnt even get close to that before the memoryError.
Will see if i can make a small script that does the same thing when i get home fron work tonight.

Hey you are in kyoto? ive spent a few months in the north of kyoto (matsugasaki/kuramaguchi)
Will be going back for a wedding in april (sakura!)


On Thu, Oct 1, 2009 at 4:01 AM, David Cournapeau <david <at> ar.media.kyoto-u.ac.jp> wrote:
David Goldsmith wrote:
> On Wed, Sep 30, 2009 at 6:25 PM, David Cournapeau <cournape <at> gmail.com
> <mailto:cournape <at> gmail.com>> wrote:
>
>     On Thu, Oct 1, 2009 at 3:58 AM, Gustaf Nilsson
>     <gustaf <at> laserpanda.com <mailto:gustaf <at> laserpanda.com>> wrote:
>     > Hiya
>     >
>     > I know someone just started a memory thread, but i didnt wanna
>     hijack it..
>     > My image processing app that im working on seems to crash with
>     "memoryError"
>     > when it hits about 1.1gb of mem usage (same on two computers;
>     has 2/4gb ram,
>     > xp 32bit)
>
>     If possible, a small script which reproduces the problem would be
>     helpful.
>
>     Keep in mind that on windows, by default, your python script cannot
>     use more than 2 Gb anyway, even if you have 4Gb of memory.
>
>
> Interesting.  Is this true in Vista?  Windows 7?

It is true for (at least) most OSes, actually, and a limitation of 32
bits addressing. The only workaround is to use several processes. The
origin is that a process cannot 'see' more than 4 Gb in 32 bits, and
part of it has to be reserved for the kernel - windows and linux by
default limit the virtual adressing to 2 Gb per process in the userland.
There are options to split between 3 Gb user /1Gb kernel or the contrary
in linux, and similar in windows.

There is this pretty good explanation here for linux for the gory
details: http://kerneltrap.org/node/2450 (I would be surprised if
windows kernel was fundamentally different - except for the fork thing
of course).

The true solution is to use a 64 bits OS.

cheers,

David
_______________________________________________
SciPy-User mailing list
SciPy-User <at> scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-user



--
■ ■ ■ ■ ■ ■ ■ ■ ■ ■
_______________________________________________
SciPy-User mailing list
SciPy-User <at> scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-user
Permalink | Reply | 3 comments |
headers
David Cournapeau | 1 Oct 12:53
Picon
Picon

Re: memoryError when i have plenty of available ram

Gustaf Nilsson wrote:
> yeah im aware of the 2 gb limit, and i expect to upgrade to 64bit (w7)
> next month...
> still, my program doesnt even get close to that before the memoryError.

Depending on the action, it may be a bug in the called function, or the
function is just memory hungry (it can also be a reference count error
somewhere, but let's hope it won't go to that).

> Hey you are in kyoto? ive spent a few months in the north of kyoto
> (matsugasaki/kuramaguchi)

Yes, still in Kyoto, in the north as well - but not as far as kurama (I
am still in the city, in Demachiyanagi near kyodai). Sakura in Kyoto is
nice :)

cheers,

David
Permalink | Reply | 1 comment |

Gmane