headers
M Trumpis | 2 Jun 00:39
Picon
Favicon

scipy SVD with alternate lapack driver

Hi all.. I posted on the numpy list a few months back about running
into some unstable SVD behavior, namely 1) a non-converging SVD and 2)
negative singular values. My colleagues and I got curious when MATLAB
and Octave did not fail or give bogus results for the two operators.
Based on the Octave code and some testing with the lapack drivers, it
would appear that using the *GESVD family of drivers is more robust in
the face of these corner cases than the *GESDD drivers.

So after leaving off for months, I finally went back to fixing this
up. Using a lot of monkey-see-monkey-do, I added the f2py function
signatures for the *GESVD functions, hopefully sane code in
calc_lwork.f, and wrote up a bit of python in linalg/decomp.py, with
some appropriate testing. I haven't submitted a patch to scipy yet..
Can I just post the "svn diff" with a description of the issue at hand
on the Trac site? Although it does look like it's working, I haven't
tested it extensively yet. And I was *very* ignorantly hacking at the
Fortran / f2py bits.

Hope this is helpful.

Kind regards,
Mike
Permalink | Reply | 2 comments |
headers
M Trumpis | 2 Jun 00:39
Picon
Favicon

scipy SVD with alternate lapack driver

Hi all.. I posted on the numpy list a few months back about running
into some unstable SVD behavior, namely 1) a non-converging SVD and 2)
negative singular values. My colleagues and I got curious when MATLAB
and Octave did not fail or give bogus results for the two operators.
Based on the Octave code and some testing with the lapack drivers, it
would appear that using the *GESVD family of drivers is more robust in
the face of these corner cases than the *GESDD drivers.

So after leaving off for months, I finally went back to fixing this
up. Using a lot of monkey-see-monkey-do, I added the f2py function
signatures for the *GESVD functions, hopefully sane code in
calc_lwork.f, and wrote up a bit of python in linalg/decomp.py, with
some appropriate testing. I haven't submitted a patch to scipy yet..
Can I just post the "svn diff" with a description of the issue at hand
on the Trac site? Although it does look like it's working, I haven't
tested it extensively yet. And I was *very* ignorantly hacking at the
Fortran / f2py bits.

Hope this is helpful.

Kind regards,
Mike
Permalink | Reply | 0 comments |
headers
Robert Kern | 2 Jun 00:45
Picon
Gravatar

Re: scipy SVD with alternate lapack driver

On Mon, Jun 1, 2009 at 17:39, M Trumpis <mtrumpis <at> berkeley.edu> wrote:
> Hi all.. I posted on the numpy list a few months back about running
> into some unstable SVD behavior, namely 1) a non-converging SVD and 2)
> negative singular values. My colleagues and I got curious when MATLAB
> and Octave did not fail or give bogus results for the two operators.
> Based on the Octave code and some testing with the lapack drivers, it
> would appear that using the *GESVD family of drivers is more robust in
> the face of these corner cases than the *GESDD drivers.
>
> So after leaving off for months, I finally went back to fixing this
> up. Using a lot of monkey-see-monkey-do, I added the f2py function
> signatures for the *GESVD functions, hopefully sane code in
> calc_lwork.f, and wrote up a bit of python in linalg/decomp.py, with
> some appropriate testing. I haven't submitted a patch to scipy yet..
> Can I just post the "svn diff" with a description of the issue at hand
> on the Trac site?

Create a new ticket, and attach the .diff to it. At the bottom of the
"New Ticket" form is a check-box: "I have files to attach to this
ticket". Make sure that is checked, and you will be prompted for the
file.

> Although it does look like it's working, I haven't
> tested it extensively yet. And I was *very* ignorantly hacking at the
> Fortran / f2py bits.

For adding new LAPACK wrappers, that technique tends to work fairly well.

--

-- 
Robert Kern
(Continue reading)

Permalink | Reply | 1 comment |
headers
M Trumpis | 2 Jun 01:42
Picon
Favicon

Re: scipy SVD with alternate lapack driver

ok.. Thanks for the help.

http://projects.scipy.org/scipy/ticket/957

Mike

On Mon, Jun 1, 2009 at 3:45 PM, Robert Kern <robert.kern <at> gmail.com> wrote:
> On Mon, Jun 1, 2009 at 17:39, M Trumpis <mtrumpis <at> berkeley.edu> wrote:
>> Hi all.. I posted on the numpy list a few months back about running
>> into some unstable SVD behavior, namely 1) a non-converging SVD and 2)
>> negative singular values. My colleagues and I got curious when MATLAB
>> and Octave did not fail or give bogus results for the two operators.
>> Based on the Octave code and some testing with the lapack drivers, it
>> would appear that using the *GESVD family of drivers is more robust in
>> the face of these corner cases than the *GESDD drivers.
>>
>> So after leaving off for months, I finally went back to fixing this
>> up. Using a lot of monkey-see-monkey-do, I added the f2py function
>> signatures for the *GESVD functions, hopefully sane code in
>> calc_lwork.f, and wrote up a bit of python in linalg/decomp.py, with
>> some appropriate testing. I haven't submitted a patch to scipy yet..
>> Can I just post the "svn diff" with a description of the issue at hand
>> on the Trac site?
>
> Create a new ticket, and attach the .diff to it. At the bottom of the
> "New Ticket" form is a check-box: "I have files to attach to this
> ticket". Make sure that is checked, and you will be prompted for the
> file.
>
>> Although it does look like it's working, I haven't
(Continue reading)

Permalink | Reply | 0 comments |
headers
Fernando Perez | 2 Jun 07:20
Picon
Gravatar

Tutorial topics for SciPy'09 Conference

Hi all,

The time for the Scipy'09 conference is rapidly approaching, and we
would like to both announce the plan for tutorials and solicit
feedback from everyone on topics of interest.

Broadly speaking, the plan is something along the lines of  what we
had last year: one continuous 2-day tutorial  aimed at introductory
users, starting from the very basics, and in parallel a set of
'advanced' tutorials, consisting of a series of 2-hour sessions on
specific  topics.

We will request that the presenters for the advanced tutorials keep
the 'tutorial' word very much in mind, so that the sessions really
contain hands-on learning work and not simply a 2-hour long slide
presentation.  We will  thus require that all the tutorials will be
based on tools that the attendees can install at least 2 weeks in
advance on all  platforms (no "I released it last night" software).

With that in mind, we'd like feedback from all of you on possible
topics for the advanced tutorials.  We have space for 8 slots total,
and here are in no particular order some possible topics.  At this
point there are no guarantees yet that we can get presentations for
these, but we'd like to establish a first list of preferred topics to
try and secure the presentations as soon as possible.

This is simply a list of candiate topics that various people have
informally suggested so far:

- Mayavi/TVTK
(Continue reading)

Permalink | Reply | 2 comments |
headers
Nils Wagner | 2 Jun 20:05
Picon
Favicon

OperationalError: database is locked

Hi all,

I cannot access http://projects.scipy.org/scipy/timeline .
Is it a temporary problem ?

Nils
Permalink | Reply | 1 comment |
headers
ctw | 2 Jun 22:56

scipy.stats._chk_asarray

I just wanted to give you a heads up that I came across a bug in
scipy.stats._chk_asarray.
I have updated the corresponding review page (
http://projects.scipy.org/scipy/ticket/44 ) with the following
comment:

_chk_asarray does not respect the class of the input. If the input is
a subclass of numpy.ndarray _chk_asarray will transform it into an
ndarray. This can lead to undesirable consequences in functions that
call _chk_asarray (e.g., scipy.stats.nanmean).

Simple example:
>>> tst = matrix(range(4))
>>> scipy.stats.nanmean(tst)
 array([0., 1., 2., 3.])
>>> np.mean(tst,0)
 matrix([[0., 1., 2., 3.]])

The problem is the call to np.asarray() which should only be made for
input that is not an ndarray instance. I just committed a
corresponding fix via SVN.
Permalink | Reply | 22 comments |
headers
Robert Kern | 2 Jun 23:03
Picon
Gravatar

Re: scipy.stats._chk_asarray

On Tue, Jun 2, 2009 at 15:56, ctw <lists.20.chth <at> xoxy.net> wrote:
> I just wanted to give you a heads up that I came across a bug in
> scipy.stats._chk_asarray.
> I have updated the corresponding review page (
> http://projects.scipy.org/scipy/ticket/44 ) with the following
> comment:
>
> _chk_asarray does not respect the class of the input. If the input is
> a subclass of numpy.ndarray _chk_asarray will transform it into an
> ndarray. This can lead to undesirable consequences in functions that
> call _chk_asarray (e.g., scipy.stats.nanmean).
>
> Simple example:
>>>> tst = matrix(range(4))
>>>> scipy.stats.nanmean(tst)
>  array([0., 1., 2., 3.])
>>>> np.mean(tst,0)
>  matrix([[0., 1., 2., 3.]])
>
> The problem is the call to np.asarray() which should only be made for
> input that is not an ndarray instance. I just committed a
> corresponding fix via SVN.

Please revert that. Some of those functions rely on the result of
_chk_asarray() being a real ndarray. For example, nanstd() uses an
exponent, which will be misinterpreted by a matrix object:

In [1]: from scipy.stats import nanstd

In [2]: m = matrix(range(4))
(Continue reading)

Permalink | Reply | 1 comment |
headers
Pauli Virtanen | 2 Jun 23:07
Picon
Picon
Favicon

Re: OperationalError: database is locked

Tue, 02 Jun 2009 20:05:34 +0200, Nils Wagner wrote:
> I cannot access http://projects.scipy.org/scipy/timeline . Is it a
> temporary problem ?

Should be temporary. Some Trac instances seem to hang around for some 
minutes, keeping the DB locked from others... No idea what they are doing.

--

-- 
Pauli Virtanen
Permalink | Reply | 0 comments |
headers
Pierre GM | 2 Jun 23:14
Picon

Re: scipy.stats._chk_asarray


On Jun 2, 2009, at 5:03 PM, Robert Kern wrote:

> On Tue, Jun 2, 2009 at 15:56, ctw <lists.20.chth <at> xoxy.net> wrote:
>> I just wanted to give you a heads up that I came across a bug in
>> scipy.stats._chk_asarray.
>> I have updated the corresponding review page (
>> http://projects.scipy.org/scipy/ticket/44 ) with the following
>> comment:
>>
>> _chk_asarray does not respect the class of the input.

On purpose, as pointed out by Robert.

Now, for simple operations like that, you can use masked arrays:
 >>>  mx=ma.array(matrix(range(4)))
 >>> mx.mean(0)
masked_matrix(data =
  [[0.0 1.0 2.0 3.0]],
               mask =
  [[False False False False]],
         fill_value = 999999)
 >>> mx.mean(0).data
  matrix([[ 0.,  1.,  2.,  3.]])
Permalink | Reply | 0 comments |

Gmane