Ralf Gommers | 27 Jun 00:43 2016
Picon

welcome Nikolay to the core team

Hi all,

On behalf of the Scipy developers I'd like to welcome Nikolay Mayorov as member of the core dev team.

Nikolay started contributing as a GSoC student last year, then he added optimize.lsq_linear/least_squares. Since then he's made many other contributions, for example: optimize.solve_bvp, interpolate.CubicSpline and a nearest neighbor chain algorithm to speed up cluster.linkage. See https://github.com/scipy/scipy/pulls/nmayorov

Keep them coming Nikolay!

Cheers,
Ralf

<at> other-newer-core-devs: if you're wondering if we all agreed on offering Nikolay commit rights, that happened already half a year ago via email (it just took a while before the acceptance came in:)).

_______________________________________________
SciPy-Dev mailing list
SciPy-Dev <at> scipy.org
https://mail.scipy.org/mailman/listinfo/scipy-dev
David Shi | 25 Jun 23:10 2016
Picon

How best to turn JSON into a CSV or Pandas data frame table?

Which are the best ways to turn a JSON object into a CSV or Pandas data frame table?

Looking forward to hearing from you.

Regards.

David
_______________________________________________
SciPy-Dev mailing list
SciPy-Dev <at> scipy.org
https://mail.scipy.org/mailman/listinfo/scipy-dev
Andrew Nelson | 22 Jun 05:08 2016
Picon

Re: [scipy/scipy] 0.18.0 release candidate 1

The release candidate passes all the tests on my machine.

OSX 10.9.5
python 3.4.4 (conda)
numpy 1.12.0.dev0+6ab89b8

On 20 June 2016 at 23:17, Evgeni Burovski <notifications <at> github.com> wrote:

This is the first release candidate for scipy 0.18.0. See https://github.com/scipy/scipy/blob/maintenance/0.18.x/doc/release/0.18.0-notes.rst for the release notes.

Please note that this is a source-only release.

If no issues are reported for this release, it will become the final 0.18.0 release. Issues can be reported via Github or on the scipy-dev mailing list (see http://scipy.org/scipylib/mailing-lists.html).


You are receiving this because you are subscribed to this thread.
View it on GitHub or mute the thread.




--
_____________________________________
Dr. Andrew Nelson


_____________________________________
_______________________________________________
SciPy-Dev mailing list
SciPy-Dev <at> scipy.org
https://mail.scipy.org/mailman/listinfo/scipy-dev
Evgeni Burovski | 20 Jun 14:51 2016
Picon
Gravatar

maintenance/0.18.x tagged

Hi,

0.18.x has been tagged and master is now open for 0.19.0 development.
I'll upload the source tarballs to GH releases shortly.

Cheers,

Evgeni
Bhavika Tekwani | 19 Jun 11:18 2016
Picon

Review pull request #6285: BUG: stats: Inconsistency in the multivariate_normal docstring #6263

Hey everyone,

I just submitted a pull request attempting to close issue #6263 regarding the pdf and logpdf methods in multivariate_normal. 
As I understood - the bug was simply an absence of default parameters in pdf and logpdf methods within class multivariate_normal_gen.

I've submitted this change along with a test for default values. 

Please see the PR here - https://github.com/scipy/scipy/pull/6285

I hope this is correct and I welcome guidance if it is not. 



Thank you,
Bhavika T. 

_______________________________________________
SciPy-Dev mailing list
SciPy-Dev <at> scipy.org
https://mail.scipy.org/mailman/listinfo/scipy-dev
Matt Newville | 15 Jun 14:19 2016
Gravatar

Re: SciPy-Dev Digest, Vol 152, Issue 6



On Wed, Jun 15, 2016 at 7:00 AM, <scipy-dev-request <at> scipy.org> wrote:
Send SciPy-Dev mailing list submissions to
        scipy-dev <at> scipy.org

To subscribe or unsubscribe via the World Wide Web, visit
        https://mail.scipy.org/mailman/listinfo/scipy-dev
or, via email, send a message with subject or body 'help' to
        scipy-dev-request <at> scipy.org

You can reach the person managing the list at
        scipy-dev-owner <at> scipy.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of SciPy-Dev digest..."


Today's Topics:

   1. optimize: what should happen if objective functions       return
      non-finite numbers? (Andrew Nelson)
   2. Re: optimize: what should happen if objective functions
      return non-finite numbers? (josef.pktd <at> gmail.com)


----------------------------------------------------------------------

Message: 1
Date: Wed, 15 Jun 2016 09:19:45 +1000
From: Andrew Nelson <andyfaff <at> gmail.com>
To: SciPy Developers List <scipy-dev <at> scipy.org>
Subject: [SciPy-Dev] optimize: what should happen if objective
        functions       return non-finite numbers?
Message-ID:
        <CAAbtOZdiCO-_bcqF0UPy-Bn6SmmqZfcOPZmdEBAiQchbCpf_zA <at> mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Consider the following example which raises an AssertionError:

import numpy as np
from scipy.optimize import minimize
def func1(x):
    return np.nan
x0 = np.array([1.3, 0.7, 0.8, 1.9, 1.2])
res = minimize(func1, x0, method='l-bfgs-b')
assert(res.success = False)

minimize simply returns the starting values of: res.x == x0. The reason I
came up with this example is that unsanitised datasets sometimes contain
nan or inf. Thus, if func1 was calculating chi2 and you were using minimize
then the entire fit would appear to succeed (res.success is True), but the
output would be garbage. Ok, so res.message is CONVERGENCE:
NORM_OF_PROJECTED_GRADIENT_<=_PGTOL, but it's not an clear indicator that
something went wrong.
A second example is:

import numpy as np
from scipy.optimize import curve_fit
def func2(x, a, b, c):
    return a * np.exp(-b * x) + c

def func3(x, a, b, c):
    return np.nan

xdata = np.linspace(0, 4, 50)
y = func2(xdata, 2.5, 1.3, 0.5)
ydata = y + 0.2 * np.random.normal(size=len(xdata))

popt, pcov = curve_fit(func3, xdata, ydata)
print(popt)

Whilst there is a warning (OptimizeWarning: Covariance of the parameters
could not be estimated) it's not a clear indicator that something has gone
wrong.
The behaviour one might expect in both examples could be to see a
ValueError raised if there arenp.nan values returned from the objective
function. I'm not totally sure of what to do if +/- np.inf is returned
(-inf would be a very good global minimum).


--
_____________________________________
Dr. Andrew Nelson


_____________________________________
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.scipy.org/pipermail/scipy-dev/attachments/20160615/1ec5735c/attachment-0001.html>

------------------------------

Message: 2
Date: Tue, 14 Jun 2016 22:40:59 -0400
From: josef.pktd <at> gmail.com
To: SciPy Developers List <scipy-dev <at> scipy.org>
Subject: Re: [SciPy-Dev] optimize: what should happen if objective
        functions return non-finite numbers?
Message-ID:
        <CAMMTP+DAK9nFV4Pg24W7=eh5kNgdGrag4mew7s51JJpnVEqdRg <at> mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

On Tue, Jun 14, 2016 at 7:19 PM, Andrew Nelson <andyfaff <at> gmail.com> wrote:

> Consider the following example which raises an AssertionError:
>
> import numpy as np
> from scipy.optimize import minimize
> def func1(x):
>     return np.nan
> x0 = np.array([1.3, 0.7, 0.8, 1.9, 1.2])
> res = minimize(func1, x0, method='l-bfgs-b')
> assert(res.success = False)
>
> minimize simply returns the starting values of: res.x == x0. The reason I
> came up with this example is that unsanitised datasets sometimes contain
> nan or inf. Thus, if func1 was calculating chi2 and you were using minimize
> then the entire fit would appear to succeed (res.success is True), but the
> output would be garbage. Ok, so res.message is CONVERGENCE:
> NORM_OF_PROJECTED_GRADIENT_<=_PGTOL, but it's not an clear indicator that
> something went wrong.
> A second example is:
>
> import numpy as np
> from scipy.optimize import curve_fit
> def func2(x, a, b, c):
>     return a * np.exp(-b * x) + c
>
> def func3(x, a, b, c):
>     return np.nan
>
> xdata = np.linspace(0, 4, 50)
> y = func2(xdata, 2.5, 1.3, 0.5)
> ydata = y + 0.2 * np.random.normal(size=len(xdata))
>
> popt, pcov = curve_fit(func3, xdata, ydata)
> print(popt)
>
> Whilst there is a warning (OptimizeWarning: Covariance of the parameters
> could not be estimated) it's not a clear indicator that something has gone
> wrong.
> The behaviour one might expect in both examples could be to see a
> ValueError raised if there arenp.nan values returned from the objective
> function. I'm not totally sure of what to do if +/- np.inf is returned
> (-inf would be a very good global minimum).
>


In my opinion optimizers should not (never?) raise exception, warn and
return whatever is available so the user can investigate.
I'm seeing nans every once in a while, but even if the objective function
returns nan, we often have finite parameters that can be used to
investigate for example gradients and similar.


We do see questions related to NaNs with lmfit, rather often.  It might be that the lmfit users are somewhat less meticulous than scipy developers when fitting data. 

My first reaction to Andrews suggestion was also to use a Warning, but he convinced me that an Exception is better. Just as a general Python Philosophy question, what does "ValueError" mean?   To me, it means "this Value caused an Error".  And, NaNs (and Infs) definitely cause the Fortran/C routines doing optimization to not even try to improve the solution -- that is, to do what they are supposed to do.  My view is that saying, "Warning, you got bad results because of bad data" would be too easy to ignore.

--Matt Newville

_______________________________________________
SciPy-Dev mailing list
SciPy-Dev <at> scipy.org
https://mail.scipy.org/mailman/listinfo/scipy-dev
Andrew Nelson | 15 Jun 01:19 2016
Picon

optimize: what should happen if objective functions return non-finite numbers?

Consider the following example which raises an AssertionError:

import numpy as np
from scipy.optimize import minimize
def func1(x):
    return np.nan
x0 = np.array([1.3, 0.7, 0.8, 1.9, 1.2])
res = minimize(func1, x0, method='l-bfgs-b')
assert(res.success = False)

minimize simply returns the starting values of: res.x == x0. The reason I came up with this example is that unsanitised datasets sometimes contain nan or inf. Thus, if func1 was calculating chi2 and you were using minimize then the entire fit would appear to succeed (res.success is True), but the output would be garbage. Ok, so res.message is CONVERGENCE: NORM_OF_PROJECTED_GRADIENT_<=_PGTOL, but it's not an clear indicator that something went wrong.
A second example is:

import numpy as np
from scipy.optimize import curve_fit
def func2(x, a, b, c):
    return a * np.exp(-b * x) + c

def func3(x, a, b, c):
    return np.nan

xdata = np.linspace(0, 4, 50)
y = func2(xdata, 2.5, 1.3, 0.5)
ydata = y + 0.2 * np.random.normal(size=len(xdata))

popt, pcov = curve_fit(func3, xdata, ydata)
print(popt)

Whilst there is a warning (OptimizeWarning: Covariance of the parameters could not be estimated) it's not a clear indicator that something has gone wrong.
The behaviour one might expect in both examples could be to see a ValueError raised if there arenp.nan values returned from the objective function. I'm not totally sure of what to do if +/- np.inf is returned (-inf would be a very good global minimum).


--
_____________________________________
Dr. Andrew Nelson


_____________________________________
_______________________________________________
SciPy-Dev mailing list
SciPy-Dev <at> scipy.org
https://mail.scipy.org/mailman/listinfo/scipy-dev
Joshua Wilson | 12 Jun 21:23 2016
Picon

PR adding a cython interface for special

Hello all,

This is a request for comments on

https://github.com/scipy/scipy/pull/6195

which adds a cython interface for scalar, typed versions of most of the functions in scipy.special. The documentation can be found here:

https://circle-artifacts.com/gh/scipy/scipy/52/artifacts/0/home/ubuntu/scipy/doc/build/html-scipyorg/linalg.cython_special.html

- Josh
_______________________________________________
SciPy-Dev mailing list
SciPy-Dev <at> scipy.org
https://mail.scipy.org/mailman/listinfo/scipy-dev
G Young | 8 Jun 14:57 2016
Picon

Re: raise on boolean sparse.__neg__

Hi,

I have had a PR open for quite some time now that raises when we attempt to take the negative of a boolean sparse matrix.  While this has been given the +1 by  <at> perimosocordiae, as this is a backwards-incompatible change, he wanted to make sure others were okay with this change before merging.

The rationale for raising on a boolean sparse matrix is nicely explained by here, and I wanted to send this out to the mailing list to see if other maintainers and developers agree with this logic.

Thanks!
_______________________________________________
SciPy-Dev mailing list
SciPy-Dev <at> scipy.org
https://mail.scipy.org/mailman/listinfo/scipy-dev
Felix Berkenkamp | 7 Jun 23:23 2016
Picon

PR: Deprecate lti cross-class properties/setters

Hi all,

I may be a bit late, but I just opened a PR that I would preferably still like to see go into 0.18. It deprecates historic properties/setters of lti systems, such as the ability to set the numerator of a StateSpace system.

Internally these kind of operations are very hacky and numerically inaccurate, so I think it is best to deprecate them.
_______________________________________________
SciPy-Dev mailing list
SciPy-Dev <at> scipy.org
https://mail.scipy.org/mailman/listinfo/scipy-dev
Evgeni Burovski | 4 Jun 15:10 2016
Picon
Gravatar

scipy 0.18 release

Hi,

It's seems to be about time to start planning the release of scipy 0.18.

I volunteer to do the legwork for this release. The proposed schedule would be:

20 June: rc1
27 June: rc2 (if necessary)
11 July:  final release

Thoughts?

Here's the current contents of the milestone:
https://github.com/scipy/scipy/milestones/0.18.0

Any help with addressing open issues and reviewing/merging open PRs
would be most appreciated.

Cheers,

Evgeni

Gmane