Developers of Scientific Python

Re: Fast (O(n log(n)) ) implementation of Kendall Tau

From: "Sturla Molden" <sturla <at> molden.no>
Sent: Thursday, October 01, 2009 6:03 AM

> Enzo Michelangeli skrev:
>> Dear all,
>>
>> A few days ago I posted to http://projects.scipy.org/scipy/ticket/999 a
>> drop-in replacement for scipy.stats.kendalltau. My code implements the
>> algorithm with complexity O(n log()) described by William R. Knight in a
>> paper of 1966 archived at http://www.jstor.org/pss/2282833 , whereas the
>> function currently in SciPy has complexity O(n^2), which makes it 
>> unusable
>>
> There is also:
>
> http://projects.scipy.org/scipy/ticket/893
>
> It has a contigency table version that would be fast for large data
> sets, in theory O(n).

Yes, but that requires a native module.

Enzo

headers

Enzo Michelangeli | 1 Oct 06:07

Re: Fast (O(n log(n)) ) implementation of Kendall Tau

From: "Sturla Molden" <sturla <at> molden.no>
Sent: Thursday, October 01, 2009 6:09 AM

> Enzo Michelangeli skrev:
>> Dear all,
>>
>> A few days ago I posted to http://projects.scipy.org/scipy/ticket/999 a
>> drop-in replacement for scipy.stats.kendalltau.
> Why do you re-implement mergesort in pure Python?
>
> ndarrays have a sort method that can use mergesort.
>
> Python lists has the same (timsort is mergesort on steroids).

Because Knight's algorithm needs to count the number of swaps (or, to be
more precise, the number of swaps that would be performed by an equivalent
bubblesort). In the code, that's the purpose of the variable exchcnt .

An alternative algorithm for the Kendall Tau developed by David Christensen,
and described at http://www.springerlink.com/content/p33qu44058984082/ (with
a Delphi implementation), replaces the mergesort step with one based on
balanced binary trees (AVL in his case, but I guess RBT would also work).
Unfortunately, neither the standard Python library nor NumPy/SciPy appear to
implement such useful data structures, and what is available either doesn't
allow O(log(n)) random access (heapq) or lacks a O(log(n)) insert (sorted
lists accessed through bisect).

Enzo

headers

Charles R Harris | 1 Oct 08:22

Re: Fast (O(n log(n)) ) implementation of Kendall Tau

On Wed, Sep 30, 2009 at 10:07 PM, Enzo Michelangeli <enzomich <at> gmail.com> wrote:

From: "Sturla Molden" <sturla <at> molden.no>

Sent: Thursday, October 01, 2009 6:09 AM

> Enzo Michelangeli skrev:
>> Dear all,
>>
>> A few days ago I posted to http://projects.scipy.org/scipy/ticket/999 a
>> drop-in replacement for scipy.stats.kendalltau.
> Why do you re-implement mergesort in pure Python?
>
> ndarrays have a sort method that can use mergesort.
>
> Python lists has the same (timsort is mergesort on steroids).

Because Knight's algorithm needs to count the number of swaps (or, to be
more precise, the number of swaps that would be performed by an equivalent
bubblesort). In the code, that's the purpose of the variable exchcnt .

An alternative algorithm for the Kendall Tau developed by David Christensen,
and described at http://www.springerlink.com/content/p33qu44058984082/ (with
a Delphi implementation), replaces the mergesort step with one based on
balanced binary trees (AVL in his case, but I guess RBT would also work).
Unfortunately, neither the standard Python library nor NumPy/SciPy appear to
implement such useful data structures, and what is available either doesn't

Yep, we could use some more of those useful data structures. A "computer science" library would be useful. The scipy.spacial library is step in that direction but it would be nice if there were more such.

Chuck

_______________________________________________
Scipy-dev mailing list
Scipy-dev <at> scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-dev

headers

Scott Sinclair | 1 Oct 10:28

Statsmodels documentation buglet

Hi,

I'm having a look at the statsmodels scikit and have noticed a minor
bug in the docs at:

http://statsmodels.sourceforge.net/generated/scikits.statsmodels.regression.OLS.html

The definition Y = [1,3,4,5,2,3,4], in the example contains a trailing
comma that causes the example to fail when used with IPython's cpaste
mode.

Cheers,
Scott

headers

Skipper Seabold | 1 Oct 17:01

Re: Statsmodels documentation buglet

On Thu, Oct 1, 2009 at 4:28 AM, Scott Sinclair
<scott.sinclair.za <at> gmail.com> wrote:
> Hi,
>
> I'm having a look at the statsmodels scikit and have noticed a minor
> bug in the docs at:
>
> http://statsmodels.sourceforge.net/generated/scikits.statsmodels.regression.OLS.html
>
> The definition Y = [1,3,4,5,2,3,4], in the example contains a trailing
> comma that causes the example to fail when used with IPython's cpaste
> mode.
>

Thanks, should be fixed shortly.

Skipper

headers

josef.pktd | 1 Oct 17:59

Re: Statsmodels documentation buglet

On Thu, Oct 1, 2009 at 11:01 AM, Skipper Seabold <jsseabold <at> gmail.com> wrote:
> On Thu, Oct 1, 2009 at 4:28 AM, Scott Sinclair
> <scott.sinclair.za <at> gmail.com> wrote:
>> Hi,
>>
>> I'm having a look at the statsmodels scikit and have noticed a minor
>> bug in the docs at:
>>
>> http://statsmodels.sourceforge.net/generated/scikits.statsmodels.regression.OLS.html
>>
>> The definition Y = [1,3,4,5,2,3,4], in the example contains a trailing
>> comma that causes the example to fail when used with IPython's cpaste
>> mode.
>>
>
> Thanks, should be fixed shortly.

I did it already in trunk, but got stuck with some examples. I moved
the tutorial files from the sandbox to the examples folder and will
push to launchpad soon. WLS with one of the regressors squared as
weights still looks "weird" (example_wls.py).

We haven't run the doctests in a while. However the scripts in the
example folder are always checked whether they run (not whether they
produce correct results).

Scott, any comments about the usage of statsmodels are very welcome.
(And I'm still waiting for missing bug reports.)

Josef

>
> Skipper
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev <at> scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>

headers

Prabhu Ramachandran | 1 Oct 22:48

[ANN] SciPy India conference in Dec. 2009

Greetings,

The first "Scientific Computing with Python" conference in India
(http://scipy.in) will be held from December 12th to 17th, 2009 at the
Technopark in Trivandrum, Kerala, India (http://www.technopark.org/).

The theme of the conference will be "Scientific Python in Action" with
respect to application and teaching.  We are pleased to have Travis
Oliphant, the creator and lead developer of numpy
(http://numpy.scipy.org) as the keynote speaker.

Here is a rough schedule of the conference:

     Sat.    Dec. 12  (conference)
     Sun.    Dec. 13  (conference)
     Mon.    Dec. 14  (tutorials)
     Tues.   Dec. 15  (tutorials)
     Wed.    Dec. 16  (sprint)
     Thu.    Dec. 17  (sprint)

The tutorial sessions will have two tracks, one specifically for
teachers and one for the general public.

There are no registration fees.

Please register at:

         http://scipy.in

The call for papers will be announced soon.

This conference is organized by the FOSSEE project (http://fossee.in)
funded by the Ministry of Human Resources and Development's National
Mission on Education (NME) through Information and Communication
Technology (ICT) jointly with SPACE-Kerala
(http://www.space-kerala.org).

Regards,
Prabhu Ramachandran and Jarrod Millman

headers

Scott Sinclair | 2 Oct 08:04

Re: Statsmodels documentation buglet

>2009/10/1  <josef.pktd <at> gmail.com>:
> Scott, any comments about the usage of statsmodels are very welcome.
> (And I'm still waiting for missing bug reports.)

Just poking around and finding it pretty useable so far. Happy to
report any bugs if I find them.

Cheers,
Scott

headers

Arkapravo Bhaumik | 2 Oct 14:24

Volunteer for Scipy Project

Dear Sir/Ma'am

I was in e-mail contact with one of your colleagues , Travis ; I am very interested in contributing to scipy project as a volunteer. I believe that I can contribute in

(1) Documentation
(2) Suggesting and developing newer functionality
(3) Study similar software as Matlab, Mathematica, Maple etc trying to look for inspiration for possible improvements in scipy

Some of my dabbling python are discussed in my blogspot, http://programming-unlimited.blogspot.com/ and I am eager to be a part of scipy : a revolution in the making.

I look forward to your reply.

Kind regards

Arkapravo

_______________________________________________
Scipy-dev mailing list
Scipy-dev <at> scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-dev

headers

Arkapravo Bhaumik | 2 Oct 14:32

Volunteer for Scipy Project

Dear Sir/Ma'am

I was in e-mail contact with one of your colleagues , Travis ; I am very interested in contributing to scipy project as a volunteer. I believe that I can contribute in

(1) Documentation
(2) Suggesting and developing newer functionality
(3) Study similar software as Matlab, Mathematica, Maple etc trying to look for inspiration for possible improvements in scipy

Some of my dabbling python are discussed in my blogspot, http://programming-unlimited.blogspot.com/ and I am eager to be a part of scipy : a revolution in the making.

I look forward to your reply.

Kind regards

Arkapravo

_______________________________________________
Scipy-dev mailing list
Scipy-dev <at> scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-dev

89	February 2012
196	January 2012
65	December 2011
44	November 2011
85	October 2011
122	September 2011
95	August 2011
33	July 2011
67	June 2011
82	May 2011
61	April 2011
62	March 2011
128	February 2011
99	January 2011
52	December 2010
123	November 2010
62	October 2010
92	September 2010
85	August 2010
335	July 2010
613	June 2010
207	May 2010
194	April 2010
101	March 2010
202	February 2010
113	January 2010
201	December 2009
178	November 2009
301	October 2009
166	September 2009
328	August 2009
162	July 2009
215	June 2009
183	May 2009
177	April 2009
251	March 2009
523	February 2009
227	January 2009
306	December 2008
417	November 2008
174	October 2008
116	September 2008
158	August 2008
130	July 2008
213	June 2008
160	May 2008
333	April 2008
170	March 2008
145	February 2008
200	January 2008
152	December 2007
42	November 2007
116	October 2007
124	September 2007
126	August 2007
192	July 2007
171	June 2007
180	May 2007
197	April 2007
130	March 2007
66	February 2007
116	January 2007
181	December 2006
51	November 2006
151	October 2006
64	September 2006
45	August 2006
71	July 2006
85	June 2006
45	May 2006
194	April 2006
230	March 2006
159	February 2006
606	January 2006
358	December 2005
477	November 2005
536	October 2005
118	September 2005
51	August 2005
45	July 2005
38	June 2005
51	May 2005
26	April 2005
52	March 2005
55	February 2005
66	January 2005
47	December 2004
254	November 2004
143	October 2004
94	September 2004
21	August 2004
12	July 2004
20	June 2004
28	May 2004
50	April 2004
43	March 2004
9	February 2004
16	January 2004
22	December 2003
42	November 2003
77	October 2003
46	September 2003
22	August 2003
14	July 2003
10	June 2003
48	May 2003
23	April 2003
99	March 2003
39	February 2003
34	January 2003
56	December 2002
64	November 2002
89	October 2002
95	September 2002
76	August 2002
16	July 2002
24	June 2002
31	May 2002
1	January 2000

Mon	Tue	Wed	Thu	Fri	Sat	Sun

			1	2	3	4
5	6	7	8	9	10	11
12	13	14	15	16	17	18
19	20	21	22	23	24	25
26	27	28	29	30	31