Re: Some Confusion with Evaluation framework in Weka API
inception <iinception <at> aol.com>
2011-01-02 04:20:34 GMT
I got the concepts.
Bernhard Pfahringer-2 wrote:
>> Thanks Bernhard for quick reply. I think I am overlooking some basic
>> Please explain what is wrong with my assumption : Given same confusion
>> matrix M1 and M2 in both case with same data and algorithm but with
>> different random value(8, 9 as previous) why am I getting two different
>> value for AUC and mean absolute error.
> AUC is about ranking, using the probs to sort your examples.
> Accuracy (and the confusion matrix) depend on a specific threshold.
> So if your probabilities "sort" the examples differently in different
> runs on either side of the threshold, you can get the exact same
> accuracy, but different AUC values.
>> I was curious because with random seed 1 i get AUC of 0.937 and 3 I get
>> 0.874. Confusion matrix is same in both case.Which value should I trust
> I suppose you are using a rather "unstable" algorithm, and/or a small
> of examples, and/or have a high number of class values. What you
> is that cross-validation has some variance as well. If the variance is as
> as it seems in your case, I'd repeat at least ten times with a new seed
> time and take the average. BTW, this is the default for the Experimenter:
> 10x10fold cross-validation, to get more robust estimates.
> hth, Bernhard
> Bernhard Pfahringer, Dept. of Computer Science, University of Waikato
> http://www.cs.waikato.ac.nz/~bernhard +64 7 838 4041
> Wekalist mailing list
> Send posts to: Wekalist <at> list.scms.waikato.ac.nz
> List info and subscription status:
> List etiquette:
View this message in context: http://old.nabble.com/Some-Confusion-with-Evaluation-framework-in-Weka-API-tp30569132p30570398.html
Sent from the WEKA mailing list archive at Nabble.com.
Wekalist mailing list
Send posts to: Wekalist <at> list.scms.waikato.ac.nz
List info and subscription status: https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html