Building a model on past data for future predictions

_______________________________________________
Wekalist mailing list
Wekalist <at> list.scms.waikato.ac.nz
https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist

Re: Building a model on past data for future predictions

Hello,

you may find useful the paper "Automatic Categorization of Email into 
Folders" by Bekkerman, McCallum and Huang. They present your kind of 
evaluation 2), which they call "Time-based split evaluation". The first 
thing to do to improve you accuracy is previously delete from test set 
those instances tagged with labels not existing in your training set; 
that is, you dont want to discover new classes but classifiy documents 
using existing classes.

cheers,
pablo.

Carmelo Saffioti escribió:
> Hi everybody,
> I'm working on a classification task for building a model to make 
> predictions on future, training it on past data.
>  
> 1) If I evaluate it with cross-validation its accuracy is about 68%
>  
> 2) If I evaluate it partitioning the data set into training set (with 
> data from Month 1 to Month N-1) and test set (with data of Month N), its 
> accuracy is about 18%
>  
> Therefore predictions are too few accurate, according to the (2) 
> experiment. Have you got any idea about how can I do to achive better 
> results?
>  
>  
> Thank you in advance for your precious help
> Cheers
> Carmelo
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Wekalist mailing list
> Wekalist <at> list.scms.waikato.ac.nz
> https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist

_______________________________________________
Wekalist mailing list
Wekalist <at> list.scms.waikato.ac.nz
https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist

Re: Building a model on past data for future predictions

Lainaus "Carmelo Saffioti" <csaffi <at> libero.it>:

> Hi everybody,
> I'm working on a classification task for building a model to make  
> predictions on future, training it on past data.
>
> 1) If I evaluate it with cross-validation its accuracy is about 68%
>
> 2) If I evaluate it partitioning the data set into training set  
> (with data from Month 1 to Month N-1) and test set (with data of  
> Month N), its accuracy is about 18%

well it seems you have about the same amount of training instances for both
so the only explanation is in the way weka is implementing  
cross-validation, so-called stratified evaluation where each  
train/test fold pair for cv has mutually exactly the same class  
distribution and classifiers thereby are trained to expect that from  
test instances

>
> Therefore predictions are too few accurate, according to the (2)  
> experiment. Have you got any idea about how can I do to achive  
> better results?

hmm do you want better or more realistic results?

that level of drop is serious so there might be something else at  
fault too, hard to say

>
>
> Thank you in advance for your precious help
> Cheers
> Carmelo

--

-- 

kind regards,
Harri M.T. Saarikoski
M.A, PhD graduate student
Helsinki University
Finland

_______________________________________________
Wekalist mailing list
Wekalist <at> list.scms.waikato.ac.nz
https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist

Re: Building a model on past data for future predictions

> Hi everybody,
> I'm working on a classification task for building a model to make
> predictions on future, training it on past data.
>
> 1) If I evaluate it with cross-validation its accuracy is about 68%
>

If this is time series data, you cannot use cross-validation.
The proper procedure is training with data up to a certain
date, and then predict for data after that date.

Cross-validation is wrong for two reasons:
- it would use data from the future to predict past events,
which you cannot do in a real application
- time series often change little (both in input and output)
from one point to the next, so e.g. nearest neighbor in
cross-validation has a good chance to produce very high
results, which again would be meaningless in practise,
unless you only always want to predict one step ahead.

One more thing: usually the more into the future you try
to predict, the larger your errors get.

Bernhard

how can i get more info about the parameters of the MLP classifier ?

_______________________________________________
Wekalist mailing list
Wekalist <at> list.scms.waikato.ac.nz
https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist

Interperting the output of MultilayerPerceptron

_______________________________________________
Wekalist mailing list
Wekalist <at> list.scms.waikato.ac.nz
https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist

Re: Building a model on past data for future predictions

Thank you Pablo for your interesting answer.

What I need is exactly a time-based split for train/test data sets. As you 
suggested, the first thing I'll do is to delete from the test set elements 
with not existing labels in my training set.

However I've got a question about this: in the real-world application I'll 
have data with not existing labels in my training set, so evaluating the 
model in this way doesn't seem to be realistic... What do you think about?

Thank you again
Cheers
Carmelo

----- Original Message ----- 
From: "Pablo Bermejo" <pbermejo <at> dsi.uclm.es>
To: "Weka machine learning workbench list." 
<wekalist <at> list.scms.waikato.ac.nz>
Sent: Sunday, February 01, 2009 12:45 PM
Subject: Re: [Wekalist] Building a model on past data for future predictions

Hello,

you may find useful the paper "Automatic Categorization of Email into
Folders" by Bekkerman, McCallum and Huang. They present your kind of
evaluation 2), which they call "Time-based split evaluation". The first
thing to do to improve you accuracy is previously delete from test set
those instances tagged with labels not existing in your training set;
that is, you dont want to discover new classes but classifiy documents
using existing classes.

cheers,
pablo.

Carmelo Saffioti escribió:
> Hi everybody,
> I'm working on a classification task for building a model to make
> predictions on future, training it on past data.
>
> 1) If I evaluate it with cross-validation its accuracy is about 68%
>
> 2) If I evaluate it partitioning the data set into training set (with
> data from Month 1 to Month N-1) and test set (with data of Month N), its
> accuracy is about 18%
>
> Therefore predictions are too few accurate, according to the (2)
> experiment. Have you got any idea about how can I do to achive better
> results?
>
>
> Thank you in advance for your precious help
> Cheers
> Carmelo
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Wekalist mailing list
> Wekalist <at> list.scms.waikato.ac.nz
> https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist

--------------------------------------------------------------------------------

_______________________________________________
Wekalist mailing list
Wekalist <at> list.scms.waikato.ac.nz
https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist

Re: Building a model on past data for future predictions

Hello Carmelo,

I thought you were trying to perform supervised classification: training 
your classifier with pre-labeled instances and then test by assigning 
the most probable class value to your test instances. If that is the 
case, you could just use in your training data method 
Instances.deleteWithMissingClass().

If what you are trying is to feed your classifier with instances with 
both existing and missing classes, thats beyond my scope, sorry.

Besides, since you have several time splits, you could also label by 
hand your first time-split and then test your second time-split. Then, 
when you train with both first and second time-split, you might use for 
your second split labels predicted previously while testing. Of course, 
this makes you keep with you prediction errors along your evaluation.

I also recommend you to read some preprocessing steps described in the 
paper by Bekkerman i suggested in the previous mail.

Hope this helps.

Regards,
Pablo.

Carmelo Saffioti escribió:
> Thank you Pablo for your interesting answer.
> 
> What I need is exactly a time-based split for train/test data sets. As 
> you suggested, the first thing I'll do is to delete from the test set 
> elements with not existing labels in my training set.
> 
> However I've got a question about this: in the real-world application 
> I'll have data with not existing labels in my training set, so 
> evaluating the model in this way doesn't seem to be realistic... What do 
> you think about?
> 
> 
> Thank you again
> Cheers
> Carmelo
> 
> 
> ----- Original Message ----- From: "Pablo Bermejo" <pbermejo <at> dsi.uclm.es>
> To: "Weka machine learning workbench list." 
> <wekalist <at> list.scms.waikato.ac.nz>
> Sent: Sunday, February 01, 2009 12:45 PM
> Subject: Re: [Wekalist] Building a model on past data for future 
> predictions
> 
> 
> Hello,
> 
> you may find useful the paper "Automatic Categorization of Email into
> Folders" by Bekkerman, McCallum and Huang. They present your kind of
> evaluation 2), which they call "Time-based split evaluation". The first
> thing to do to improve you accuracy is previously delete from test set
> those instances tagged with labels not existing in your training set;
> that is, you dont want to discover new classes but classifiy documents
> using existing classes.
> 
> cheers,
> pablo.
> 
> Carmelo Saffioti escribió:
>> Hi everybody,
>> I'm working on a classification task for building a model to make
>> predictions on future, training it on past data.
>>
>> 1) If I evaluate it with cross-validation its accuracy is about 68%
>>
>> 2) If I evaluate it partitioning the data set into training set (with
>> data from Month 1 to Month N-1) and test set (with data of Month N), its
>> accuracy is about 18%
>>
>> Therefore predictions are too few accurate, according to the (2)
>> experiment. Have you got any idea about how can I do to achive better
>> results?
>>
>>
>> Thank you in advance for your precious help
>> Cheers
>> Carmelo
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> Wekalist mailing list
>> Wekalist <at> list.scms.waikato.ac.nz
>> https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist
> 
> 
> 
> -------------------------------------------------------------------------------- 
> 
> 
> 
> _______________________________________________
> Wekalist mailing list
> Wekalist <at> list.scms.waikato.ac.nz
> https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist
> 
> 
> _______________________________________________
> Wekalist mailing list
> Wekalist <at> list.scms.waikato.ac.nz
> https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist
> 

_______________________________________________
Wekalist mailing list
Wekalist <at> list.scms.waikato.ac.nz
https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist

Re: Building a model on past data for future predictions

Thank you Bernhard.

I'm understanding that for a more realistic model evaluation I cannot use 
cross-validation, but time-based train/test splitting...
Anybody has ever worked with this type of problem? What can you suggest me 
for developing better predictive models and achiving better results?

----- Original Message ----- 
From: "Bernhard Pfahringer" <bernhard.pfahringer <at> gmail.com>
To: "Weka machine learning workbench list." 
<wekalist <at> list.scms.waikato.ac.nz>
Sent: Sunday, February 01, 2009 8:23 PM
Subject: Re: [Wekalist] Building a model on past data for future predictions

>> Hi everybody,
>> I'm working on a classification task for building a model to make
>> predictions on future, training it on past data.
>>
>> 1) If I evaluate it with cross-validation its accuracy is about 68%
>>
>
> If this is time series data, you cannot use cross-validation.
> The proper procedure is training with data up to a certain
> date, and then predict for data after that date.
>
> Cross-validation is wrong for two reasons:
> - it would use data from the future to predict past events,
> which you cannot do in a real application
> - time series often change little (both in input and output)
> from one point to the next, so e.g. nearest neighbor in
> cross-validation has a good chance to produce very high
> results, which again would be meaningless in practise,
> unless you only always want to predict one step ahead.
>
> One more thing: usually the more into the future you try
> to predict, the larger your errors get.
>
> Bernhard
>
> _______________________________________________
> Wekalist mailing list
> Wekalist <at> list.scms.waikato.ac.nz
> https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist 

stacking classifiers with different attribute sets

Hi,
I would like to use the stacking classifier provided in Weka. 
I would like to choose different attribute sets (from the same training set)
for the different classifiers chosen (both at level 0 and level 1) .  Is it
possible to do that using WEKA explorer ?
Thanks in advance!

Regards,
Sharat

--

-- 
View this message in context: http://www.nabble.com/stacking-classifiers-with-different-attribute-sets-tp21789997p21789997.html
Sent from the WEKA mailing list archive at Nabble.com.

_______________________________________________
Wekalist mailing list
Wekalist <at> list.scms.waikato.ac.nz
https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist