1

I have data that is quite heteroscedastic, and therefore decided to try fitting a GLS model in python with the statsmodels package in python.

The data has two continuous feature variables with skewed distributions with a continuous response variable. The data is NOT time series. I did not know how to specify "sigma" in the model, so I just left it as "None".

The model performed well, with an r2 value of ~ 98%! However I am completly clueless as how else to evaluate the model, particularly in comparison to my earlier OLS and polynomial regression models.

I tried to compute the Mean Average Error, by running a batch of predictions and finding their mean difference with the actuals. But this approach yielded a pretty large deviation.

How else can I evaluate the model? Am I running predictions correctly?

I have never worked with GLS or WLS models before, and the math goes a bit over my head. Any tips on how to help? Thanks!

Tuomas Talvitie
  • 29
  • 5
  • $R^2$ of 98% is often too good to be true.... – Nick Cox Mar 29 '20 at 12:09
  • Mean Average Error has no obvious virtues if the fitting criterion is any flavour of Least Squares. There are few details here, but a key point that can be made is that GLS or WLS doesn't imply a different functional form, just different ways of handling errors. I think you need to cross-reference your previous thread all over again and explain that you can't post any data. https://stats.stackexchange.com/questions/455049/how-to-fix-heteroscedasticity-funnel-shape I fear that this is too broad if the answer sought is general or too vague if you want insights for your current project. – Nick Cox Mar 29 '20 at 12:16
  • @NickCox So would it be helpful to post my data? I am trying to avoid having to go into too much detail. Secondly, is Mean Average Error not particularly useful then? Why? My end goal is to use the model to make predictions real time. – Tuomas Talvitie Mar 30 '20 at 13:11
  • I've been urging all along that you post data. You are minimising a sum of squares criterion, so some kind of root mean square error seems far more pertinent. Otherwise you are judging baseball by the criteria of tennis. – Nick Cox Mar 30 '20 at 13:16
  • Okay, would you recommend I repost? Do you want to see a table of the data? Or data that's charted? What exactly would be helpful? – Tuomas Talvitie Mar 30 '20 at 13:19
  • On the evidence of your previous thread, charts don't help enough. I would like to see a listing of your data. If it's too much to post it all, a sample of ~100 observations would still be helpful. This is already your second thread. We don't need another. – Nick Cox Mar 30 '20 at 13:28
  • @NickCox my data is not publicly available. Would you prefer I DM you my data set ? – Tuomas Talvitie Mar 30 '20 at 13:30
  • Thanks for the offer, but absolutely not. Public or nothing. Sorry, but this is not a forum to arrange private consultation. In any case why trust me but not anybody else? We respect the limits on how you operate, but without publicly available data I can't see this as a viable thread myself. – Nick Cox Mar 30 '20 at 13:37
  • @NickCox oh I just suggested it because otherwise it would be a long post. But sure, I'll post my data on my next post. – Tuomas Talvitie Mar 30 '20 at 13:40
  • No offence taken or intended, but all of a sudden "not available" looks surprisingly flexible! – Nick Cox Mar 30 '20 at 13:44

0 Answers0