0

Statistical models of time series tend to underestimate points above average and overestimate points below average, due to regression to the mean. If points above and below average are treated separately, the error associated with the two subsets is systematic, not random. Also, the intensity of the error seems proportional to deviation from the mean itself. I would thus assume it would be possible to compensate for it; however I have not seen this done in any of the commonly used algorithms. Why? Many thanks

Stefano
  • 1
  • 1
  • 2
    Research "regression to the mean:" that is a mathematical phenomenon underlying the tendency of almost any statistical procedure to underestimate maxima and overestimate minima. – whuber Apr 02 '21 at 18:33
  • 2
    Your model estimates the conditional expectation. The observations vary around that expectation; this is residual variance. The expectation will always be smaller than larger observations and larger than smaller ones. See https://stats.stackexchange.com/q/390210/1352. – Stephan Kolassa Apr 02 '21 at 20:04
  • If you consistently over predict or under predict the results than that you could address (expert judgement if nothing else). But from other comments here that is not what you mean. – user54285 Apr 02 '21 at 20:38
  • Thank you for your replies. It seems to me then that regression to the mean introduces a systematic error, always in the same direction (if we separate observations above and below average). The effect seems also proportional to the deviation from the average. If so, couldn't it be taken into consideration in a correction term? – Stefano Apr 03 '21 at 11:31
  • 2
    I may be misinterpreting your suggestion, but it seems to me like this would require you to predict which "class" (i.e. "above average" or "below average") the next data point is in to make your forecast, which is not typically easy to do reliably, so it's not clear that this would produce a better forecast. – Chris Haug Apr 05 '21 at 11:56
  • @Chris Haug - I was thinking of applying the correction in the training phase, where the two classes are easy to separate reliably, in order to obtain parameters that take into consideration the correction. Consequently, the forecast would do the same (?) – Stefano Apr 05 '21 at 13:44
  • Do you not have a "correction" for each class, requiring you to predict the class to know which one to apply? For example, say your model is just $Y_t = \mu + \varepsilon_t$, with $\varepsilon_t \sim_{iid} \mathcal{N}(0,1)$. Do you then compute $\mu_L$ and $\mu_H$ as the average of low/high observations separately? And then when you forecast, which do you use, $\mu_H$ or $\mu_L$? I think it would help if you could be more explicit about a simple application of your suggested method. – Chris Haug Apr 05 '21 at 13:55
  • @Chris Haug-As I'm a beginner, my question was whether something of this kind was already implemented and I missed it, rather than proposing a method. Thanks – Stefano Apr 05 '21 at 16:26

0 Answers0