2

Extrapolation is in general "unreliable". (See "What is wrong with extrapolation?")

But it is also commonly said that extrapolation is "less reliable" than interpolation.

But why should we generally assume that the model is "more reliable" between two known data points than to the right of the right-most data point (or the left of the left-most data point)?

From empirical examples, I can see that indeed, interpolation is often "more reliable" than extrapolation. But is there a more formal, theoretical justification for why this assertion is, in general, true?

Or is it just a purely empirical observation that interpolation tends to be "better"?

Community
  • 1
  • `extrapolation “less reliable” than interpolation` that is true not in all contexts. It may be true when a specific number, more than one, of known points is demanded to infer the unknown point. In interpolation, you estimate unknown t2 value from t1 and t3 values, both known and both adjacent to t2. Consider now the comparable extrapolation: t1 is known, t3 in not known, extrapolation t2 from t1 can be seen as interpolation from t1 and (unknown) t3 onto t2. Of course, it is less "reliable". – ttnphns Jun 30 '16 at 07:52
  • (cont.) Alternatively, the extrapolation can be taken as extrapolation from t1 and t0 (both known) onto t2. But here t0 is "far away", so less "reliable" once again. – ttnphns Jun 30 '16 at 07:52

3 Answers3

2

It's partly empirical, partly theoretical. Theoretically, we are trying to fit a model to the observed data, hence our choice of function is determined by the data points we have, not by what we don't have. Thus, our model is selected by the observed data, and we have no guarantees that we'd choose a similar model if we had more extreme observations.

Related to this is the fact that we can only cross-validate with observed data, so we can only check it's accuracy against observed points. If a model has very high cross-validation accuracy, then all we know is that it is probably a reliable estimator for points within the boundary of the observed data. Outside this region, we have no data and no estimate of how we would do.

That's the theoretical part. Empirically, we usually assume continuity and smoothness of a response surface, since many processes behave that way. Hence, we assume that the true model comes from a class of well-behaved functions, not crazy, jagged-edged monsters. This is a key regularization step in model fitting. With that assumption, we get the idea that points close in inputs space should be close in their function values, with convergence to $0$ difference as the points approach one another.

  • 3
    +1 for smoothness vs. crazy, jagged-edged monsters. Not so sure about the convex hull statement. Imagine we're fitting a function and our training inputs lies within a crescent moon shape. The function may behave differently outside the region enclosing the training points. But, this outside-the-data region includes points within the convex hull and outside the moon shape. Without further assumptions, there are no guarantees on generalization performance in this region. – user20160 Jun 30 '16 at 06:04
  • 1
    @user20160 You're absolutely correct. Most of the datasets I deal with result in "cloudlike" scatter in input space so I was trying to loosely indicate where the function is likely to hold. I'll remove. –  Jun 30 '16 at 11:31
2

It is a theoretical result, at least for linear regression. Indeed, if one computes the so-called ''prediction error'' (see this link, slide 11), one can easily see that the further the independent variable $x$ is away from the sample average $\bar{x}$ (and for extrapolation one may be far away), the larger the prediction error. In the link that I referred to one can also see that in a graphical way.

0

Check Goedel's Theorem, for a more rigorous explanation of why extrapolation is less reliable than interpolation. Actually, empiricism merely confirms the rigor of Goedel's theorem. Intuition however is the tool we normally use to discredit the more everyday argument against what we then call generalizing, (or more formally, inductive logic) in everyday parlance. (Many mental illnesses, especially paranoia, result from this intuitive leap of logic from the specific to the general.)

Brerlou L. King
  • 1