Understanding the confidence band from a polynomial regression

Question

I'm trying to understand the result I see in my graph below. Usually, I tend to use Excel and get a linear-regression line but in the case below I'm using R and I get a polynomial regression with the command:

ggplot(visual1, aes(ISSUE_DATE,COUNTED)) + geom_point() + geom_smooth()

So my questions boil down to this:

1. What is the gray area (arrow #1) around the blue regression line? Is this the standard deviation of the polynomial regression?

2. Can I say that the whatever is outside the gray area (arrow #2) is an 'outlier' and whatever falls inside the gray area (arrow #3) is within the standard deviation?

Answer 1

The gray band is a confidence band for the regression line. I'm not familiar enough with ggplot2 to know for sure whether it is a 1 SE confidence band or a 95% confidence band, but I believe it is the former (Edit: evidently it is a 95% CI). A confidence band provides a representation of the uncertainty about your regression line. In a sense, you could think that the true regression line is as high as the top of that band, as low as the bottom, or wiggling differently within the band. (Note that this explanation is intended to be intuitive, and is not technically correct, but the fully correct explanation is hard for most people to follow.)

You should use the confidence band to help you understand / think about the regression line. You should not use it to think about the raw data points. Remember that the regression line represents the mean of $Y$ at each point in $X$ (if you need to understand this more fully, it may help you to read my answer here: What is the intuition behind conditional Gaussian distributions?). On the other hand, you certainly do not expect every observed data point to be equal to the conditional mean. In other words, you should not use the confidence band to assess whether a data point is an outlier.

---

(Edit: this note is peripheral to the main question, but seeks to clarify a point for the OP.)

A polynomial regression is not a non-linear regression, even though what you get doesn't look like a straight line. The term 'linear' has a very specific meaning in a mathematical context, specifically, that the parameters you are estimating--the betas--are all coefficients. A polynomial regression just means that your covariates are $X$, $X^2$, $X^3$, etc., that is, they have a non-linear relation to each other, but your betas are still coefficients, thus it is still a linear model. If your betas were, say, exponents, then you would have a non-linear model.

In sum, whether or not a line looks straight has nothing to do with whether or not a model is linear. When you fit a polynomial model (say with $X$ and $X^2$), the model doesn't 'know' that, e.g., $X_2$ is actually just the square of $X_1$. It 'thinks' these are just two variables (although it may recognize that there is some multicollinearity). Thus, in truth it is fitting a (straight / flat) regression plane in a three dimensional space rather than a (curved) regression line in a two dimensional space. This is not useful for us to think about, and in fact, extremely difficult to see since $X^2$ is a perfect function of $X$. As a result, we don't bother thinking of it in this way and our plots are really two dimensional projections onto the $(X,\ Y)$ plane. Nonetheless, in the appropriate space, the line is actually 'straight' in some sense.

From a mathematical perspective, a model is linear if the parameters you are trying to estimate are coefficients. To clarify further, consider the comparison between the standard (OLS) linear regression model, and a simple logistic regression model presented in two different forms:
$$ Y = \beta_0 + \beta_1X + \varepsilon $$ $$ \ln\left(\frac{\pi(Y)}{1 - \pi(Y)}\right) = \beta_0 + \beta_1X $$ $$ \pi(Y) = \frac{\exp(\beta_0 + \beta_1X)}{1 + \exp(\beta_0 + \beta_1X)} $$ The top model is OLS regression, and the bottom two are logistic regression, albeit presented in different ways. In all three cases, when you fit the model, you are estimating the $\beta$s. The top two models are linear, because all of the $\beta$s are coefficients, but the bottom model is non-linear (in this form) because the $\beta$s are exponents. (This may seem quite strange, but logistic regression is an instance of the generalized linear model, because it can be rewritten as a linear model. For more information about that, it may help to read my answer here: Difference between logit and probit models.)

Answer 2

To add to the already existing answers, the band represents a confidence interval of the mean, but from your question you clearly are looking for a prediction interval. Prediction intervals are a range that if you drew one new point that point would theoretically be contained in the range X% of the time (where you can set the level of X).

library(ggplot2)
set.seed(5)
x <- rnorm(100)
y <- 0.5*x + rt(100,1)
MyD <- data.frame(cbind(x,y))

We can generate the same type of plot you've shown in your initial question with a confidence interval around the mean of the smoothed loess regression line (the default is a 95% confidence interval).

ConfiMean <- ggplot(data = MyD, aes(x,y)) + geom_point() + geom_smooth()
ConfiMean

For a quick and dirty example of prediction intervals, here I generate a prediction interval using linear regression with smoothing splines (so it is not necessarily a straight line). With the sample data it does pretty well, for the 100 points only 4 are outside the range (and I specified a 90% interval on the predict function).

#Now getting prediction intervals from lm using smoothing splines
library(splines)
MyMod <- lm(y ~ ns(x,4), MyD)
MyPreds <- data.frame(predict(MyMod, interval="predict", level = 0.90))
PredInt <- ggplot(data = MyD, aes(x,y)) + geom_point() + 
           geom_ribbon(data=MyPreds, aes(x=x,ymin=lwr, ymax=upr), alpha=0.5)
PredInt

(note: actual confidence intervals are smoother, as there was a code typo in the original answer)

Now a few more notes. I agree with Ladislav that you should consider time series forecasting methods since you have a regular series since sometime in 2007 and it is clear from your plot if you look hard there is seasonality (connecting the points would make it much clearer). For this I would suggest checking out the forecast.stl function in the forecast package where you can choose a seasonal window and it provides a robust decomposition of the seasonality and trend using Loess. I mention robust methods because your data have a few noticeable spikes.

More generally for non-time series data I would consider other robust methods if you have data with occasional outliers. I do not know how to generate prediction intervals using Loess directly, but you may consider quantile regression (depending on how extreme the prediction intervals need to be). Otherwise if you just want to fit to be potentially non-linear you can consider splines to allow the function to vary over x.

Answer 3

Well, the blue line is a smooth local regression. You may control the wiggliness of the line by the span parameter (from 0 to 1). But your example is a "time-series" so try to look for some more proper methods of analysis than only fit a smooth curve (which should serve only to reveal possible trend).

According to documentation to ggplot2(and book in comment below): stat_smooth is a confidence interval of the smooth shown in grey. If you want to turn the confidence interval off, use se = FALSE.