6

I am doing auto-regress by usual linear regression package.
e.g. $y_t=φx+ε_t$ with $x =y_{t-1}$

My reason is that,
Auto-regression does assumes iid errors, same for linear regression. Linear Regression doesn't have assumption on independent variables. What's different is merely that the independent variables is replaced with lagged dependent variable y.

But I see AR always is fitted by specific methods, I am afraid I'm doing wrong.

WeiChing 林煒清
  • 665
  • 6
  • 17
  • 2
    Actually, Hamilton ["Time Series Analysis"](http://press.princeton.edu/titles/5386.html) says explicitly on p. 123 that OLS is the common method of estimation of AR processes. – Richard Hardy Nov 18 '15 at 10:37

2 Answers2

6

To answer the title question, fitting an AR($p$) model using OLS will yield biased estimates. The reason is that for unbiasedness, the model errors should be uncorrelated with past, current and future values of regressors, which is not the case in autoregressive models. For example, in case of AR(1)

$$ y_t=\varphi y_{t-1}+\varepsilon_t $$

(assuming zero mean for simplicity). Lag this by 1 to obtain

$$ y_{t-1}=\varphi y_{t-2}+\varepsilon_{t-1}. $$

Note that $\varepsilon_{t-1}$ enters the model of $y_{t-1}$; hence, the regressor $y_{t-1}$ will be correlated with lagged error $\varepsilon_{t-1}$. The argument is given (without proof) e.g. in this lecture note, p. 5-6.

On a positive note, OLS gives consistent estimators for an autoregressive model (see the same lecture note, p. 4-5)

Also, in my experience OLS is quite popular for fitting AR models, and is pretty standard for fitting multivariate AR, i.e. VAR, models.

Richard Hardy
  • 54,375
  • 10
  • 95
  • 219
1

If you fit by regressing $\mathbf{y}_{p+1:n}=(y_{p+1},...,y_n)^\top$ on its lags $X=[\mathbf{y}_{p:n-1},\mathbf{y}_{p-1:n-2},...,\mathbf{y}_{1:n-p}]$ the lags of that on you're going to be conditioning on the first $p$ values (for an AR(p)). If you fit by say maximum likelihood you're able to incorporate the likelihood for the first $p$ values.

If $n$ is not very large relative to $p$ is can sometimes make a substantial difference.

Glen_b
  • 257,508
  • 32
  • 553
  • 939
  • I thought that estimating by OLS would amount to effectively cutting the sample by $p$ observations so that lagged regressors could be obtained -- instead of appending the data with some arbitrary values for the negative lags. As such there would be no conditioning on the first $p$ values. – Richard Hardy Nov 17 '15 at 18:34
  • @RIchardHardy you just *exactly* described why it is conditioning on the first $p$ values. They're removed from $y$, but they're all still in $X$ in the regression (and in regression, $y$ is conditioned on $X$). ergo the likelihood you maximize is one conditioning on the first $p$ y-values. (You can also show it algebraically by decomposing the likelihood, it's quite straightforward. The likelihood for the AR can be split via the prediction error decomposition into two components, that for $y_{p+1},...,y_n$ and $y_1,...,y_p$, ...ctd – Glen_b Nov 17 '15 at 21:18
  • ctd... if you write the likelihood conditional on $y_1,...,y_p$ the second term drops out and you're left with the regression model). – Glen_b Nov 17 '15 at 21:22
  • Thanks for the explanation! Apparently, my confusion is in terminology. Also, I did not make the connection between OLS and maximum likelihood (I only thought in terms of OLS), which apparently explains conditioning. It still gives me some thought to digest. – Richard Hardy Nov 17 '15 at 21:22
  • @RIchard I was just googling for something to link you to for the algebraic version. [This](http://faculty.chicagobooth.edu/ruey.tsay/teaching/uts/lec8-08.pdf) document in section 2 shows it for the AR(1) case; the AR(p) case works the same way. – Glen_b Nov 17 '15 at 21:25
  • Thank you for the material. Now I see what I was missing. In an AR(1) example without intercept, I did not consider the contribution of $y_1$ to the likelihood -- I did not realize that it was informative. Intuitively, I thought that the process we started observing at time $t=1$ did not actually launch as a zero-mean realization of a random variable independent of the past; I thought of an infinite past (t=0,-1,-2, $\dotsc$) that we cannot observe. But I failed to realize that given the structure of the process, $y_1$ contributes with its unconditional density to the likelihood. Thanks again! – Richard Hardy Nov 18 '15 at 09:28