10

I found that for a simple linear regression model, both OLS and maximum likelihood method (assuming Normal distribution) give the same output (parameter values). From this, can we say that OLS also make implicit assumption about the Normal distribution or vice-versa? I am not interested in why both produce same value but which one make less stringent assumption about the data?

Richard Hardy
  • 54,375
  • 10
  • 95
  • 219
Neeraj
  • 2,150
  • 18
  • 28

1 Answers1

8

OLS does not make a normality assumption for the model errors. OLS can be used under different distributional assumptions and the estimator will still make sense as the minimum variance linear unbiased estimator.

Maximum likelihood (ML) can also accommodate different distributions, but the distribution has to be chosen in advance. If the actual distribution appears to be different from the assumed distribution, ML estimator will no longer make sense as the estimator that maximizes the joint probability density of the data.

Thus we can say that in a particular application ML makes a more stringent assumption about the model errors than OLS does.

Richard Hardy
  • 54,375
  • 10
  • 95
  • 219
  • Thanks @RihchardHardy.....can you please explain what does it mean, "OLS can be used under different distributional assumptions" ? Because as per my knowledge OLS does not make any assumption about data, it only minimize sum of square of residuals. – Neeraj Dec 05 '15 at 10:00
  • You are right. That was intended to mean "Regardless of what distributional assumption, if any, you are making, OLS will be fine". – Richard Hardy Dec 05 '15 at 10:06
  • @thanks RichardHardy. It means that maximum likelihood is as good as assumption of distribution. But does it mean that for linear regression model OLS is always better than ML ? – Neeraj Dec 05 '15 at 10:49
  • If data is not normally distributed than neither OLS nor ML(assuming Normal distribution) is good because both produce the same output. So can we say that OLS is as good as data is normally distributed ? – Neeraj Dec 05 '15 at 10:50
  • 1
    Which is better depends on your objective function (or loss function). **1.** If you want to maximize the joint probability density of errors, maximum likelihood is your choice (but you have to check how well the model residuals match the distributional assumption). Also, maximum likelihood may still work even if the error distribution is not matched; the keyword here is quasi maximum likelihood. **2.** If you want to minimize the sum of squared residuals, go for OLS. – Richard Hardy Dec 05 '15 at 10:51
  • But both give the same output. How does it matter what is being used. – Neeraj Dec 05 '15 at 10:52
  • how ML work if distribution is not matched ? – Neeraj Dec 05 '15 at 10:53
  • OLS is the same as ML with normality assumption -- but not the same as ML with some other distributional assumption. When it comes to quasi maximum likelihood, this is an entire topic and merits a separate question (which might have been asked already). – Richard Hardy Dec 05 '15 at 10:54
  • this is exactly what I am asking. If both are same then can we say that OLS unknowingly use assumption of normal distribtution ? – Neeraj Dec 05 '15 at 10:58
  • By the way, I have asked a question about QMLE [here](http://stats.stackexchange.com/questions/185154/idea-and-intuition-behind-quasi-maximum-likelihood-estimation) since I did not find it asked before. – Richard Hardy Dec 05 '15 at 11:07
  • 1
    I think it is a question of interpretation. OLS and MLE have different goals. However, incidentally OLS and MLE are the same under normal distribution. Can this be interpreted as OLS unknowingly using the nomality assumption? Perhaps, but I would not put it that way. OLS is not based on it nor it has a goal of maximizing likelihood. I would call it a coincidence. In general, you may ask a new question rather than pose it in the comments. Comments should ideally be strongly related to the OP or the answer, while broader discussion may merit new questions and new answers. – Richard Hardy Dec 05 '15 at 11:09
  • does it mean OLS has all limitation which ML has with Normal distribution? – Neeraj Dec 05 '15 at 11:10
  • thanks for asking question on QMLE. Here we will definitely get answer. – Neeraj Dec 05 '15 at 11:14