4

I was reading about Maximum likelihood estimation from various sources on the internet and I noticed that MLE makes an assumption about the data known as IID but I didn't completely understand why is it necessary to make this assumption? Are there any other assumptions that MLE makes?

Akash Dubey
  • 425
  • 4
  • 15
  • 3
    It isn't necessary. MLE maximizes a wide array of likelihoods. Your sources are assuming iid because it is a reasonable modeling assumption in a particular setting. If your model assumes the data are iid, it is easy to write down the likelihood and then use MLE to estimate the unknown parameters. – Taylor Jan 08 '19 at 23:38
  • 2
    ... it's reasonable and it's **extremely convenient**, because it implies that the overall likelihood of the data set is the product of the likelihoods of the individual observations (independent), which all have the same form (identically distributed) $\to$ log-likelihood is the *sum* of individual LLs ... – Ben Bolker Jan 08 '19 at 23:40
  • 1
    @Taylor I didn't quite understand 'wide array of likelihoods'. Can you name a few? – Akash Dubey Jan 08 '19 at 23:40
  • 2
    Consider a time series model like an AR(1) process. Then the order of the data matter, and your likelihood is a product of conditional densities, not marginal densities. If you have three data points, you could write the likelihood as $p(x_3 \mid x_2) p(x_2 \mid x_1) p(x_1)$. MLE would have no problem with this, but we are not assuming the data are iid. – Taylor Jan 08 '19 at 23:44
  • @BenBolker can you please write an answer to this question explaining from a general and mathematical point of view? – Akash Dubey Jan 08 '19 at 23:44
  • 2
    Possible duplicate of [What does it mean by independently and identically distributed random variables?](https://stats.stackexchange.com/questions/253861/what-does-it-mean-by-independently-and-identically-distributed-random-variables) – Taylor Jan 08 '19 at 23:45
  • not now (I gave the partial answer in a comment because I don't have time to write a real/full answer at the moment) – Ben Bolker Jan 08 '19 at 23:49
  • @BenBolker Alright, Please trying answering this question whenever you have some spare time. – Akash Dubey Jan 08 '19 at 23:53

1 Answers1

2

Assuming independence is not necessary for maximum likelihood estimation (ML) (or other likelihood based methods). But if independence is a reasonable assumption, then it makes ML easy to implement, since the log likelihood is simply the sum of the individual log likelihoods. But there are lots of examples where ML is used without Independence: time series with ARMA (or ARIMA) models, spatial models including spatial dependence, mixed models where there are correlation between observations within some groups, others.

When is it reasonable to assume independence? is discussed already multiple times on this site, see for example Are "random sample" and "iid random variable" synonyms? or Independence of events in real-life data

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467