-1

According to these notes from Penn State on MLE and notes from Stanford CS229 on logistic regression, training data is assumed to be IID.

The Penn State notes say "...which implies by definition that the X are independent." and the Stanford notes state that "Assuming that the m training examples were generated independently, we can then write down the likelihood of the parameters as..." However, according to this Cross Validated answer the observations do not need to be IID.

Isn't X a random variable and in order to take the product for the joint probability, don't the observations need to be independent? Also is there a difference in saying that the labels (y's) are independent versus the observations (x's) are independent?

Could someone explain? What am I misunderstanding?

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
Soubriquet
  • 141
  • 1
  • 7
  • They may be referring to autocorrelation, where $y_i$ values could be dependent on previous values, like $y_{i-1}$, such that $y_i = f(y_{i-1}) + \alpha + \beta x_i + \epsilon$. – ERT Jul 06 '18 at 23:44
  • 1
    Look at the examples of the joint likelihood functions for dependent data in the answers to the linked CV post. They're not the product of identical likelihood functions for individual observations. – Scortchi - Reinstate Monica Jul 07 '18 at 07:21

1 Answers1

3

You are misunderstanding the documents linked to. They probably makes an iid assumption for their examples, so that they exclude other cases, they do not make an universal assumption. To use likelihood methods you need to be able to write down a joint density function (or joint probability mass function), and that is simple in the independent case. But it can be done in many cases for models with non-independent observations.

Some examples, time series model with serial correlation, an example Beginner learning resources : Pdf and likelihood function for non-Gaussian time series model. Spatial models, I do not find really good examples on site, but here is one: Maximum likelihood fitting of truncated, mixed, two population systems or mixed models, for instance: http://www2.stat.duke.edu/~sayan/Sta613/2017/lec/LMM.pdf

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467