3

What is an example of a random variable that is not i.i.d? The usual ones (coin flips, rolling of a dice) are all i.i.d, so I am trying to understand what is an example of a random variable that is not i.i.d?

Victor
  • 5,925
  • 13
  • 43
  • 67
  • 1
    [Same question here](http://stackoverflow.com/questions/13058379/example-of-non-iid-data), although on stackoverflow. – COOLSerdash Dec 05 '14 at 17:06
  • 4
    IID is about independence, so you need at least two random variables to talk about independence or not! 'a' random variable is __always__ iid, so to say, emptily.... You should ask about a __sequence__ of random variables which are not iid... – kjetil b halvorsen Dec 05 '14 at 17:10

2 Answers2

8

A sequence of random variables (we need at least two!) can fail to be iid (independent and identically distributed) in three different ways:

  1. The random variables are not independent but they are identically distributed,
  2. The random variables are independent but are not identically distributed,
  3. The random variables are neither independent nor identically distributed.

For an example of 1., consider the "sampling without replacement" method described in this answer. Or consider an experiment in which we have a bag containing two coins with different probabilities $p_1$ and $p_2$ of turning up Heads. We choose one of the coins at random and toss the chosen coin $n$ times. Let $X_i$ be the indicator function of Heads on the $i$-th toss. Then, the law of total probability tells us that $$P\{X_i = 1\} = P\{X_i = 1\mid ~\text{coin #1}\}\times \frac 12 + P\{X_i = 1\mid ~\text{coin #2}\}\times \frac 12 = \frac{p_1+p_2}{2}.$$ Thus, the $X_i$'s are identically distributed Bernoulli random variables with parameter $\frac{p_1+p_2}{2}$. However, they are not independent random variables since the law of total probability gives us that $$P\{X_i=1, X_j=1\}=\frac{P\{X_i=1, X_j=1 \mid~\text{#1}\} + P\{X_i=1, X_j=1 \mid~\text{#2}\}}{2} = \frac{p_1^2+p_2^2}{2}$$ which does not equal $P\{X_i=1\}P\{X_j=1\} = \left(\frac{p_1+p_2}{2}\right)^2$.

For an example of 2., consider a similar experiment in which the the $i$-th (independent toss) is of a coin that has probability $p_i$ different from $p_1, p_2, \ldots, p_{i-1}$. (Let $p_i = e^{-i}$ if you need an explicit example). Then the $X_i$ are independent Bernoulli random variables (by assumption) but they are not identically distributed since they all have different parameters.

I will leave the exercise of coming up with an example of 3. to you.

Community
  • 1
Dilip Sarwate
  • 41,202
  • 4
  • 94
  • 200
2

A classic example would be a Markov chain. As a simple example - imagine this process to generate variables.

xi+1 = xi + 1 with 50% chance and xi - 1 with 50% chance

The probability distribution for the value of x at any given iteration of the process is different and dependent on the history of the process - therefore the variable output by the process (or the Markov Chain) are random, but not i.i.d.

J Richard Snape
  • 319
  • 3
  • 7
  • Thanks. So they are not independent, but still identically distributed, correct? – Victor Dec 05 '14 at 17:17
  • @Tim You express a misunderstanding. Markov chains are typically used only after sufficiently *thinning* them to make their outputs *approximately* iid. The presence of nonzero autocorrelation immediately implies the variables are not independent! – whuber Dec 05 '14 at 17:45
  • @whuber actually thinning does not help for reducing autocorrelation (see http://doingbayesiandataanalysis.blogspot.com/2011/11/thinning-to-reduce-autocorrelation.html) and the chains are iid by Markov property. – Tim Dec 05 '14 at 17:49
  • 2
    @Tim You must be using "iid" in an unconventional way. Are you referring perhaps to the *ensemble* distribution where each chain is viewed as a (vector-valued) random variable? It doesn't sound like that, unfortunately, and you have already pointed out the presence of autocorrelation among values within a chain, so those values *cannot possibly* be independent (unless you have some novel idea of "independence"). – whuber Dec 05 '14 at 17:53
  • Then to restate my comment: this is a misleading example because Markov Chains are commonly used for generating iid values. By definition, subsequent draws from Markov Chain are dependent on previous state but not on the sequence of events that preceded it. In practice some amount of autocorrelation is observed in this kind of chain but in the long run the values generated approach being iid. – Tim Dec 05 '14 at 18:07
  • 2
    I think the example is correct as stated: the process defined by this random walk is a Markov chain and hence the sequence $(x_t)_t$ is made of dependent random variables. This has nothing to do with the use of MCMC methods to generate from a given distribution. Note that the random walk example means that the $x_t$'s cannot be identically distributed since there is no stationary distribution. So the elements of that Markov chain are ninid variables. – Xi'an Dec 05 '14 at 18:21