20

A time-honored reminder in statistics is "uncorrelatedness does not imply independence". Usually this reminder is supplemented with the psychologically soothing (and scientifically correct) statement "when, nevertheless the two variables are jointly normally distributed, then uncorrelatedness does imply independence".

I can increase the count of happy exceptions from one to two: when two variables are Bernoulli-distributed , then again, uncorrelatedness implies independence. If $X$ and $Y$ are two Bermoulli rv's, $X \sim B(q_x),\; Y \sim B(q_y)$, for which we have $P(X=1) = E(X) = q_x$, and analogously for $Y$, their covariance is

$$\operatorname{Cov}(X,Y)= E(XY) - E(X)E(Y) = \sum_{S_{XY}}p(x,y)xy - q_xq_y $$

$$ = P(X=1,Y=1) - q_xq_y = P(X=1\mid Y=1)P(Y=1)-q_xq_y$$

$$= \Big(P(X=1\mid Y=1)-q_x\Big)q_y $$

For uncorrelatedness we require the covariance to be zero so

$$\operatorname{Cov}(X,Y) = 0 \Rightarrow P(X=1\mid Y=1) = P(X=1)$$

$$\Rightarrow P(X=1,Y=1) = P(X=1)P(Y=1) $$

which is the condition that is also needed for the variables to be independent.

So my question is: Do you know of any other distributions (continuous or discrete) for which uncorrelatedness implies independence?

Meaning: Assume two random variables $X,Y$ which have marginal distributions that belong to the same distribution (perhaps with different values for the distribution parameters involved), but let's say with the same support eg. two exponentials, two triangulars, etc. Does all solutions to the equation $\operatorname{Cov}(X,Y) = 0$ are such that they also imply independence, by virtue of the form/properties of the distribution functions involved? This is the case with the Normal marginals (given also that they have a bivariate normal distribution), as well as with the Bernoulli marginals -are there any other cases?

The motivation here is that it is usually easier to check whether covariance is zero, compared to check whether independence holds. So if, given the theoretical distribution, by checking covariance you are also checking independence (as is the case with the Bernoulli or normal case), then this would be a useful thing to know.
If we are given two samples from two r.v's that have normal marginals, we know that if we can statistically conclude from the samples that their covariance is zero, we can also say that they are independent (but only because they have normal marginals). It would be useful to know whether we could conclude likewise in cases where the two rv's had marginals that belonged to some other distribution.

Alecos Papadopoulos
  • 52,923
  • 5
  • 131
  • 241
  • Logically, there is no question here: take any pair of independent variables as the distribution. Whether or not they are correlated, they are independent by *fiat*! You really need to be more precise about what you mean by "distribution" and what kinds of answers you will find useful. – whuber Nov 02 '13 at 21:00
  • @whuber I don't understand your comment. I _start_ by uncorrelatedness and ask "if I can prove that they are uncorrelated when does this imply that they are also independent"? Since the two results stated in the question depend on the rv's having a specific distribution (normal or Bernoulli), I ask "is there any other known distribution for which, if the two variables follow it, this results holds"? – Alecos Papadopoulos Nov 02 '13 at 21:04
  • 3
    Take any two independent variables $X,Y$ and let $F$ be their distribution. $F$ is a valid answer to your question. Note that you are asking to prove a conditional, which by definition is true whenever the consequent is true, *no matter what* the truth value of its antecedent may be. Thus, by the basic rules of logic, *all* distributions of independent variables are answers to your question. – whuber Nov 02 '13 at 21:07
  • @Whuber, you are evidently right. I added some text related to the motivation for this question, which I hope clarifies what my motivation was. – Alecos Papadopoulos Nov 02 '13 at 21:20
  • I'm still puzzled. Precisely *how* are you expecting to be "given" a theoretical distribution? In many cases it is immediate just from looking at the formula for a CDF, PDF, CF, or MGF, that the variables are independent. – whuber Nov 02 '13 at 21:24
  • 2
    What information do you start with when making this decision? From the formulation of your example, it seems like you are given the marginal pdf for each variable and the information that each pair of variables are uncorrelated. You then decide if they are also independent. Is this accurate? – probabilityislogic Nov 02 '13 at 22:48
  • @probabilityislogic Yes what I have is the marginals. I added some more on the question that answers your question. – Alecos Papadopoulos Nov 02 '13 at 23:51
  • I don't know exactly which distribution, but my intuition is as follows. Independence means all comoments should vanish. For Gaussian 2nd order comoment COV is all it takes. The question is is there any other bivariate distribution that has this property. Maybe moment generating functions is a good place to start. – Cagdas Ozgenc Nov 03 '13 at 01:17
  • @CagdasOzgenc Good idea. – Alecos Papadopoulos Nov 03 '13 at 01:57
  • It's late here so I might not be accurate. I think you could ask: "For which distributions any 2 dependent Random Variables, $ X $ and $ Y $ must have non zero correlation?". As it cancels the case @whuber created. – Royi Jun 02 '18 at 23:05
  • For the opposite see https://stats.stackexchange.com/questions/85363/simple-examples-of-uncorrelated-but-not-independent-x-and-y – kjetil b halvorsen Dec 07 '21 at 02:04

1 Answers1

13

"Nevertheless if the two variables are normally distributed, then uncorrelatedness does imply independence" is a very common fallacy.

That only applies if they are jointly normally distributed.

The counterexample I have seen most often is normal $X \sim N(0,1)$ and independent Rademacher $Y$ (so it is 1 or -1 with probability 0.5 each); then $Z=XY$ is also normal (clear from considering its distribution function), $\operatorname{Cov}(X,Z)=0$ (the problem here is to show $\mathbb{E}(XZ)=0$ e.g. by iterating expectation on $Y$, and noting that $XZ$ is $X^2$ or $-X^2$ with probability 0.5 each) and it is clear the variables are dependent (e.g. if I know $X>2$ then either $Z>2$ or $Z<-2$, so information about $X$ gives me information about $Z$).

It's also worth bearing in mind that marginal distributions do not uniquely determine joint distribution. Take any two real RVs $X$ and $Y$ with marginal CDFs $F_X(x)$ and $G_Y(y)$. Then for any $\alpha<1$ the function:

$$H_{X,Y}(x,y)=F_X(x)G_Y(y)\left(1+\alpha\big(1-F_X(x)\big)\big(1-F_Y(y)\big)\right)$$

will be a bivariate CDF. (To obtain the marginal $F_X(x)$ from $H_{X,Y}(x,y)$ take the limit as $y$ goes to infinity, where $F_Y(y)=1$. Vice-versa for $Y$.) Clearly by selecting different values of $\alpha$ you can obtain different joint distributions!

Silverfish
  • 20,678
  • 23
  • 92
  • 180
  • 1
    @Alecos Since marginal distributions don't determine joint distribution in general (just edited my answer to make this clear), where does this leave your question? – Silverfish Nov 03 '13 at 02:07
  • In the Bernoulli case, we do not need to have a joint Bernoulli distribution for the result to hold. – Alecos Papadopoulos Nov 03 '13 at 02:08
  • 1
    @Alecos I think I have a better understanding of the substance of the question now: given two marginal distributions, there is an infinite set of possible joint distributions. In what circumstances does imposing the condition of zero covariance leave us with only one of those joint distributions still possible, viz the one in which the random variables are independent? – Silverfish Nov 03 '13 at 23:48
  • I would dare say, you hit right on the spot. – Alecos Papadopoulos Nov 04 '13 at 00:50
  • 1
    If I stick to the bivariate case, with joint MGF $M_{X,Y}(s,t)$ and marginal MGFs $M_X(s)=M_{X,Y}(s,0)$ and $M_Y(t)=M_{X,Y}(0,t)$, the question becomes: when does $\frac{\partial^2}{\partial s \partial t}M_{X,Y}(s,t)|_{s=0,t=0} = \frac{\partial}{\partial s} M_{X,Y}(s,t)|_{s=0,t=0} \cdot \frac{\partial}{\partial t} M_{X,Y}(s,t)|_{s=0,t=0} $ imply that $M_{X,Y}(s,t)=M_{X,Y}(s,0) \cdot M_{X,Y}(0,t)$? – Silverfish Nov 04 '13 at 01:07
  • 2
    @Silverman I would check the concept of _subindependence_, http://en.wikipedia.org/wiki/Subindependence, to see whether this problem can be formulated in terms of moment generating functions. – Alecos Papadopoulos Nov 04 '13 at 09:35
  • I see that the condition for independence in terms of MGF was verified to be both necessary and sufficient. Subindependence is a case of zero covariance between _dependent_ rv's. Namely, a case of _purely non-linear dependence_. My hunch is that such a purely non-linear dependence is rather rare in practice, so instead of just re-iterating "uncorrelatedness does not imply independence" it would be much more useful to theory and applications to extend the cases where uncorrelatedness does imply independence, say, for some known and widely used distributions. (CONT'D) – Alecos Papadopoulos Nov 06 '13 at 19:02
  • CONT'D For example, I think (not fully checked though) that if the joint distribution is produced from the formula that you gave in your answer, then if the variables are uncorrelated, they will be also independent (i.e. the only way that they can be uncorrelated is for $\alpha =0$, which makes them also independent). And this is a very general result _irrespective_ of what are the marginals. – Alecos Papadopoulos Nov 06 '13 at 19:06