5

I was confused by what it means when a Random variable is a deterministic function of another Random variable yet is independent of it? How is this possible?

Here's the question: Consider three random variables $X,Y$ and $Z$ such that $Z$ is independent of $X$ and of $Y$ and is a deterministic function of $(X,Y)$. Is $H(Z)=0$? If so prove it; if not, give a counter example.

Christopher
  • 53
  • 1
  • 4
  • 3
    Can you provide reference mentioning it and a broader context? Basically, if one random variable is a function of another one it is *dependent* of it (http://stats.stackexchange.com/questions/16321/are-the-random-variables-x-and-fx-dependent and http://stats.stackexchange.com/questions/231425/can-someone-help-to-explain-the-difference-between-independent-and-random/231432#231432 ) – Tim Sep 18 '16 at 17:53
  • @Tim That's what i thought too . Here's the question . Consider three random variables X,Y and Z such that Z is independent of X and of Y and is a deterministic function of (X,Y). Is H(Z)=0 ? If so prove it ; if not , give a counter example – Christopher Sep 18 '16 at 17:59
  • please edit to add those details. Btw is it a homework? – Tim Sep 18 '16 at 18:38
  • @Tim No , we had a test and i couldn't solve it . Wanted to find out what it meant – Christopher Sep 18 '16 at 18:54
  • 1
    Do you mean Z=H(X,Y)=0 or does H(Z) mean something else? – Patty Sep 18 '16 at 19:01
  • @Patty The function ain't defined . It just says Z is deterministic of X and Y . And you have to prove either entropy of Z is zero or not. – Christopher Sep 18 '16 at 19:09
  • 2
    @Christopher still I think you should add `[self-study]` tag and check: http://stats.stackexchange.com/tags/self-study/info – Tim Sep 18 '16 at 19:10
  • @Tim , i did provide a counter example but i don't understand what it means to be independent and deterministically related – Christopher Sep 18 '16 at 19:19
  • Can you provide the counter example here? Loosely speaking, it means that while Z is dependent on X and Y, but knowledge of X and Y tells us nothing of Z, which appears contradictory. I'm interested in your counter example because I can't think of such a X,Y, and Z myself. – Patty Sep 18 '16 at 19:20

2 Answers2

9

A non trivial univariate real random variable that is a deterministic function of another random variable is not independent of it (see Mark L. Stone's answer for an example with a constant random variable). However, when more than two random variables are involved independence shows counterintuitive behaviours.

I'll give an example of $Z$ deterministic function of $X$ and $Y$ but independent from $X$ and $Y$.

Let $X$ and $Y$ be independent Bernouilli variables with $p=0.5$ (for example, $X$ and $Y$ are the results of tossing a coin each).

Let $f(X,Y)$ equal $1$ if $X=Y$ and $0$ if $X$ is different from $Y$.

Let $Z=f(X,Y)$.

You can easily see that $P(X=0)=0.5=P(X=0\mid Z=1)=P(X=0\mid Z=0)$ and that $P(X=1)=0.5=P(X=1\mid Z=1)=P(X=1\mid Z=0)$, proving that $X$ and $Z$ are independent, or using another definition of independence:

\begin{align*}P(x=0\text{ and }Z=0) &= P(X=0\text{ and }X \text{ different from }Y) \\ &= P(X=0\text{ and }Y=1) \\ &= 0.5 \cdot 0.5 = 0.25 \\\\ P(X=0) \cdot P(Z=0) &= 0.5 \cdot 0.5 = 0.25 \end{align*}

The same operation can be done for all values of $X$ and $Z$, thus proving that $P(X=a\text{ and }Z=b) = P(X=a)\cdot P(Z=b)$ for every value of $a$ and $b$.

Furthermore, the same proof holds for $Y$, therefore proving that $Y$ and $Z$ are independent and $X$ and $Z$ are independent. In fact, $X$, $Y$ and $Z$ are pairwise independent while $X$, $Y$ and $Z$ are not independent considered as a whole (not jointly independent). Interestingly, $Z$ is not independent of $(X, Y)$.

Pere
  • 5,875
  • 1
  • 13
  • 29
  • 1
    Your statement "An univariate real random variable that is a deterministic function of another random variable is not independent of it. " is incorrect, as shown in my answer. – Mark L. Stone Sep 18 '16 at 19:32
  • @Patty , the counter example i provided was an XOR . Seems like this is correct . Yes , Pere . +1 (Pairwise independence does not imply joint independence ) . – Christopher Sep 18 '16 at 19:34
  • Well, the correctness of your edited statement depends on your definition of non-trivial function. Consider any function $f(x)$ which is one to one. Apply it to a random variable $Y$ which is a constant with probability one. Then $f(Y)$ is independent of $Y$. – Mark L. Stone Sep 18 '16 at 19:39
  • Yes, I edited it again to apply "trivial" to the resulting random variable and not to the function. I'm not sure if there is a rigorous definition of trivial random variable, but I think it's clear enough - in fact, I'm not sure if there is a definition of random variable that includes something that always gives the same result (a deterministic random variable). However, since it points to your answer I think readers will easily understand what is the exception about. – Pere Sep 18 '16 at 19:44
  • @Pere I took the liberty of adding MathJax to your (great) answer to improve readability. I hope that is okay - otherwise reject the edit! – Therkel Sep 19 '16 at 07:12
6

A random variable which is a constant with probability 1 is independent of itself. I leave the trivial proof to you as an exercise.

Consider the deterministic function f(x) = x, applied to the random variable $Y$, which equals a constant, say $e^\pi$, with probability one. Therefore, the random variable $f(Y)$ is independent of $Y$. If you don't like that example because $f(Y)$ is the same as $Y$, then use the function $f(x) = 2x$. Same conclusion.

This provides a counterexample to the incorrect statement "An univariate real random variable that is a deterministic function of another random variable is not independent of it." in the first paragraph by the answer by @Pere .

Mark L. Stone
  • 12,546
  • 1
  • 31
  • 51