2

I have a random experiment with its probability space $(\Omega, \cal{F}, P)$ but no random variable defined over it. Is there any formal language for describing the variability of the random experiment? E.g. I feel the following three experiments have increasing variability, but I am not sure how to formalize this feeling (or if you have a different feeling, how to formalize that):

  1. $\Omega=\{\omega_1\}$, $\quad\ $ $Pr(\omega_1)=1$
  2. $\Omega=\{\omega_1,\omega_2\}$, $Pr(\omega_1)=0.9$, $Pr(\omega_2)=0.1$
  3. $\Omega=\{\omega_1,\omega_2\}$, $Pr(\omega_1)=Pr(\omega_2)=0.5$

I am not opposing to the idea of introducing a random variable if that helps. I am just saying there is none in the original setup.

Richard Hardy
  • 54,375
  • 10
  • 95
  • 219
  • 1
    Entropy comes immediately to mind, because the only thing that varies is the probability measure itself. Is that the sort of thing you mean by "formal language"? – whuber Jun 09 '20 at 13:45
  • @whuber, yes, I think so. But when I checked [Wikipedia's entry on entropy](https://en.wikipedia.org/wiki/Entropy_(information_theory)), I found entropy defined on a random variable, not the underlying experiment, hence my question. Can entropy be defined without reference to a random variable? – Richard Hardy Jun 09 '20 at 13:45
  • 1
    This is an unfortunate case where Wikipedia fails us. Entropy most fundamentally is a property of a discrete probability measure. Everything else is derived from that concept. – whuber Jun 09 '20 at 13:48
  • @whuber, thank you! Occasions like this are one more reason for why I respect you so much and feel indebted! Are there any mainstream alternatives to entropy relevant to my question? – Richard Hardy Jun 09 '20 at 13:54
  • The concept of *comparison of statistical experiments* comes to mind. See the classical paper by [Blackwell (of Rao-Blackwell fame)](https://projecteuclid.org/euclid.bsmsp/1200500222). I don't have time for an answer now ... – kjetil b halvorsen Sep 03 '20 at 13:09

1 Answers1

1

A salient candidate is entropy. For a random experiment $X$ with possible outcomes $\omega_1,\dots,\omega_n$ and the corresponding probabilities $P(\omega_1),\dots,P(\omega_n)$, entropy is defined as $$ H(X) =-\sum_{i=1}^n P(\omega_i) \ln \left( P(\omega_i) \right). $$ It measures the average level of "information", "surprise", or "uncertainty" inherent in the experiment's possible outcomes.

The fact that Wikipedia's entry on entropy defines entropy w.r.t. a random variable rather than the underlying experiment can be considered a failure. According to @whuber,

This is an unfortunate case where Wikipedia fails us. Entropy most fundamentally is a property of a discrete probability measure. Everything else is derived from that concept.

Richard Hardy
  • 54,375
  • 10
  • 95
  • 219
  • Since entropy can be used as a measure of (bio)diversity, alternatively other measures of (bio)diversity could also be used. Obviously diversity is variation is variability! See for instance https://stats.stackexchange.com/questions/483535/how-to-include-the-observed-values-not-just-their-probabilities-in-information/485555#485555 – kjetil b halvorsen Jul 01 '21 at 01:22
  • @kjetilbhalvorsen, thanks! I have seen and upvoted that answer already before. – Richard Hardy Jul 01 '21 at 10:52