6

In decision theory, we define the risk associated with a particular predictor function as the expected value of the loss function. Since the input and output are considered random variables therefore the loss function is also a random variable.

I am wondering why do we assume that the expected value of loss is considered a good description of the random variable? In my understanding,the expected value of a random variable is not necessarily a good description of it.

So,why is the risk function defined as the expected value of the loss function?

Richard Hardy
  • 54,375
  • 10
  • 95
  • 219
curious131
  • 61
  • 1
  • 3
    Are you familiar with a utility function in economics? A utility function represents preferences: $x$ is preferred to $y$ if and only $u(x) \geq u(y)$. If you buy the [Von Neumann and Morgenstern axioms](https://en.wikipedia.org/wiki/Von_Neumann%E2%80%93Morgenstern_utility_theorem#The_axioms) then preferences over a random outcome can be represented by expected utility: random variable $X$ is preferred to $Y$ if and only if $\operatorname{E}[u(X)] \geq \operatorname{E}[u(Y)]$. In a basically identical way, you can interpret the loss function as a representation of your preferences. – Matthew Gunn Oct 03 '18 at 22:55
  • 2
    I agree with @MatthewGunn 's answer. If you find a utility (loss) that is better in terms of expected value then every other utility, then that utility is the most rational choice. Well to be more accurate "the most VNM-rational choice". – idnavid Oct 04 '18 at 00:53

2 Answers2

4

In my understanding, the expected value of a random variable is not necessarily a good description of it.

This depends on what you mean by "description". The expectation has a number of interpretations, all of which might or might not be "good" for you.

In frequentist terms, it is the long-run average of a data-generating process. If you draw from a random variable $X$ an infinite number of times, the average of the observations will converge on $E(X)$.

Mathematically, it is a weighted average of the possible outcomes (even in the continuous case if you squint at it). The more probable an outcome, the greater its weight.

The expectation is also the center of mass of the probability distribution. This description is appealing in higher dimensions (where you can think of the data as occupying a "blob" in space), and is analogous to the center of mass in physics.

Finally, the expectation is a location parameter. This means that a change in the expectation of the distribution represents a shift in the density of the distribution. If you change the expected value, it's like you are picking the density of the distribution up off the graph, and just dropping it elsewhere, without otherwise modifying its shape.

The "not necessarily a good description" criticism is probably related to the fact that, in highly skewed or heavy-tailed distributions, very few observations are actually near the expected value point. This is valid, but probably not something we have the luxury to care about. As I mention below, we don't really have an alternative.

I am wondering why do we assume that the expected value of loss is considered a good description of the random variable?

  1. It is a location parameter. Smaller loss is good. If the location of the loss distribution is lower, then loss on average is smaller. This is what we want.
  2. It's relatively easy to compute. The fact that it's linear is especially helpful.
  3. The alternative location parameters (median, mode, ...?) are not so easy to compute, and are arguably less representative than the mean.
  4. We use it everywhere else anyway.
  5. In economics and decision theory, some of the easiest utility functions to work with imply that agents minimize expected loss (or equivalently maximize expected gain).

This is what it comes down to: we can compute it, it works for the most part, and there isn't a clear alternative.

shadowtalker
  • 11,395
  • 3
  • 49
  • 109
  • 1
    Thanks for the answer but some of the points seem circular. For example 'long-run *average* of a data-generating process. Why should I care about average anyway? – curious131 Oct 03 '18 at 20:32
  • I am thinking about general distributions without any assumption. – curious131 Oct 03 '18 at 20:41
  • @curious131 because averages have an intuitive meaning as "the middle" (median is a different kind of "middle"), because everyone knows what an average is, and because nobody has a generic alternative. – shadowtalker Nov 07 '18 at 20:12
1

My intuition about the topic:

In a statistical parametric setting, which is in Desion Theory bounds, we'd like to estime, say, $\theta \in \Theta$ in the best way possible, by choosing a function of the sample data (a statistic) before we met the data. Let 'best' be measured by a loss function $l: t \times \theta \to \!R^+$, where $t$ is the estimate for $\theta$. So $l(t, \theta)$ is high for poor values of $t$ and zero for $t=\theta$.

Now we want to compare statistics $T_1$ and $T_2$. One is clearly the winner if the loss function is less or equal than the other for all samples. If that's not the case, we cannot say which one is better. In other words, $T_1$ may be better than $T_2$ in some subset of sample space, but $T_2$ may be better than $T_1$ in another subset.

To remove the dependency on sample space, we may take the average value over it. That's the risk function.

Now, suppose $T_1$ is better than $T_2$ in average. $T_2$ may be better than $T_1$ in certain circumstances yet! $T_2$ mat be better than $T_1$ for some values of $\theta \in \Theta_1 \subset \Theta$, yet worse in average!

To further remove the dependency on $\Theta$ is to set a priori over $\Theta$ - this is the Bayesian approach. Here, we set 'importance' over $\Theta$: more reasonable values of $\theta$ are more important, since they're more likely to be found in reallity.

In a nutshell, I think we consider the risk function the way it is because it removes dependency on information, in a setting where we are making decision before we gather it.

In complementation of @shadowtalker's answer, it's important to notice that sometimes (like @shadowtalker suggested on heavy-tailed distributions) expectation is not sufficient to summarise a random variable (though being a good "descriptor"). In those cases, we may need variance, skewness or kurtosis.

Also, other central measure of tendendy as median are very useful in nonparametric statistcis for instance. Alhough historicaly theory was developed first for pararmetric statistics, where expected value has more appeal.

Victor Dalla
  • 31
  • 6
  • I think it does not answer the fundamental question why take average value and not some other functional? – curious131 Oct 03 '18 at 21:11
  • @curious131 if you can think of a better one, by all means let us know! – shadowtalker Oct 03 '18 at 21:16
  • @shadowtalker Sorry for being pedantic but I am trying to find if there is a mathematical justification for using expectation or we use it *only because* it just works. – curious131 Oct 03 '18 at 21:26
  • @curious131 that was the gist of my answer, but you didn't like that either. Maybe someone has some historical insight that might be more satisfying. – shadowtalker Oct 03 '18 at 21:53
  • Well, I thought it couold a good insight for your question.Anyway... you said "[...] good description of the random variable". Whats does it mean by "description of random variable"? A characterization of a random variable could be its density/cumulative and support (possible values). – Victor Dalla Oct 03 '18 at 23:43
  • Expected value is a central measure - as it is the median, mode, etc. Expected value is interesting because: it has intuition appeal, as @shadowtalker pointted out; it pops up in many theorems as LLN; it has nice properties, etc. So if you wonder about the use of expectation value itself, I suggest comparing the Wikipedia articles of it and other central measures. – Victor Dalla Oct 03 '18 at 23:52