2

I am wondering what could we obtain more from the predictive distribution. Give a set of data, say $\mathcal{D}=\{(x_i,y_i)\}$, we want to predict the value s$Y_{new}$ at new locations, say $X_{new}$.

Suppose we have the predictive distribution $$ Y_{new}|X_{new}, \mathcal{D} \sim p(Y_{new}|X_{new}, \mathcal{D}). $$ Then what is our prediction at $X_{new}$? If we need a deterministic prediction, are people using the mean, i.e, $$ \text{The prediction at } X_{new} = E[Y_{new}|X_{new},\mathcal{D}] ? $$

Any comments or answers will very be appreciated. Thanks.

induction601
  • 155
  • 6

1 Answers1

3

The statistical part of the prediction exercise ends when you output a predictive distribution. Incidentally, this is a perfect end point - much better than a point prediction.

What follows after you give your predictive distribution is the decision that someone will make based on your predictive distribution. However, more than just your distribution enters into decisions: costs of "wrong" decisions, costs of "correct" decisions, just how "correct" or "wrong" a decision is made, and so forth. These can typically be included in , and the task of the decision maker is to minimize the loss based on your $\mathcal{D}$ and the cost structure.

Sometimes the "decision" is just a one-number summary of $\mathcal{D}$, and the loss can be assumed to be proportional to the squared difference between the decision and the actual outcome. If so, then the optimal decision is the one that minimizes the expected squared error. Then the optimal decision is the expectation of $\mathcal{D}$.

Or the loss may be proportional to the absolute difference between this one-number summary and the actual outcome. Then the optimal decision would be the median of $\mathcal{D}$, which minimizes the expected absolute error.

Bottom line: a point prediction makes no sense without considering the cost or the loss function which it aims to minimize. A predictive density, in contrast, can be set up (and evaluated using ) even without such costs.

I have written before on similar topics, usually shamelessly stealing from Frank Harrell and his blog, e.g.: Why use a certain measure of forecast error (e.g. MAD) as opposed to another (e.g. MSE)?

Lei Huang
  • 756
  • 6
  • 13
Stephan Kolassa
  • 95,027
  • 13
  • 197
  • 357
  • Thank you so much for your great answer! Actually, I have a follow-up question. Given a predictive distribution $p(y_{new}|x_{new},\mathcal{D})$, how one can make a decision using the predictive distribution? (for a moment, let set aside the goodness of the decision). As you mentioned, very often a loss-function is needed for a decision. But the only context I can come up with where a loss function is used is to find some optimal parameters in our prediction model. In this case, since we already have a model for our prediction, I don't see how the predictive distribution help. – induction601 Sep 20 '18 at 22:17
  • For example, let $f_{model}(x; \theta)$ be our prediction model for $y$. Then given $\mathcal{D}$, we can employ a loss function, e.g., $$ \ell(\theta;\mathcal{D}) = \sum_{(x_i,y_i)\in \mathcal{D}} (y_i - f_{model}(x_i;\theta))^2.$$ By minimizing $\ell$ over $\theta$, we can find the optimal parameter $\theta$ based on the above crierion. In this case, I don't see where and how the predictive distribution can be used in order to make a decision. – induction601 Sep 20 '18 at 22:21
  • Given your loss function and your predictive distribution, you can assume a decision and derive the distribution of the *loss* given this decision. Let's assume further that we want to minimize the expected loss. (This is a reasonable assumption - if we want to optimize something else, we can simply reformulate the loss.) We then consider the expectation of the loss *as a function of the decision* (we essentially integrate over the predictive distribution), and then we simply minimize this with respect to the decision. Does that help? – Stephan Kolassa Sep 22 '18 at 13:40
  • Thank you so much! It definitely is a great help. For example, $\mathcal{D}$ could be used to obtain the predictive distribution $p(Y_{new}|X_{new},\mathcal{D})$. Then with this distribution, one can define a loss function, e.g., $$ \ell(\theta; X_{new},\mathcal{D}) = \mathbb{E}_{\textbf{y}\sim p(\textbf{y}|X_{new},\mathcal{D})} \|\textbf{y} - f_{model}(X_{new}; \theta)\|_2^2. $$ In this case, we obtain an optimal parameter based on the predictive distribution. – induction601 Sep 22 '18 at 19:01