1

I have a Bayesian model and was looking to do some model checking via Posterior Predictive p-values.

From Bayesian Data Analysis (Gelman et al) it is stated that we adopt a test quantity $T(y,\theta)$ in order to assess whether the statistic with random replications of the data exceeds the statistic from our original data.

$$T(y^{\text{rep }s},\theta^{s})\geq T(y,\theta^{s})$$

Now, it states that the test quantity can either be a function of the data $T(y)$ or a function of both the data and the parameters $T(y,\theta)$.

I can easily think of statistics to assess that are a function of the data (mean, variance etc.). However, I am struggling to think of a statistic that could also depend on the parameters. I think I am obviously overlooking some simple examples.

epp
  • 2,372
  • 2
  • 12
  • 31

1 Answers1

1

I'm sure there are many answers. Bayesian Data Analysis (3rd edition) page 146 already gives an example:

$$ T(y, \theta) = |y_{(61)} - \theta| - |y_{(6)} - \theta| $$

where $\theta$ is the mean of your normal model. This is from an example with 66 observations, so ordered observations 6 and 61 are chosen to roughly assess the 10% and 90% quantiles. This test quantity evaluates whether the normal model is adequate for the 'main body' of the distribution - the middle excluding the extreme tails.

Another example could be mean squared error in a regression model: $$ T(y, \theta) = \frac{||y - X \theta||^2_2}{n} $$

This also depends on both your data and parameters.

Maurits M
  • 578
  • 1
  • 5
  • 10
  • Thanks for your response. I'm confused as my understanding is the data is a function of the parameters i.e. $y=f(\theta)$. So how does it make sense to compare the $y$ with the $\theta$? – epp May 22 '18 at 07:43
  • I'm not sure I can take away your confusion, but I'll try. I think when you say 'the data is a function of the parameters' this is not quite true - the data arise randomly from a process with certain parameters. We are trying to understand that process and find the parameters that make most sense in the context of the observed data and the process. In this sense the data depend on the parameters but are not a function of those parameters. For example, if flip a coin twice, both observations can be different even though they come from the same process (with identical parameter). Hope this helps – Maurits M May 22 '18 at 07:49
  • Sorry, I probably wasn't clear. Take your MSE example. Surely, when we perform the replication (i.e. generating the data using a random parameter(s) from the posterior), the replicated data $y^{\text{rep }s}$ will always be equal to $X\theta^{s}$ (because it was generated from that $\theta^{s}$)? so the MSE for the replicated data will always be zero? – epp May 22 '18 at 07:55
  • Ah, now I understand where you're going. With regular linear regression you would generate $y^{rep}$ from a normal distribution with mean $X \theta$ and some variance $\sigma$ that you estimated, so $T(y^{rep}, \theta)$ is definitely not always zero. In fact, you should compare this statistic to $T(y, \theta)$ to see if your simulated data behaves similarly to the original data. Then you can formulate alternative tests (like fit only on the left or right tail) to further assess how your model compares with the data. – Maurits M May 22 '18 at 08:08
  • Thanks again. Does the above MSE statistic only work in this kind of regression context? What if my model was simply $Y\sim F(y;\theta)$ where $F$ denotes some distribution? I have data $y$ and can sample $\theta^{s}$ which gives me $T(y,\theta^{s})$ but how would I do the same for $T(y^{\text{rep} s},\theta^{s})$? – epp May 22 '18 at 08:42
  • So when you have your model you can simulate $y^{rep}$ from the same data-generating process that you estimated, so $y^{rep} \sim F(y; \theta^s)$. In essence, you want to compare these data from your process with the original data, as a way to measure whether your process could reasonably explain how the data were generated. – Maurits M May 24 '18 at 13:42