1

I have two values $a$ and $b$ with 95% CIs (estimated via Bootstrap if that matters). I want to take a difference between them but I am not sure of a meaningful statistic.

If I had $\sigma_a$ and $\sigma_b$, I would do $\frac{a-b}{\sqrt{\sigma_a^2 + \sigma_b^2}}$ but (a) I do not readily have that, and (b) I am really much more interested in the 95% CI.

Is there a well-known meaningful metric here?

I can't just use $\sqrt{CI_a^2 + CI_b^2}$, right? At least not without some assumption. Or can I?

Thanks!

Justin Winokur
  • 109
  • 1
  • 5
  • It sounds like you might be interested in this thread: https://stats.stackexchange.com/questions/18215. Do you know whether the CIs are independent? Do you have any sense of what the underlying sampling distribution of $a-b$ might look like? – whuber Feb 03 '20 at 23:14
  • Good questions. They are based on validation data and then prediction at the validation data location. I am not sure whether that makes them dependent. They are based on binning along a certain dimension and those bins are decided by the empirical density. They are uniform in the other directions – Justin Winokur Feb 04 '20 at 17:48
  • "Prediction at the validation data location" does not describe a confidence interval. Could you therefore be more specific about how these intervals are constructed and from what data? – whuber Feb 04 '20 at 18:25
  • @whuber, I have a surrogate function $f(\vec{x})$, trained on ${\vec{x},f}$ and I have validation data ${\vec{x'},f'}$. For all intents and purposes, I am looking at $E(f(\vec{x'})) - E(f')$ where I also have the CI's computed from a regular bootstrap estimate on $E(\cdot)$. Does that help? – Justin Winokur Feb 05 '20 at 00:03
  • I don't understand that because I can think of many possible interpretations. For instance, is $x^\prime$ random or not? How about the "surrogate function"? Exactly what parameters do these "CIs" apply to? It helps (immensely) to describe a concrete example, preferably a version of the actual problem you face. – whuber Feb 05 '20 at 14:51

0 Answers0