1

I have two samples - one is predicted values and one is actual values. Paired samples t-test suggested that there are no differences between these two samples.

Because I remember from stats courses that the absence of a difference doesn't mean that the samples are similar, I wanted to ask if there is a statistical significance test to check for similarity of 2 samples?

Alexis
  • 26,219
  • 5
  • 78
  • 131
Itai Sevitt
  • 11
  • 1
  • 2
  • But a prediction doesn't meet the condition of a "sample" does it? Do the predictions even correspond to the "actual values" (the thing that is a sample)? How is this different, if at all, from evaluating predictive accuracy? – AdamO Feb 05 '18 at 17:22
  • "Paired samples t-test suggested that there are no differences between these two samples." Statement suggests confirmation bias. Likely should be "Paired samples t-test **failed to find evidence of** differences between these two samples." Remember: we don't accept the null hypothesis, just fail to reject it. (Of course, we *can* try to find evidence of the equivalence of two quantities, but that requires [tests for equivalence](https://stats.stackexchange.com/tags/tost/info)). – Alexis Feb 05 '18 at 21:10
  • Another user has posted [a similar question to yours regarding tests for similarity](https://stats.stackexchange.com/questions/327451/test-if-mean-is-similar-to-a-expected-value), and I was able to walk through the construction of the test statistics and inference. – Alexis Feb 08 '18 at 06:51

3 Answers3

2

You want to perform a test for equivalence of two samples. One was to do this is to perform two one-sided tests based on a preferred Type I error rate, and an a priori equivalence threshold (i.e. the smallest difference between the two samples that you would care about).

Your general null and alternate hypothesis in such a case would be:

$H^{-}_{0}: |\mu_{A}-\mu_{B}|\ge\Delta$
$H^{-}_{0}: |\mu_{A}-\mu_{B}|< \Delta$

Which will translate into the two specific null and alternatives:

$H^{-}_{01}: \mu_{A}-\mu_{B}\ge\Delta$
$H^{-}_{01}: \mu_{A}-\mu_{B}< \Delta$

and

$H^{-}_{02}: \mu_{A}-\mu_{B}\le-\Delta$
$H^{-}_{02}: \mu_{A}-\mu_{B}> -\Delta$

If you reject both $H^{-}_{01}$ and $H^{-}_{02}$, then you conclude that the difference in these means is greater than $\Delta$ but less than $\Delta$.

Two one-sided tests are implemented for R in the equivalence package (within R type install.packages(c("equivalence")) to install), and for Stata in the tost package (within Stata type net install tost, from(https://alexisdinno.com/stata) to install the package and net get tost, from(https://alexisdinno.com/stata) for the example data).

Alexis
  • 26,219
  • 5
  • 78
  • 131
-1

No.

First, significance tests make inferences about differences between populations, not samples. You don't need a significance test if all you want to do is check something like which sample mean is larger—you can do that directly.

Second, the idea of a significance test is to examine how unlikely observations would be under a null hypothesis. If the answer is "extremely unlikely", you can infer that the null hypothesis is implausible. But if the answer is "likely", it doesn't follow that the null hypothesis is plausible. The null hypothesis is no more or less plausible than it was to begin with. In short, significance-testing can't find evidence in favor of a null hypothesis.

Kodiologist
  • 19,063
  • 2
  • 36
  • 68
  • [tost](https://stats.stackexchange.com/tags/tost/info) – Alexis Feb 06 '18 at 00:10
  • @Alexis It seems pretty clear to me that the question is about how you can confirm the null hypothesis of precisely 0 difference after a failed test of this hypothesis. So I explained why you can't. The TOST method is a different animal. – Kodiologist Feb 06 '18 at 01:10
  • The OP's language was "similar", so I disagree. But reasonable people can do this. :) – Alexis Feb 06 '18 at 01:14
-1

You can't do that exactly, but something very similar. Take the difference of all pairs (actual value minus simulated) and compute a one-sample t-test to test, whether the difference is different from 0. The p-value will not be of any interest to you, as that deals with the question of the difference being exactly $0.0000000000\dots$

However, most statistic programs will not only give you a p-value but also a 95% confidence intervall of the true difference. You may define for yourself, that you consider the sample mean differences as similar, if the 95%-confidence interval does not contain values above 0.5 or below -0.5 or any other limit of practical equality (ok, this may work even better with a Baysian credible interval, but given the question you will need an answer that es easy to produce).

A worked example in R:

simulated <- c(3.8, 3.2, 4.7, 9.9, 1.2)
measured  <- c(3.2, 3.8, 4.8, 10.3, 1.3)

delta <- simulated - measured
print(t.test(delta, mu = 0)$conf.int)

This will give a 95% confidence interval from -0.685 to 0.445. Now you can decide, whether that is similar enough for your purposes. Other software will be able to achieve the same similarly easy.

Bernhard
  • 7,419
  • 14
  • 36