I have two samples, say sample A and B, and a negative control.
I expect that the sample A to be different than the negative control, so I use a t-test to check for that.
Then, I expect sample B not to be different from the negative control. How do I test for that? I could use a t-test for that too and treat a high p value as confirmation, but I'm wondering if there is a better way?
One way to approach this question is called “equivalence testing”. You can look up information on that but generally speaking you cannot prove that two means are exactly equal (nor should you expect them to be in most cases) so you first have to define an equivalence margin (i.e. a difference that you consider negligible in practice).
Importantly, as Maarten already mentioned, a t-test is used to compare population means. It does not tell you if the two distributions are the same, if the difference is big, if there is a lot of overlap or anything like that buy only if the observed difference in means is likely to have come about purely through sampling variability. It also does not test if samples are different (to determine that, you just need to look at your data directly). The whole point of statistical inference is that your sample is a noisy representation of a broader population. Generally speaking, you should not expect statistics to tell you if something is “true” or “real” but only to help you deal with random variation.
To answer your question: not really. Statistics is very good at rejecting or not rejecting hypotheses, but confirming hypotheses is very hard (some say impossible). This is more a philosiphical than a statistical problem. An accessible introduction to this issue can be found in:
Ian Hacking (2001) "An Introduction to Probability and Inductive Logic". Cambridge: Cambridge University Press.
As an aside: With a t-test you are not testing whether the samples are equal, but only whether or not the means are equal. Below is a silly example using Stata: the three variables x, y, and z are clearly from different distributions, but we (correctly) cannot reject the hypothesis that the means are equal.
. clear
. set obs 10000
obs was 0, now 10000
. gen x = rnormal()
.
. gen u = runiform()
. gen y = (u < .5)*( rnormal() - 2 ) + ///
> (u >= .5)*( rnormal() + 2 )
.
. gen z = rchi2(2) - 2
.
. twoway kdensity x || ///
> kdensity y || ///
> kdensity z
.
. ttest x = y, unpaired unequal
Two-sample t test with unequal variances
------------------------------------------------------------------------------
Variable | Obs Mean Std. Err. Std. Dev. [95% Conf. Interval]
---------+--------------------------------------------------------------------
x | 10000 -.0169198 .010027 1.002698 -.0365747 .0027351
y | 10000 -.0180979 .022326 2.232599 -.0618613 .0256655
---------+--------------------------------------------------------------------
combined | 20000 -.0175089 .0122368 1.730549 -.0414941 .0064763
---------+--------------------------------------------------------------------
diff | .0011781 .0244743 -.0467948 .049151
------------------------------------------------------------------------------
diff = mean(x) - mean(y) t = 0.0481
Ho: diff = 0 Satterthwaite's degrees of freedom = 13875
Ha: diff < 0 Ha: diff != 0 Ha: diff > 0
Pr(T < t) = 0.5192 Pr(|T| > |t|) = 0.9616 Pr(T > t) = 0.4808
. ttest x = z, unpaired unequal
Two-sample t test with unequal variances
------------------------------------------------------------------------------
Variable | Obs Mean Std. Err. Std. Dev. [95% Conf. Interval]
---------+--------------------------------------------------------------------
x | 10000 -.0169198 .010027 1.002698 -.0365747 .0027351
z | 10000 .0086713 .0198435 1.98435 -.0302259 .0475686
---------+--------------------------------------------------------------------
combined | 20000 -.0041242 .0111166 1.572121 -.0259136 .0176652
---------+--------------------------------------------------------------------
diff | -.0255912 .022233 -.0691705 .0179882
------------------------------------------------------------------------------
diff = mean(x) - mean(z) t = -1.1510
Ho: diff = 0 Satterthwaite's degrees of freedom = 14792.6
Ha: diff < 0 Ha: diff != 0 Ha: diff > 0
Pr(T < t) = 0.1249 Pr(|T| > |t|) = 0.2497 Pr(T > t) = 0.8751
You design your study to have sufficient "statistical power" to detect a meaningful difference. This requires that you pre-specify the size of a difference between the "negative control" and treatment B that you would consider "small" or unimportant. You then back-calculate to see how probable it would be to find a null result (fail to reject the null) if the true value is at least that large. You want the probability to be small and you choose your sample size to make it sufficiently small, If the design goal of testing similarity of treatment B to the control is an importance design criterion, it would be wise to make the probability of such a failure 10% or lower. You would then say you have 90% "power" to detect an important difference. There are many places you can find formulas to calculate this sample size, although for complex designs where you may be controlling for confounding factors, it may require simulation.
