What statistic test to use to measure difference between samples from the same person?

Question

I have a table that looks like this:

I want for each person, to check how statistically significant the difference between the means of the two weekdays is. In my current implementation I am conducting a student's t-test. Are there other statistical tests that can help me here? I am worried because we are talking about the same person and not about two different populations and I can not find any info on how to handle that.

Thanks in advance.

BruceET · Accepted Answer · 2018-07-22T06:59:56.513

Assuming data are normal: For each person you would use a 2-sample t test to compare Day 1 and Day 3 (not a paired test, see @SalMangiafico's Comment). In your question, for subject A you show only three observations on each day. With so little data on any one person (and such large within-day variability) you will have trouble getting significant results.

For the data in your question, rounded to three places, the P-value is much larger than 0.05, so you cannot reject $H_0: \mu_1 = \mu_3$ in favor of $H_a: \mu_1 \ne \mu_3$ at the 5% level of significance. A two-sample t test in R follows:

A1 = c(.692, .198, .489);  A3 = c(.091, .446, .513)
t.test(A1, A3)

        Welch Two Sample t-test

data:  A1 and A3
t = 0.56484, df = 3.9676, p-value = 0.6026
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
  -0.4311325  0.6504658
sample estimates:
mean of x mean of y 
0.4596667 0.3500000

Not assuming normal data: With so little data, there is no point in testing for normality (or for any other particular population distribution). In case you believe data are not normal, you might use a Wilcoxon rank sum test. But such a test would require at least four measurements per person per day in order to have any chance of giving a significant result. (Even if all three measurements on one of the days are larger than any of the three measurements on the other day, you still could not possibly get a significant result at the 5% level: $2/{6 \choose 3} = 2/20 = 0.10.)$

B1 = c(.20, .30, .40);  B3 = c(.13, .02, .18)  # fake data for illustration
wilcox.test(B1, B3)

        Wilcoxon rank sum test

data:  B1 and B3
W = 9, p-value = 0.1
alternative hypothesis: true location shift is not equal to 0

Thanks a lot for this detailed answer! I have already figured that the data is not enough to produce results using a 95% confidence interval. Luckily, the usage is such that there is no reason to use such a strict confidence interval. I can probably go with 85% without a problem since false positives are not a problem in the use case. — Toutsos, Jul 23 '18 at 08:58

score -1 · Answer 2 · answered Jul 19 '18 at 18:05

-1

I suppose you did a paired t-test. You could also look at two way ANOVA. But please go through the specifications to further understand if your experiment aligns to the requisites.

answered Jul 19 '18 at 18:05

Srikrishna

40
3

Thanks for the answer! Yes, I conducted a paired t-test. Will look into a two way ANOVA as well. – Toutsos Jul 19 '18 at 18:13
If you have multiple days, you want to look at a repeated measures ANOVA. – Noah Jul 19 '18 at 19:05
1

The question talks about comparing days within each person. I don't see how a paired t-test is applicable here. – Sal Mangiafico Jul 19 '18 at 22:44
Test would have to be carried out for each person. – Srikrishna Jul 20 '18 at 03:39
By what criteron would you then pair the observations? – Sal Mangiafico Jul 22 '18 at 12:38
Using the weekday column – Srikrishna Jul 23 '18 at 15:31

What statistic test to use to measure difference between samples from the same person?

2 Answers2

Linked