1

This is a spinoff of

How to calculate the confidence interval of the mean of means?

and related to

When making inferences about group means, are credible Intervals sensitive to within-subject variance while confidence intervals are not?

Dataset 1 here is taken from the first link above. Dataset 2 has the approximately the same experimental means but different within experiment variance. My first question is:

1) How do I calculate a confidence interval for the overall mean for each of these data sets?

If I understand @Stéphane Laurent's answer in the two linked questions they should be the same.If that is true this goes strongly against all my scientific intuition and also appears to be a paradox.

2) How can it be that the confidence interval is apparently both sensitive to and not sensitive to within experiment error?

enter image description here

dataset 1:

  Experiment Value
          1    34
          1    41
          1    39
          2    45
          2    51
          2    52
          3    29
          3    31
          3    35

structure(list(Experiment = structure(c(1L, 1L, 1L, 2L, 2L, 2L, 
3L, 3L, 3L), .Label = c("1", "2", "3"), class = "factor"), Value = c(34, 
41, 39, 45, 51, 52, 29, 31, 35)), .Names = c("Experiment", "Value"
), row.names = c(NA, -9L), class = "data.frame")

dataset2:

  Experiment    Value
          1 38.20744
          1 37.99410
          1 37.96299
          2 49.27085
          2 49.40519
          2 49.24894
          3 31.81259
          3 31.73708
          3 31.73834

structure(list(Experiment = structure(c(1L, 1L, 1L, 2L, 2L, 2L, 
3L, 3L, 3L), .Label = c("1", "2", "3"), class = "factor"), Value = c(38.2074373061779, 
37.9941025108851, 37.9629896019425, 49.2708491636015, 49.4051867974062, 
49.2489418702291, 31.8125943239769, 31.7370826901692, 31.7383364604132
)), .Names = c("Experiment", "Value"), row.names = c(NA, -9L), class = "data.frame")
Community
  • 1
Flask
  • 1,711
  • 1
  • 14
  • 24
  • The answer is given by my answer in the first link: take the group means and draw a classical confidence interval for a Gaussian mean. So obivously CI are the same if group means are the same. – Stéphane Laurent Oct 24 '13 at 06:13
  • Right now I believe that a statement such as "In the long run, the between variance is a function of the within variance" is what I was looking for but I was asking the wrong questions and you were answering too mathematically for me to get. I need to think more on @John's answer below. – Flask Oct 24 '13 at 06:59
  • Sorry, I like to give less mathematical answers when needed, but I'm too busy. The important variance is $\sigma^2_b+\frac{\sigma^2_w}{J}$, which is the variance of the observed means. Here you're looking at one example only, but if you run simulations, the dispersion of the observed means increase with $\sigma^2_w$. – Stéphane Laurent Oct 24 '13 at 11:43
  • @Stéphane Your answers were good and I get it now. My remaining issue is that I am still not positive that it makes sense to assume within variation is related to between variation when analyzing a single data set. In the long run sure, but I am not sure it is useful to me to care about "the long run" as a researcher. – Flask Oct 24 '13 at 21:21
  • You have a sharp mind. I think I see why you are puzzled. Consider [my notations here](http://stats.stackexchange.com/a/72610/8402) for the random one-way ANOVA model. The between-variance $\sigma^2_b$ is the variance of the *theoretical means* of the groups. It is not related to the within-variance $\sigma^2_w$. But the variance of the *observed means* $\bar y_{i\bullet}$ is $\sigma^2_b+\frac{\sigma^2_w}{J}$. You may think about this. – Stéphane Laurent Oct 24 '13 at 22:25

1 Answers1

1

I think you're confusing things with a manufactured example. Yes, you could have a specific case where the two CI's matched and you could have ones where the Dataset 1 CI was lower but on average the Dataset 1 CI will be higher. Furthermore, if these really are supposed to be multiple experiments tackling the same problem (within each Dataset) then there's something seriously wrong with Dataset 2. Lower within experiment variability should be leading to lower between experiment variability.

John
  • 21,167
  • 9
  • 48
  • 84
  • In biology an example would be to measure protein expression with a western blot. In this case the phenomenon could be explained by the within experiment error as due to pipetting errors (human error in loading the biological sample unevenly) while between experiment error as coming from different cell cultures which are not exactly the same. I would not expect a priori that the variance due to the first factor would correlate with the second. – Flask Oct 24 '13 at 04:59
  • But they would. Increased pipetting errors would increase the variance across samples as well. – John Oct 24 '13 at 05:01
  • Ok, I follow your logic. In that case perhaps this is purely a problem with me wanting a confidence interval to be something it is not. – Flask Oct 24 '13 at 05:18