2

I suspect this question is related to: When combining p-values, why not just averaging?

Let me explain my problem:

  • Suppose you have a set of sample means and associated p-values, standard errors and degrees of freedom. Let’s denote them as: $\mu_i$, $p_i$, $\text{SE}_i$, $df_i$ where $i=\{1,n\}$

  • It’s straight forward to figure out the mean of sample means. But what is the resultant standard error and p-value of the mean of sample means? I suspect p-values multiply in some manner…

Community
  • 1
Pam
  • 123
  • 3
  • Do you expect that the samples all come from the same mean (and that there is just noise in estimating each) or do you expect the samples to have different means? – David Robinson Mar 04 '14 at 20:04
  • I expect samples to have the same mean, but with noise and experimental error . – Pam Mar 04 '14 at 20:10
  • If you want to test whether the samples belong to the same population, you could test their differences directly. If your samples are distinctly different and unordered, you could mark each observation in your dataset with a nominal variable representing which sample it belongs to, and test mean differences based on this variable with an ANOVA. If the samples are ordered in some way, you could test a linear contrast or use penalized regression on the same variable, which would be ordinal. If differences in the distributions are negligible, you could just calculate the combined sample's stats. – Nick Stauner Mar 04 '14 at 21:10
  • 1
    How does this relate to Fisher's exact test (in the tags)?? $\quad\quad$ What is the hypothesis being tested? What is the alternative? What are the assumptions? – Glen_b Mar 04 '14 at 22:24
  • 1
    If you expect all the samples to have the same mean, why not just pool them all and test that? – David Robinson Mar 04 '14 at 22:38
  • Removed Fisher exact tag – Pam Mar 05 '14 at 14:18

1 Answers1

1

You can use fixed effects meta-analysis to do this. Here's an example, in R.

library(metafor)

#set up means
df <- as.data.frame(rnorm(1000))
names(df) <- "y"


#create 10 groups and calculate mean and se for each
df$group <- sample(1:10, 1000, replace=TRUE)

summaryStats <- data.frame(1:10)
summaryStats$m <- NA
    summaryStats$se <- NA

summaryStats$m <- tapply(df$y, df$group, mean)
    summaryStats$se <- tapply(df$y, df$group, function(x) sd(x)/sqrt(length(x)))
summaryStats$v <- summaryStats$se^2
summaryStats <- as.data.frame(summaryStats)


#Run meta-analysis to get combined mean and se for 10 groups
rma(yi=summaryStats$m, vi=summaryStats$v, method="FE")
#Run t-test to get mean and SE for original sample
t.test(df$y)

Here's the meta-analysis output:

> rma(yi=summaryStats$m, vi=summaryStats$v, method="FE")

Fixed-Effects Model (k = 10)

Test for Heterogeneity: 
Q(df = 9) = 4.3262, p-val = 0.8887

Model Results:

estimate       se     zval     pval    ci.lb    ci.ub          
 -0.0034   0.0316  -0.1078   0.9142  -0.0653   0.0585

Here's the t-test output:

t.test(df$y)

    One Sample t-test

data:  df$y
t = -0.0551, df = 999, p-value = 0.9561
alternative hypothesis: true mean is not equal to 0
95 percent confidence interval:
 -0.06380216  0.06031609
sample estimates:
   mean of x 
-0.001743035

They're not exactly the same, but they're very close.

Jeremy Miles
  • 13,917
  • 6
  • 30
  • 64