Can you use ANOVA to compare two models that are not nested?

Question

Is it possible to use ANOVA to compare to non-nested models that have the same response variable? If not, what is the best way to compare non-nested models that have the same response variable?

Answer 1

You could use a one-factor ANOVA, but a two-sample t test seems a more natural choice.

For example, suppose you have data from two populations and you want to test whether the population means are equal.

Data sampled in R:

set.seed(118)
x1 = rnorm(50, 100, 15)
summary(x1);  length(x1);  sd(x1)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  63.46   90.72  102.62  100.18  110.84  137.16 
[1] 50          # sample size
[1] 16.94674    # sample standard deviation

x2 = rnorm(45, 105, 17)
summary(x2);  length(x2);  sd(x2)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  76.10   94.07  103.96  104.08  114.42  139.13 
[1] 45
[1] 14.88124

stripchart(list(x1,x2), pch="|", ylim=c(.5,2.5))

t.test(x1, x2)

        Welch Two Sample t-test

data:  x1 and x2
t = -1.1933, df = 92.949, p-value = 0.2358
alternative hypothesis: 
  true difference in means is not equal to 0
95 percent confidence interval:
  -10.382125   2.588149
sample estimates:
 mean of x mean of y 
  100.1837  104.0807 

I used a Welch two-sample t test rather than a pooled 2-sample t test because the population standard deviations are not equal. The sample mean of the first sample is $\bar X_1 = 100.18$ and for the second sample the mean is $\bar X_2 = 104.08,$ but considering the variability of the two samples, these sample means are not 'significantly different' at the 5% level of significance because the P-value $0.24$ of the test exceeds 5% = $0.05.$

A one-factor ANOVA can compare more than two samples. Results for the oneway.test. It is usually used when you have three or more samples. in R are as follows.

x = c(x1, x2);  gp = rep(1:2, c(50,45))
oneway.test(x ~ gp)

        One-way analysis of means 
        (not assuming equal variances)

data:  x and gp
F = 1.424, num df = 1.000, denom df = 92.949, 
  p-value = 0.2358

Notice that the P-value is the same as for the two-sided, two-sample Welch t test.