Statistically testing the impact of a within-subject vs. between-subject design

Question

I would like to statistically test the impact of an experimental design that opposes a within-subject to a between-subject manipulation of the independent variable. In other words, does the design type interacts with the effect of my independent variable.

There is an article from Erlebacher (1977, see also 1978) that addresses this question. However, I have a hard time understanding the formulas or even reproducing the example they provide.

I was also wondering if there is a more recent version of such a test (since the article is quite old), and also if there is a R package/script to implement it.

Ultimately, it would really help me to have a concrete example in R. Below is the data they used in their 1977 article:

X.within <- data.frame(
    id  = 1:20,
    LCS = c(60, 73, 93, 10, 90, 80, 83, 37, 83, 70, 77, 7, 100, 70, 100, 43,43, 83, 40, 73),
    SCS = c(36, 53, 66, 0, 73, 43, 20, 10, 26, 40, 60, 3, 53, 26, 63, 6, 3, 30, 7, 10)
)

X.between <- data.frame(
    id = 21:60,
    cond = rep(c("LCS", "SCS"), each = 20),
    score = c(
        c(53, 77, 2, 38, 68, 92, 3, 15, 67, 53, 58, 20, 17, 40, 85, 60, 25, 3, 82, 67),
        c(62, 0, 57, 42, 3, 55, 22, 28, 45, 47, 52, 75, 38, 45, 65, 50, 2, 0, 10, 60)
    )
)

The basic idea is to test whether the X.within$LCS - X.within$SCS is different from X.between$score[X.between$cond == "LCS"] - X.between$score[X.between$cond == "SCS"].

Answer 1

Edited in response to mat's comment. Thanks for the clarification! Following idea:

First bring also the X.within in a 'long' format

library(reshape2)
X.within_long = melt(X.within, id.vars = c("id"),value.name="score") 
colnames(X.within_long)[which(colnames(X.within_long)=="variable")] = "cond"

now mark the experiment in each data.frame

X.within_long$design = "within"
X.between$design = "between"

and put both data.frames together as follows

X_all = rbind(X.between,X.within_long)

> head(X_all)
  id cond score  design
1 21  LCS    53 between
2 22  LCS    77 between
3 23  LCS     2 between
4 24  LCS    38 between
5 25  LCS    68 between
6 26  LCS    92 between

> tail(X_all)
   id cond score design
75 15  SCS    63 within
76 16  SCS     6 within
77 17  SCS     3 within
78 18  SCS    30 within
79 19  SCS     7 within
80 20  SCS    10 within

Then do a simple mixed-effect model with condition, experimantel design and the interaction of both (plus individual random-effects to model possible individual heterogenity in the within design).

library(lme4)
library(lmerTest)

lmer_model = lmer(score ~ cond + design + cond*design + (1 | id), REML = FALSE, data = X_all)
anova(lmer_model)
rand(lmer_model)

> anova(lmer_model)
Analysis of Variance Table
            Df  Sum Sq Mean Sq F value
cond         1 10793.4 10793.4 79.9210
design       1   123.7   123.7  0.9162
cond:design  1  1148.5  1148.5  8.5042

> rand(lmer_model)
ANOVA-like table for random-effects: Single term deletions

Model:
score ~ cond + design + (1 | id) + cond:design
         npar  logLik    AIC    LRT Df Pr(>Chisq)    
<none>      6 -363.18 738.37                         
(1 | id)    5 -372.79 755.57 19.207  1 0.00001173 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

The interaction is statistically significant let aside the discussion whether you should use statistical significance or p-values, indicating that the experimental design indeed matters. This is based on the identity link function, but if you plot your scores, it will indicate an excess of low scores which you might want to model separately.

The random-effects are highly significant, meaning that individual heterogeneity between participants seems to make a differences. Whether this is good or bad, depends on the context but often this is interpreted as a rejection of the between design which ignores individual heterogeneity (implicating you should go for the within approach).

Please be aware that there is no consensus about how to calculate p-values for lmer models, see here.