ANOVA when group differences aren't clear-cut

Question

I'm wondering if anyone can help me with a really tricky one-way independent ANOVA problem I'm having? Here's a bit of background:

There are 3 diagnostic groups (depression, anxiety, non-clinical). The groups are independent, but participants in the depression group might also have an anxiety diagnosis. The dependent variable is a continuous measures of self-esteem. A-priori power analysis indicated that I need around 50 participants per group to detect a medium effect, which I do. However, after doing an initial ANOVA to look at differences between these groups, I need to do a follow-up to see whether controlling for anxiety in the depression group affects the group differences (as they might just be similar on the DV due to shared high levels of anxiety).

I can't use the continuous measure of anxiety I've used to categorise participants into the diagnostic groups as a covariate, as this would remove the effect of anxiety in the anxiety group (which doesn't make much sense!) The only alternative option I've found is to split the depression group into 'depression with anxiety' and 'depression without anxiety' and then run another one-way ANOVA this way. However, the 'depression without anxiety' group is far smaller than 50, and so the test wouldn't be adequately powered.

Can anyone think of a way around this that results in a suitable, adequately powered test? Essentially, I need to control for a variable in one group but not the others!

Answer 1

As you seem to have a continuous measure of anxiety, you should take advantage of it.

Instead of continuing with an ANOVA-type analysis you could consider a linear model that specifically allows for both anxiety and depression scores for each individual. If you have anxiety and depression scores for each individual (including the non-clinical group) then you could use those scores plus an interaction between anxiety and depression scores to try to characterize your outcome. If we call the anxiety score "A", the depression score "D", and use ":" to represent the interaction then you could use a linear regression model:

outcome ~ A + D + A:D

That would also cover the non-clinical group, whose members presumably have low "A" and "D" scores. As your "A" score is continuous, this could work quite well even if "D" is binary. That would allow modeling of any combination of anxiety and depression score. This linear modeling of "A" and "D" and their interaction only uses up 3 degrees of freedom, versus the 2 in your ANOVA model, so your 150 cases should still be enough even if you, say, have to do some non-linear modeling of the "A" scores with a spline.

You will, however, have to be careful when interpreting the individual "A" and "D" coefficients with an interaction in the model. The coefficient for "A" will be the association of outcome with "A" when "D" is at its reference value (or 0 if continuous), and that for "D" will be the association of outcome with "D" when your continuous measure "A" is at a value of 0. If the interaction is significant then there won't be a single value representing either of anxiety or depression. That might be more complicated than you were hoping for, but if that's the reality then that is what your audience deserves to know.

That said, I am concerned about the issues raised by @BruceET with respect to re-design of the analysis after the data have been collected. I think this type of analysis will help get around those issues, but I haven't thought fully through whether the procedure for selecting the individuals for this study might lead to some problems if you proceed this way.

Answer 2

Re-designing an experiment after you see the data can be a direct path to false discovery.

If you can use pre-experiment information as criteria to split the Depression group for a post hoc analysis whether the two parts of it differ significantly. If so, then your report needs to admit the difficulty and unorthodox re-design along with findings.

Here are data for a design such as yours.

set.seed(2020)
NC = rnorm(50, 100, 20)
D  = rnorm(50, 115, 30)
A  = rnorm(50, 110, 20)
x = c(NC, D, A)
g = rep(1:3, each=50)
boxplot(x ~ g, col="skyblue2", pch=20)

oneway.test(x ~ g)

        One-way analysis of means (not assuming equal variances)

data:  x and gp
F = 3.5155, num df = 2.000, denom df = 95.537, p-value = 0.03364

A "re-design" based on illegitimate use of data, can produce bogus differences. This is an extreme case, but it is why I mentioned that splitting the Depression group needs to be based on pre-experiment information.

D1 = sort(D)[1:20]    # smallest into D1
D2 = sort(D)[21:50]   # largest into D2
y = c(NC, D1, D2, A)
gp = rep(1:4, c(50,20,30,50))
boxplot(y~gp, col="skyblue2", pch=20, varwidth=T)

oneway.test(y~gp)

        One-way analysis of means (not assuming equal variances)

data:  y and gp
F = 26.23, num df = 3.000, denom df = 65.733, p-value = 2.843e-11

t.test(D1,D2)$p.val
[1] 1.492356e-11