5

What is the best way to analyze these data:

Subjects are divided in two "Group" (Treatment A and B).

"Weight" is recorded before and 3 months after treatment.

Outcome variable: Percent reduction in weight

Main question is: whether there is any difference between 2 treatments in terms of percent reduction in weight?

Which of following is appropriate for this (or will they give same result)?

  1. Repeated measures ANOVA (with "Weight" as outcome, ["Group", "Time"] as within-factors and adjusting for "subject"). But can we use "Percent reduction in weight" here?

  2. ANCOVA (with "Percent reduction in weight" as outcome, "Group" as between-factor and "baseline weight" as a covariate)

  3. Linear mixed effects method with "Weight" as outcome, [group, time, group*time] as fixed effects and [subject] as random effect. Again, can we use "Percent reduction in weight" here?

  4. Linear model with interaction: "Percent reduction in weight" ~ "Group" * "Baseline weight"

Edit: As asked in comments, added information is about N. There are 100 subjects in each group included using randomization.

rnso
  • 8,893
  • 14
  • 50
  • 94
  • Please provide info. How were subjects assugned to the groups ? How many subjects ? – Robert Long Jul 09 '20 at 09:08
  • I have added this in question above (N=100 in each group). I thought size would be a secondary point after choosing the test. – rnso Jul 09 '20 at 10:44
  • I can't see anywhere where you have answered *"How were subjects assigned to the groups ?"* Were they just randomly assigned to treatment goups ? Modelling percent change as an outcome with baseline as a regressor is almost always a vey bad idea due to mathematical coupling – Robert Long Jul 09 '20 at 11:31
  • Yes, subjects were randomized to 2 groups. I did not get your exact question earlier. I have clarified in my question above. – rnso Jul 09 '20 at 11:38

1 Answers1

7

First there is the question of whether it is OK to use percent change as the outcome. In a regression model with baseline as a regressor this is a very bad idea because the outcome is mathematically coupled to the regressor which will induce correlation (ie statistically significant associations) where none is actually present (or mask actual change). This is easy to show with a simulation:

We simulate 2 goups of 100 each where in the first instance there is no change from baseline in either group:

set.seed(15)
N <- 200
x1 <- rnorm(N, 50, 10)
trt <- c(rep(0, N/2), rep(1, N/2))  # allocate to 2 groups
x2 <- rnorm(N, 50, 10)   # no change from baseline

So we expect to find nothing of any interest:

summary(lm(x2 ~ x1 * trt))

Coefficients:
Estimate Std. Error t value Pr(>|t|)    
(Intercept) 45.75024    5.37505   8.512 4.43e-15 ***
x1           0.06776    0.10342   0.655    0.513    
trt          3.25135    7.12887   0.456    0.649    
x1:trt      -0.01689    0.13942  -0.121    0.904

as expected. But now we create a percent change variable and use it as the outcome:

pct.change <- 100*(x2 - x1)/x1
summary(lm(pct.change ~ x1 * trt))

Coefficients:
Estimate Std. Error t value Pr(>|t|)    
(Intercept)  97.5339    12.7814   7.631 9.93e-13 ***
x1           -1.9096     0.2459  -7.765 4.44e-13 ***
trt          45.1394    16.9519   2.663  0.00839 ** 
x1:trt       -0.7662     0.3315  -2.311  0.02188 *  

Everything is significant ! So we would interpet this as: the expected percent change in weight for a subject in the control group with zero baseline weight is 97; the expected change in the percent change in weight for a subject in the control group for each additional unit of baseline weight is -1.91; the expected difference in the percent change in weight between the control and treatment group for a subject with zero baseline weight is 45; and the expected difference in the percent change in weight between the treatment and control groups for each additional unit of baseline weight is -0.77.... All completely suprious !!!! Note also that with a "percent change" variable, then we have to use language like "expected change in the percent change" which does not help with understanding.

Now let's introduce an actual treatment effect of 10,

x3 <- x1 + rnorm(N, 0, 1) + trt*10  
summary(lm(x3 ~ x1 * trt))

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept) -0.95933    0.54404  -1.763   0.0794 .  
x1           1.01921    0.01047  97.365   <2e-16 ***
trt         10.78643    0.72156  14.949   <2e-16 ***
x1:trt      -0.01126    0.01411  -0.798   0.4260    

...all good.

Now again, we create a percent change variable and use it as the outcome:

pct.change.trt <- 100*(x3 - x1)/x1
summary(lm(pct.change.trt ~ x1 * trt))

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept) -1.77928    1.23337  -1.443    0.151    
x1           0.03439    0.02373   1.449    0.149    
trt         49.11734    1.63580  30.027   <2e-16 ***
x1:trt      -0.54947    0.03199 -17.175   <2e-16 ***

..more spurious results.

As to the specific models :

Repeated measures ANOVA (with "Weight" as outcome, ["Group", "Time"] as within-factors and adjusting for "subject").

This is one option that could work.

ANCOVA (with "Percent reduction in weight" as outcome, "Group" as between-factor and "baseline weight" as a covariate)

Besides the mathematical coupling problem, this would not control for repeated measures

Linear mixed effects method with "Weight" as outcome, [group, time, group*time] as fixed effects and [subject] as random effect. Again, can we use "Percent reduction in weight" here?

This would be my preferred option, but again, not with percent reduction. This should be equivalent to repeated measures ANOVA. For example with your data:

lmer(wt ~ group*time + age + gender + (1 |Subject, data=mydata)
lme(wt ~ group*time + age + gender, random= ~ 1 | Subject, data=mydata)

You may want to add random slopes by placing one or more of the fixed effects that vary within subjects (only time in this case) to the left of the |, if justified by the theory, study design, and supported by the data. Personally I always start from a model with only random intercepts.

Linear model with interaction: "Percent reduction in weight" ~ "Group" * "Baseline weight"

This should be avoided due to the mathematical coupling problem. Even if baseline was removed as a regressor, this would then just an ANOVA model, and while repeated measures are handled by the percent variable, the residuals may not be close to normal, so inference may be affected.

Robert Long
  • 53,316
  • 10
  • 84
  • 148
  • 1
    Thanks for a detailed and well illustrated answer. One query (regarding your comment on linear regression in last part of your answer): why do we need to correct for 'repeated measures' if we do regression as `PercentReduction ~ Group`. Percent reduction is only ONE value that we are obtaining from subject, even though to get it we have to test him/her twice. Testing twice is part of experiment, not data. – rnso Jul 09 '20 at 18:10
  • Also, can `Difference` (Postvalue - Prevalue) be used as outcome with `BaselineValue` as a covariate? – rnso Jul 09 '20 at 18:20
  • Good point (in your first comment). In that case it is basically an ANOVA, but using the percent variable may result in non normal residuals which could affect the inference. I will update the answer soon. Analysis of change is a deceptively tricky subject and I am always very wary of composite variables, change scores (ie your second comment), percent changes and other things. [This](https://pubmed.ncbi.nlm.nih.gov/16355543) is an excellent paper that goes into detail about the pitfalls and recommends the mixed effects approach. Change scores are generally not good. – Robert Long Jul 09 '20 at 18:27
  • And further to your send comment the `pre` variable would be the same as baseline, so this would also have mathematical coupling. The paper linked in my last comment talks about this. – Robert Long Jul 09 '20 at 18:30
  • Yes, by BaselineValue I meant Prevalue. I should have used term `Prevalue` in above comment. – rnso Jul 09 '20 at 18:32
  • Ok, so `post - pre` is often called "change score" and there is quite a big literature about when that can be a problem (again see the paper I linked). Even a simple ANCOVA model doesn't work when groups are unbalanced (hence why I asked about how subjects were assigned to the groups in my comment to the question) – Robert Long Jul 09 '20 at 18:42
  • Can you suggest some good Python package for Mixed effects analysis? Also, any package for MLM method as discussed in the paper you linked? – rnso Jul 09 '20 at 18:45
  • `smf` from `statsmodels.formula.api` can fit mixed models in python but the options available in R like `lme4` and `GLMMAdaptive` among many others, are much better. I never fit mixed models in python. – Robert Long Jul 09 '20 at 18:49
  • It will be great if you can add a line of code for proper syntax for mixed effects in R (since you say that that is the best option). Is it: `lme(wt ~ group*time, random=~time|Subject, data=mydata)` ? Also, how to add other variables to be adjusted here, e.g. `age` (continuous) and `gender` (categorical) ? – rnso Jul 10 '20 at 01:12
  • A question to clarify change vs baseline: https://stats.stackexchange.com/questions/478310/is-there-a-mathematical-proof-for-change-being-correlated-with-baseline-value – rnso Jul 22 '20 at 04:35
  • Actually we should not take x2 to be independent of x1. We should take `x1` and `change` to be random in example. Then `change ~ x1 + age + gender ...` can be done and change will not be related to x1. See answer here: https://stats.stackexchange.com/questions/478310/is-there-a-mathematical-proof-for-change-being-correlated-with-baseline-value/ – rnso Jul 22 '20 at 06:12
  • Why shouldn't we take x2 to be independent of x1 ? That is a perfectly valid scenario where there is no actual change but by using change scores you would find a spurious result. Of course where they are not independent is a differenmt scenario, where again you will get spurious results. – Robert Long Jul 22 '20 at 06:18
  • If a child has height `h1` at baseline and we give a height increasing drug. The change in height has to be positive (since height cannot decrease). Moreover, it will be close to initial height. So final height cannot be random. While we may assume the change (increase) to be random in a distribution. Can we test here `change ~ h1 + trt + age + gender`? – rnso Jul 22 '20 at 06:20
  • No you can't. Of course there are examples like that, but you get the wrong answer if you use change scores ! – Robert Long Jul 22 '20 at 06:24
  • Ok. That's why I asked this question: https://stats.stackexchange.com/questions/478310/is-there-a-mathematical-proof-for-change-being-correlated-with-baseline-value – rnso Jul 22 '20 at 06:46
  • No problem. I am going to answer it now. – Robert Long Jul 22 '20 at 06:48
  • I have posted my doubt in another question: https://stats.stackexchange.com/questions/478529/why-we-cannot-take-baseline-as-predictor-for-change-in-this-case – rnso Jul 23 '20 at 06:43
  • I know. But it's the same question. Have you read the paper by [Tu and Gilthorpe] (https://pubmed.ncbi.nlm.nih.gov/16526009) ? – Robert Long Jul 23 '20 at 06:47
  • 1
    I read the paper. This topic has been discussed extensively. – rnso Jul 23 '20 at 07:34