1

I am looking for some literature about linear regression and students t-test to cite them in my discussion within my paper. In a nutshell: I would like to argue that I prefer using the results of a regression compared to t-tests of my individual variables. Is this an acceptable argument? Does someone know some paper about this?

My (simplyfied) Problem: I have two groups (group A and group B) solving an assessment to achieve points. Now, I would like to run a linear regression with the achieved points as dependent variable (actually there are more independet variables e.g. gender, age) and group classification as independent variable, instead of using a t-test to compare the means of both groups.

My regression indicates that the membership to be either in group A or B have significant effects on the achieved points. However, the means of groups A and B do not significantly differ from each other...That`s why I am looking for an argument to use the regression.

Continuation from here: Superiority of linear regression compared to students t-test

einervonwenigen
  • 63
  • 5
  • What do you mean that the means of your A and B groups do not differ from each other? // Your previous question mentioned that you have additional explanatory variables. Please elaborate on that. Without that, this question appears to be the same as before. – Dave Oct 24 '20 at 13:24
  • In my experiment participants were adviced to one out of two groups (A and B) and experienced different experimental manipulations. Then they had to run a test to achieve up to 10 points. I expected that participants of group A gain more points. However, the t-test does not show any difference between both groups. As I know, the t-test compares means. – einervonwenigen Oct 24 '20 at 13:29
  • My regression took more variables into account for example gender and age – einervonwenigen Oct 24 '20 at 13:30
  • And what do you hope to get out of the regression? – Dave Oct 24 '20 at 13:31
  • My regression (dependend variable: achieved points) shows a significant coefficient for group classification. Therefore, I expected to get significant differences by running a t-test (comparing means of achieved points, grouped by group classification). – einervonwenigen Oct 24 '20 at 13:34
  • Now I am confused which method may be more valid to make a statement if group classification has a significant impact on the achieved points. – einervonwenigen Oct 24 '20 at 13:35
  • Are any of the coefficients of other independent variables (gender, age etc) also coming out to be significant? – Dayne Oct 24 '20 at 13:43
  • Yes, they are. Interestingly, the group classification coefficient describes not the most fraction of explained variance. – einervonwenigen Oct 24 '20 at 13:48
  • How about F-statistic of the regression? – Dayne Oct 24 '20 at 14:03
  • Sum of Squares: 93.962 df: 4 Mean Square: 23.491 F: 18.319 Sig: >0.001 – einervonwenigen Oct 24 '20 at 14:11
  • Okay. One final question: are the signs of coefficients opposite? As in sign of coefficient of gender or age or both is opposite of that of group? Sorry for not asking everything in one go. – Dayne Oct 24 '20 at 14:38
  • No problem, I am anxious to hear your assessment. Since, I am not sure, if I understood your quesstion correctly I provide my coefficients: (Constant) 4.974 (0.345)*** Age - 0.549 (0.29)* Group classification (A=1, B=0) 0.682 (0.277)** Gender 0.491 (0.276)* Education (Academic=1, other=0) 0.004 (0.00)*** – einervonwenigen Oct 24 '20 at 14:49

2 Answers2

1

Interesting question: Here is one possibility I could think of:

Say you have two groups: $A$ and $B$. Both groups have males and females, females outnumbering males significantly in group $A$ and opposite in group $B$. You give a medicine to only group $A$ and measure effectiveness by some measure $y$.

Now assume a scenario in which the medicine is actually effective (and raises score) but the average score is generally low for females.

Since females greatly outnumber males in group $A$, on average the score should be lower. However, since this group is also getting medicine, the average increases such that both groups have close averages. Hence the t-test could not reject the null of same means.

On the other hand, you run the following regression:

$$y=\beta_0 + \beta_1 D_B + \beta_2 D_F$$

Here, $D_B=1$ if score is from individual in group $B$. Similarly, $D_F=1$ for females.

Interpretation of $\beta_0$ is the average score of all males from group $A$.

Interpretation of $\beta_1$ is how much does the average score changes for group $B$, keeping gender fixed. In our set-up we would rightly expect it to be significant.

Interpretation of $\beta_2$ is how much does the average score changes for for females as compared to males in either group (since no interaction term - the effect of medicine is considered same in both groups). This will also be significant of course, but the coefficient will be opposite in sign.

So this could be one possible set-up that can lead to the results you are getting (since you are getting negative coefficient for age).

Dayne
  • 2,113
  • 1
  • 7
  • 24
0

This main issue here is potential correlations among predictors that are correlated with outcome.

In a linear regression, if you omit a predictor that is correlated both with outcome and with included predictors, you will have an omitted-variable bias in the resulting coefficient estimates. This answer puts the situation pretty clearly.* The magnitude and direction of such biases depend on the specifics of the situation. Provided that you aren't including so many predictors as to be overfitting, a linear regression including all predictors associated with outcome is thus generally the best choice.

It sounds like you might have had random assignments to your two groups. If so, that provides a type of control for predictors that aren't the main object of study (covariates). It's the standard in clinical trials, as it will help control for covariates associated with outcome that you don't even know about. Over the years there has been discussion whether further control for known covariates between treatment groups (often called analysis of covariance or ANCOVA in this context) should be performed in randomized trials. Current FDA draft guidance for studies with continuous outcomes says in part:

Sponsors can use ANCOVA to adjust for differences between treatment groups in relevant baseline variables to improve the power of significance tests and the precision of estimates of treatment effect.

The recommendations do include cautions on things like not adjusting for covariates whose values might be affected by treatment, and pre-specifying the covariate modeling prior to the study.

*Even if an omitted predictor correlated with outcome isn't correlated with the included predictors, leaving it out might increase the variance of the unbiased coefficient estimates.

EdM
  • 57,766
  • 7
  • 66
  • 187