6

Problem:

I have a multiple regression model with categorical/binary ($c_i$) and continuous ($x_i$) variables:

$ v = \alpha + \beta_1 x_1 + \beta_2 x_2 + ... + \gamma_1 c_1 + \gamma_2 c_2 + $

I am interested in the categorical binary variable $c_1$. I want to compute the Cohen D of the two groups defined by $c_1$ controlling for the other variables.

What do I know so far:

There are some formulas on how to convert F to D (for example https://www.campbellcollaboration.org/media/k2/attachments/converting_between_effect_sizes.pdf pg 13) but they assume a ANCOVA model, that is a linear model with one continuous and one categorical variable. The formula uses the r.square of the covariate (the continuous variable) in relation to the dependent variable $v$.

The formula is:

$ d = \sqrt{\frac{(n1+n2)(1-r^2) F}{n1 n2}} $

This is the formula also used in the compute.es R package to convert effect sizes, but the reference they use to justify the formula Borenstein (2009). Effect sizes for continuous data. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta analysis (pp. 279-293) also is explicit on the ANCOVA modeling (one categorical and one continuous variable.

I am not sure that the formula is valid for a multiple variable regression.

Furthermore I assume that the r.square would be of the whole regression without the categorical variable of interest ($c_1$).

Finally, the is the issue of how is the F computed. I assume that it is a Type II ANOVA, but again not 100% sure.

There was at least two similar questions on CV: Can I calculate Cohen's $d$ from multiple regression coefficient? , unanswered, and Cohen's d from regression coefficient? whose answer refer to the residual standard error which I am reasonable sure is not the correct answer to my case.

Thus my questions:

1) Does the formula holds for multiple variables regression?

2) Does the r.squared refer to the regression without the categorical variable of interest?

3) is the F calculated using a Type II ANOVA?

Jeremy Miles
  • 13,917
  • 6
  • 30
  • 64
Jacques Wainer
  • 5,032
  • 1
  • 20
  • 32
  • What do you mean by "hold"? Is there a particular property, such as null hypothesis normality, that you want to utilize? For multiple regression, I believe that $\eta^2$ or $\omega^2$ are commonly cited statistics. I believe they are mentioned on the "Effect size" Wikipedia page. But you should know that in many fields, the use/abuse of effect sizes has been criticized, much like the use of p-values. – deasmhumnha May 27 '18 at 20:27
  • I do not understand the question regarding the null hypothesis. I am not planning on performing significant tests, but to work only with effect sizes. In particular this is a research on Education and it seems to me that Cohen D is the more useful effect size, since it can be compared across experiments (for example https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1750-8606.2008.00061.x ) But I would be interested and thankful on any reference on criticism of effect sizes. – Jacques Wainer May 27 '18 at 23:02
  • Sorry, I misspoke. It's been a while since I was researching about effect size. It wasn't so much criticism as much as emphasis on providing confidence intervals alongside effect size point estimates as a measure of uncertainty. – deasmhumnha May 28 '18 at 00:40

2 Answers2

2

In your regression model, $c_1$ (an unstandardized regression coefficient for a 0/1 indicator variable) is an adjusted mean difference, adjusting for the other variables in the model. As such, you can use $c_1$ as the numerator in the Cohen's $d$ computation, such as: $$ d = \frac{B}{s_{pooled}} ~, $$ where $B$ is the coefficient $c_i$, and $s_{pooled}$ is: $$ s_{pooled} = \sqrt{ \frac{ s_1^2 \left( n_1-1 \right) + s_2^2 \left( n_2-1 \right) } { n_1 + n_2 - 2 } } ~ $$ The subscripts in the above are for the two groups that are part of $c_i$. The challenge is that unless you have the raw data (which you might but someone conducting a meta-analysis might not), these values are typically not reported for a complex regression model. However, $s_{pooled}^2$ is simply the overall variance for $y$, minus the variance attributed to the treatment effect (e.g., $c_i$). Thus, assuming you have both the overall standard deviation for $y$ and the sample sizes for the two groups created by the $c_i$ indicator variable, $s_{pooled$} can be computed as follows:

$$ s_{pooled} = \sqrt{ \frac{s_y^2(N-1) - B^2\left(\frac{n_1n_2}{n_1+n_2}\right) } {N-2}} ~. $$

dbwilson
  • 1,543
  • 7
  • 10
  • dbwilson @dbwilson can you provide me to a reference to this formula, or a hint for the derivation? I have never seen this formulation of s_pooled before. – Jacques Wainer May 29 '18 at 20:27
  • Lipsey and Wilson's (2001) book, Practical Meta-analysis, has a version of this in Appendix B. The formula above is an improvement on what is reported there (simplified but mathematically the same). The logic for this formula is subtracting out the variance associated with the treatment effect. – dbwilson May 29 '18 at 20:30
0

So if you aren't making an statistical arguments, what needs to "hold"? Are you asking if it accurately measures the effect of $c_1$ when controlling for other variables? Probably not, since in its standard formulation Cohen's $d$ is the mean difference normalized by an estimate of the pooled variance. On the other hand, $\eta^2$/$\omega^2$, and their partial versions, are explicitly designed to express the effect size(s) you are looking for. Here is a fairly well cited paper about a generalization that works for many common experimental design structures. This article, however, discusses ways to use pairwise t-scores to calculate Cohen's $d$ while controlling for covariates.

deasmhumnha
  • 849
  • 4
  • 9