10

The effect-size tag has no wiki. The wikipedia page about the effect size does not provide a precise general definition. And I have never seen a general definition of the effect size. However when reading some discussions such as this one I am under the impression that people have in mind a general notion of effect size, in the context of statistical tests. I have already seen that the standardized mean $\theta=\mu/\sigma$ is termed as the effect size for a normal model ${\cal N}(\mu,\sigma^2)$ as well as the standardized mean difference $\theta=(\mu_1-\mu_2)/\sigma$ for a "two Gaussian means" model. But how about a general definition ? The interesting property shared by the two examples above is that, as far as I can see, the power depends on the parameters only through $\theta$ and is an increasing function of $|\theta|$ when we consider the usual tests for $H_0:\{\mu=0\}$ in the first case and $H_0:\{\mu_1=\mu_2\}$ in the second case.

Is this property the underlying idea behind the notion of effect size ? That would mean that the effect size is defined up to a monotone one-to-one transformation ? Or is there a more precise general definition ?

Community
  • 1
Stéphane Laurent
  • 17,425
  • 5
  • 59
  • 101
  • +1, great question. One way to think about effect size is that p-values simultaneously measure magnitude & N, so ES is p decoupled from N (this is, of course, only quite loose, though). – gung - Reinstate Monica May 20 '13 at 21:44
  • Effect size is only easy to pin down in some specific cases. With a two-sample test of means, the notion of effect size is straightforward. But add in a third sample and it becomes less clear (if you do ANOVA, you can write it in terms of variance, though). For some tests, it just boils down to nothing more clear than "whatever this test statistic measures". – Glen_b May 20 '13 at 22:33
  • great question too! +1 – Tim May 21 '13 at 02:00
  • @Glen_b For any Gaussian linear model the power of a $F$-test is an increasing function of the noncentrality parameter (see the second part of my answer here http://stats.stackexchange.com/a/59428/8402). It is someting like $(\sum \alpha_i^2)/\sigma^2$ for ANOVA. – Stéphane Laurent May 21 '13 at 05:26
  • Yes, I was trying to give a nontechnical sense of what the noncentrality parameter represents in terms of the original problem - relating it to a variance is an oversimplification but at least with of ANOVA we're used to multiple measures of variation (such as between means as well as around means). It's not much good saying "the effect size is monotonic in the noncentrality parameter" if the person reading a tag wiki won't follow. Your question would need to guide us where you want the comments pitched, if you don't want fairly basic responses (which is normally where I'd aim) – Glen_b May 21 '13 at 05:32
  • So let me rephrase: "Effect size is only easy to pin down in a non-technical way in some specific cases" – Glen_b May 21 '13 at 05:35
  • 1
    @Glen_b I have nothing against basic answers! Any comment is welcomed. Thanks. – Stéphane Laurent May 21 '13 at 07:44

1 Answers1

5

I don't think there can be a general and precise answer. There can be general answers that are loose, and specific answers that are precise.

Most generally (and most loosely) an effect size is a statistical measure of how big some relationship or difference is.

In regression type problems, one type of effect size is a measure of how much of the dependent variable's variance is accounted for by the model. But, this is only precisely answerable (AFAIK) in OLS regression - by $R^2$. There are "pseudo-$R^2$" measures for other regression. There are also effect size measures for individual independent variables - these are the parameter estimates (and transformations of them).

In a t-test, a good effect size is the standardized difference of the means (this also works in ANOVA, and may work in regression if we pick particular values of the independent vairables)

and so on.

There are whole books on the subject; I used to have one, I believe that Ellis is an updated version of it (the title sounds familiar)

Peter Flom
  • 94,055
  • 35
  • 143
  • 276
  • Hello Peter. Why do you say that the standardized difference $\theta$ is a *good* choice for the $t$-test ? Is it because of the property I pointed out: the power depends on the parameters $\mu_1$, $\mu_2$, $\sigma$ only through $\theta$ and is an increasing function of $|\theta|$. – Stéphane Laurent May 21 '13 at 05:29
  • Hi @StéphaneLaurent , yes, that is a more formal way of putting it. Or, you could say that it gets bigger as the difference gets bigger, but is not affected by scaling. – Peter Flom May 21 '13 at 10:05