What is effect size... and why is it even useful?

Question

I have an introductory-graduate-level statistics background (assume I know mathematical statistics and probability at an undergraduate level (e.g., Wackerly et al., Ross' Probability), and have some knowledge of measure theory).

I have recently started a job doing experimental design and statistical reporting in education statistics, and have been placed on a project where I am basically assessing accountability metrics for schools and have to analyze the data, propose changes, etc. Note that I am the only one in my department with a mathematical statistics background.

In my position, people have strongly suggested using effect size to measure effectiveness of programs. The only time I've ever heard of effect size is from my friend, who studied psychology. My impression is that $$\text{Effect Size} = \dfrac{\text{Difference of Means}}{\text{Standard Deviation}}\text{.}$$

What is so useful about this metric over traditional hypothesis testing, and why should I care about it? To me, it looks like nothing more than a test statistic for a two-sample $t$-test. I don't see this useful at all apart from maybe putting everything on the same scale (which is why anyone really "normalizes" anything), but I thought test statistics (which is what effect size seems like to me) were out of fashion, and $p$-values are preferred.

Answer 1

That is one measure of effect size, but there are many others. It is certainly not the $t$ test statistic. Your measure of effect size is often called Cohen's $d$ (strictly speaking that is correct only if the SD is estimated via MLE—i.e., without Bessel's correction); more generically, it is called the 'standardized mean difference'. Perhaps this will make it clearer that $t\ne d$:
\begin{align} d &= \frac{\bar x_2 - \bar x_1}{SD} \\[10pt] &\ne \\[10pt] t &= \frac{\bar x_2 - \bar x_1}{SE} \\[10pt] t &= \frac{\bar x_2 - \bar x_1}{\frac{SD}{\sqrt N}} \\ \end{align} That is, the "$/\sqrt N$" is missing from the formula for the standardized mean difference.

More generally, taking the sample size out of the value provides real information. Assuming the true effect is not exactly $0$ to infinite decimal places, you can achieve any level of significance you might like with sufficient $N$. The $p$-value provides information about how confident we can be in rejecting the null hypothesis, but does so by conflating how big the effect is with how much data you have. It is certainly nice to know if we should reject the null hypothesis, but it would also be nice to know if the effect of your educational intervention produces large gains for schoolchildren or is trivial and was only significant due to large $N$.

Answer 2

I expect someone with a background in a more relevant area (psychology or education, say) will chime in with a better answer, but I'll give it a shot.

"Effect size" is a term with more than one meaning -- which many years past led some some confused conversations until I eventually came to that realization. Here we're clearly dealing with the scaled-for-standard-deviation version ("how many standard deviations did that change by?")

Part of the reason for looking at that sort of "effect size" in the subject areas they're common in is that they frequently have variables whose particular values are not inherently meaningful but are constructed to attempt to measure some underlying thing that's hard to get at.

For example, imagine you are trying to measure job satisfaction (perhaps for a model which relates it to some set of independent variables, perhaps including some treatment of interest, for example). You don't have any way to get at it directly, but you could (for example) try to construct some questionnaire to get at different aspects of it, perhaps using something like a Likert scale.

A different researcher may have a different approach to measuring job satisfaction, and so your two sets of "Satisfaction" measurements are not directly comparable -- but if they have the various forms of validity and so on that these things are checked for (so that they may reasonably be measuring satisfaction), then they may be hoped to have very similar effect sizes; at the least effect size is going to be more nearly comparable.

Answer 3

The formula above is how you calculate Cohen's d for related samples (which is probably what you have?), if they're unrelated you can use pooled variance instead. There are different stats that will tell you about effect size, but Cohen's d is a standardised measure that can vary between 0 and 3. If you have lots of different variables, it can be nice to have a standardised measure when you're thinking about them all together. On the other hand, many people prefer understanding the effect size in terms of the units being measured. Why calculate d when you already have p values? Here's an example from a dataset I'm currently working with. I am looking at a behavioural intervention conducted in schools, measured using validated psychological questionnaires (producing Likert data). Almost all of my variables show statistically significant change, perhaps unsurprising as I have a large sample (n=~250). However, for some of the variables, the Cohen's d is quite miniscule, say 0.12 which indicates that although there is certainly change, it might not be a clinically important change and so it is important to the discussion and interpretation of what's going on in the data. This concept is widely used in psychology and health sciences where the practitioners (or schools, in your case) need to consider the actual clinical utility of treatments (or whatever they're experimenting with). Cohen's d helps us answer questions about whether its really worth doing an intervention (regardless of p values). In medical sciences they also like to consider the NNT, and evaluate this in terms of the severity of the condition in question. Have a look at this great resource from @krstoffr http://rpsychologist.com/d3/cohend/

Answer 4

In fact, p-values are now finally 'out of fashion' as well: http://www.nature.com/news/psychology-journal-bans-p-values-1.17001. Null hypothesis significance testing (NHST) produces little more than a description of your sample size.(*) Any experimental intervention will have some effect, which is to say that the simple null hypothesis of 'no effect' is always false in a strict sense. Therefore, a 'non-significant' test simply means that your sample size wasn't big enough; a 'significant' test means you collected enough data to 'find' something.

The 'effect size' represents an attempt to remedy this, by introducing a measure on the natural scale of the problem. In medicine, where treatments always have some effect (even if it's a placebo effect), the notion of a 'clinically meaningful effect' is introduced to guard against the 50% prior probability that a 'treatment' will be found to have 'a (statistically) significant positive effect' (however minuscule) in an arbitrarily large study.

If I understand the nature of your work, Clarinetist, then at the end of the day, its legitimate aim is to inform actions/interventions that improve education in the schools under your purview. Thus, your setting is a decision-theoretic one, and Bayesian methods are the most appropriate (and uniquely coherent[1]) approach.

Indeed, the best way to understand frequentist methods is as approximations to Bayesian methods. The estimated effect size can be understood as aiming at a measure of centrality for the Bayesian posterior distribution, while the p-value can be understood as aiming to measure one tail of that posterior. Thus, together these two quantities contain some rough gist of the Bayesian posterior that constitutes the natural input to a decision-theoretic outlook on your problem. (Alternatively, a frequentist confidence interval on the effect size can be understood likewise as a wannabe credible interval.)

In the fields of psychology and education, Bayesian methods are actually quite popular. One reason for this is that it is easy to install 'constructs' into Bayesian models, as latent variables. You might like to check out 'the puppy book' by John K. Kruschke, a psychologist. In education (where you have students nested in classrooms, nested in schools, nested in districts, ...), hierarchical modeling is unavoidable. And Bayesian models are great for hierarchical modeling, too. On this account, you might like to check out Gelman & Hill [2].

[1]: Robert, Christian P. The Bayesian Choice: From Decision-Theoretic Foundations to Computational Implementation. 2nd ed. Springer Texts in Statistics. New York: Springer, 2007.

[2]: Gelman, Andrew, and Jennifer Hill. Data Analysis Using Regression and Multilevel/hierarchical Models. Analytical Methods for Social Research. Cambridge ; New York: Cambridge University Press, 2007.

---

For more on 'coherence' from a not-necessarily-beating-you-on-the-head-with-a-Bayesian-brick perspective, see [3].

[3]: Robins, James, and Larry Wasserman. “Conditioning, Likelihood, and Coherence: A Review of Some Foundational Concepts.” Journal of the American Statistical Association 95, no. 452 (December 1, 2000): 1340–46. doi:10.1080/01621459.2000.10474344.

(*) In [4], Meehl scourges NHST far more elegantly, but no less abrasively, than I do:

Since the null hypothesis is quasi-always false, tables summarizing research in terms of patterns of “significant differences” are little more than complex, causally uninterpretable outcomes of statistical power functions.

[4]: Meehl, Paul E. “Theoretical Risks and Tabular Asterisks: Sir Karl, Sir Ronald, and the Slow Progress of Soft Psychology.” Journal of Consulting and Clinical Psychiatry 46 (1978): 806–34. http://www3.nd.edu/~ghaeffel/Meehl(1978).pdf

---

And here's a related quote from Tukey: https://stats.stackexchange.com/a/728/41404

Answer 5

What you wrote is not a test statistic. It's a measure used to define how different the two means are. Generally, effect sizes are used to quantify how far from the null hypotheses the something is. For example, if you are doing power analysis for the two sample $t$-test, you might quantify the power as a function of the effect size (for a fixed $n$) you just wrote (which, I think, is called Cohen's D). In other contexts, the effect size might be something else.

It is also not uncommon to report effect sizes using sample quantities, which may coincide with some familiar statistics, such as the pearson correlation - the true effect size is the underlying correlation coefficient that generated the data, but the sample correlation is also useful information to have sometimes. The purpose to quantify how far from the null hypothesis the observed data are, in one way or another, rather than just reporting a $p$-value and calling it a day.