12

Wikipedia says

effect size is a measure of the strength of a phenomenon or a sample-based estimate of that quantity. An effect size calculated from data is a descriptive statistic that conveys the estimated magnitude of a relationship without making any statement about whether the apparent relationship in the data reflects a true relationship in the population.

To understand it better, I was wondering what descriptive statistics are not effect size, except graphs and plots.

Jeromy Anglim
  • 42,044
  • 23
  • 146
  • 250
Tim
  • 1
  • 29
  • 102
  • 189
  • Graph and plots can actually be great to gauge the size of an effect in more intuitive ways than effect size measures. If you actually *see* the overlap between two groups on some measures (which would roughly correspond to a smaller *d*), it's perhaps easier to realize that a significant difference does not mean that member of one group all have lower scores than members of the other group, etc. – Gala May 16 '13 at 07:09

2 Answers2

18

Effect sizes

  • Common standardised effect sizes typically quantify the amount or degree of a relationship or effect. The most common effect size measures are probably cohen's d, Pearson's r, and the odds ratio (particularly for a binary predictor).
  • Less common effect size measures: That said, you can have standardised and unstandardised effect size measures. Any statistic that communicates the degree of relationships and is not especially contaminated by sample size is probably an effect size measure. Thus, Beta coefficients, R-square, covariance, raw mean differences between groups, and so on all capture the degree of effect. That said, I find that some researchers apply effect size measures somewhat blindly and forget that the broader aim is to give readers a sense of the degree of effect. And thus, they often don't realise that measures like mean differences or raw regression coefficients are in some sense an effect size measure. Another example of blind use of effect sizes involves the use of effect size measures that do not have an intuitive interpretation, but have been recommended by some textbook.

Not effect sizes:

  • Most test statistics are not effect sizes. E.g., Chi-square test, t-test, z-test, F-test. They get larger both as the size of the population effect increases and as sample size increases. In many respects the whole language of effect sizes has been emphasised in recent years because researchers were focussing too much on how large their test statistics were rather than how large their effect sizes were. This is especially important where you have a large sample size when even small effects can be statistically significant.
  • Most univariate statistics are not effect sizes. For most purposes, effect size is concerned about the relationship between at least two variables. Thus, the sample mean, standard deviation, skew, kurtosis, min, max, and so on are not effect size measures.
  • Statistics not pertaining to degree of relationship are not effect size measures. For example, tests of multivariate normality, the eigenvalues of a matrix, and so on generally are not directly aimed at quantifying an effect in the ordinary sense of the word.

Broader considerations

  • Scaling considerations: The utility of a statistic as an effect size measure largely relates to its ability to communicate the size of an effect. Sometimes this is achieved by using familiar standardised measures of effect (e.g., cohen's d). Other times, careful consideration of scaling of the variables can yield an even clearer interpretation of the size of the effect. For example, say I had a study looking at a training program on income levels. I could report that the training program increased income by .2 standard deviations or I could say that the program increased income by $3,500 US dollars. Both are useful; both are effect size measures. The first is standardised (cohen's d), the second is unstandardised (raw group mean differences).
  • Precision in estimating effect sizes: We often extract sample estimates of effect size measures (e.g., cohen's d, pearson's r, etc.). This context can lead to a contrasting of significance testing with effect size measures. Nonetheless, the aim should still be to estimate in a precise and unbiased way, the population effect size. From a frequentist perspective, confidence intervals around effect sizes provide an estimate of precision. From a Bayesian perspective, there are posterior densities on effect sizes. In many cases, care needs to be taken to ensure that you are using an unbiased effect size measure.
Community
  • 1
Jeromy Anglim
  • 42,044
  • 23
  • 146
  • 250
  • 1
    (+1) Nice answer. – chl May 16 '13 at 06:40
  • The third and last points probably explain where the author(s) of the Wikipedia article is coming from. Given the emphasis on psychology, I think the point is not so much to contrast effect size with other descriptive stats but rather with test statistics and p-values (i.e. inferential statistics) and to emphasize that effect size measures say nothing about sampling variability. – Gala May 16 '13 at 07:03
  • Thanks a lot for your nice answer. I have a question though: do you mean confidence interval cannot be used as an effect size measure, because it is directly related to sample size? (by confidence interval, I mean the value that is both added to or subtracted from the prevalence, mean, etc. - not the upper and lower bounds of a CI). – Vic Jun 26 '13 at 06:45
  • 2
    @Vic you can have a confidence interval on an effect size measure, but the confidence interval itself is not the effect size. – Jeromy Anglim Jun 26 '13 at 08:27
  • Uh many thanks dear Jeromy. For all these years I was mistaken. :) – Vic Jun 26 '13 at 19:29
6

First, effect sizes can be used inferentially as well as descriptively. r and ORs are all effect sizes and are certainly all used in inferential stats.

Univariate statistics are usually not effect sizes, although they can be. E.g. If you are comparing ages of men and women who are married to each other, the mean age of men is not an effect size (then the difference of means would be one effect size). But if you want to see if the mean of something is 0, then the mean would be an effect size.

If it measures an effect, it's an effect size!

Peter Flom
  • 94,055
  • 35
  • 143
  • 276
  • I guess that is true @Peter, but effect size is a term that has been more narrowly defined by Cohen: (Mean1-Mean2)/PooledSD. This sounds a bit like is the difference significant, or only statistically significant - the use of common words to define a statistical term. – doug.numbers May 15 '13 at 21:58
  • 2
    Where does Cohen define it that way? If you mean his book on Power Analysis, I think he uses that as the sort of standard to convert other effect sizes to. But every power analysis table in that book (and there are a LOT) uses some effect size (and not all use that one) – Peter Flom May 15 '13 at 22:01
  • 1
    Cohen's d is always the way I understood it. Similar as described https://en.wikipedia.org/wiki/Effect_size. But you are absolutely right, there are a lot of methods described as effect size. – doug.numbers May 15 '13 at 22:09
  • Thanks! What does "effect" mean, in "If it measures an effect, it's an effect size"? Is comparison always involved in "effect"? I am trying to understand it in your examples too. – Tim May 15 '13 at 22:46
  • Now you're getting into philosophy! – Peter Flom May 15 '13 at 22:48
  • I really have no idea. :-) – Tim May 15 '13 at 22:53
  • True, a philosophy. If something exists, then we should be able to measure it. If we can measure two groups, then we need to be able to describe the differences between them. Cohen's d came into being due to the dissatisfaction with the over-reliance on statistical significance. He wanted something that compared the two groups that could be calculated in your head and understood by simple people - like me. – doug.numbers May 15 '13 at 22:54
  • 4
    The t-test and the z-test are not effect sizes. the same effect size will yield substantially different t and z values for different sample sizes. – Jeromy Anglim May 16 '13 at 05:24
  • Dear Peter, you said in your answer that univariate statistics are not effect sizes (possibly since they are not telling us anything about an effect if I understood correctly). I have a question here: http://stats.stackexchange.com/questions/62552/how-to-draw-a-funnel-plot-for-an-observational-meta-analysis-which-concerns-only?noredirect=1#comment119985_62552 – Vic Jun 26 '13 at 06:52
  • I think the prevalence (number of affected persons divided by the sample size) is a univariate statistic which does not indicate any effects. Can I use it as an effect size measure in a funnel plot? Should I use its 95% CI instead (in the funnel plot)? Could you please kindly have a look at my question in that link. Would appreciate so much. :) – Vic Jun 26 '13 at 06:53
  • 1
    @JeromyAnglim is right; +1. I edited my answer – Peter Flom Jun 26 '13 at 12:00