61

I see the concept of 'exchangeability' being used in different contexts (e.g., bayesian models) but I have never understood the term very well.

  1. What does this concept mean?

  2. Under what circumstances is this concept invoked and why?

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
sxv
  • 805
  • 1
  • 8
  • 6

1 Answers1

66

Exchangeability is meant to capture symmetry in a problem, symmetry in a sense that does not require independence. Formally, a sequence is exchangeable if its joint probability distribution is a symmetric function of its $n$ arguments. Intuitively it means we can swap around, or reorder, variables in the sequence without changing their joint distribution. For example, every IID (independent, identically distributed) sequence is exchangeable - but not the other way around. Every exchangeable sequence is identically distributed, though.

Imagine a table with a bunch of urns on top, each containing different proportions of red and green balls. We choose an urn at random (according to some prior distribution), and then take a sample (without replacement) from the selected urn.

Note that the reds and greens that we observe are NOT independent. And it is maybe not a surprise to learn that the sequence of reds and greens we observe is an exchangeable sequence. What is maybe surprising is that EVERY exchangeable sequence can be imagined this way, for a suitable choice of urns and prior distribution. (see Diaconis/Freedman (1980) "Finite Exchangeable Sequences", Ann. Prob.).

The concept is invoked in all sorts of places, and it is especially useful in Bayesian contexts because in those settings we have a prior distribution (our knowledge of the distribution of urns on the table) and we have a likelihood running around (a model which loosely represents the sampling procedure from a given, fixed, urn). We observe the sequence of reds and greens (the data) and use that information to update our beliefs about the particular urn in our hand (i.e., our posterior), or more generally, the urns on the table.

Exchangeable random variables are especially wonderful because if we have infinitely many of them then we have tomes of mathematical machinery at our fingertips not the least of which being de Finetti's Theorem; see Wikipedia for an introduction.

  • 12
    (+1) The exchangeability assumption is also at the heart of permutation tests. – chl Oct 12 '10 at 15:53
  • 12
    Given the question of the when & why of exchangeability, chl's pointer to permutation tests may merit a few additional words. Permutation tests are a nonparametric technique used when normality and similar assumptions are untenable - instead one uses the much weaker "null assumption" of exchangeability, approximates the distribution of a test statistic under this null assumption (by permuting) and looks whether the actually observed test statistic is extreme compared to this null distribution. There is an accessible book by P. Good, "Permutation, Parametric, and Bootstrap Tests of Hypotheses". – Stephan Kolassa Oct 12 '10 at 20:43
  • @Stephan I like this book! Still, exchangeability is weaker than independence... – chl Oct 12 '10 at 21:01
  • Thanks very much for this helpful answer, GJK, and for the notes about permutation tests, @StephanKolassa and chi. I'm confused about what role the multiple urns are playing in the example, though. The sequence of reds and greens would be exchangeable with a single, urn, right? What does the possibility of multiple urns (multiple distributions) add? – Mars Jul 29 '15 at 16:31
  • Hmm maybe I'm not clear about the example: Do I choose an urn according to the prior distribution, and then take all of my balls from that urn, or do I choose a (possibly) new urn each time I take a ball? – Mars Jul 29 '15 at 16:37
  • @chl The final sentence of the first paragraph reads, "Every exchangeable sequence is identically distributed, though." Is that really correct? That seems to directly contradict the sentence before it. (Sorry to ping you on this. It's just that you're a moderator and the OP hasn't visited the site in several years... Thanks!) – Josh O'Brien May 25 '16 at 22:25
  • 3
    I would not trust **anything** in that book by Good. I once spotted what I though was an error in that book, wrote him, and received a rather vulgar noanswer. – kjetil b halvorsen Feb 20 '18 at 14:08
  • Is identically sequence exchangeable? – High GPA Jun 02 '19 at 11:48