What precisely does it mean to borrow information?

Question

I often people them talk about information borrowing or information sharing in Bayesian hierarchical models. I can't seem to get a straight answer about what this actually means and if it is unique to Bayesian hierarchical models. I sort of get the idea: some levels in your hierarchy share a common parameter. I have no idea how this translates to "information borrowing" though.

1. Is "information borrowing"/ "information sharing" a buzz word people like to throw out?

2. Is there an example with closed form posteriors that illustrates this sharing phenomenon?

3. Is this unique to a Bayesian analysis? Generally, when I see examples of "information borrowing" they are just mixed models. Maybe I learned this models in an old fashioned way, but I don't see any sharing.

I am not interested in starting a philosophical debate about methods. I am just curious about the use of this term.

Answer 1

This is a term that is specifically from empirical Bayes (EB), in fact the concept that it refers to does not exist in true Bayesian inference. The original term was "borrowing strength", which was coined by John Tukey back in the 1960s and popularized further by Bradley Efron and Carl Morris in a series of statistical articles on Stein's paradox and parametric EB in the 1970s and 1980s. Many people now use "information borrowing" or "information sharing" as synonyms for the same concept. The reason why you may hear it in the context of mixed models is that the most common analyses for mixed models have an EB interpretation.

EB has many applications and applies to many statistical models, but the context always is that you have a large number of (possibly independent) cases and you are trying to estimate a particular parameter (such as the mean or variance) in each case. In Bayesian inference, you make posterior inferences about the parameter based on both the observed data for each case and the prior distribution for that parameter. In EB inference the prior distribution for the parameter is estimated from the whole collection of data cases, after which inference proceeds as for Bayesian inference. Hence, when you estimate the parameter for particular case, you are use both the data for that case and also the estimated prior distribution, and the latter represents the "information" or "strength" that you borrow from the whole ensemble of cases when making inference about one particular case.

Now you can see why EB has "borrowing" but true Bayes does not. In true Bayes, the prior distribution already exists and so doesn't need to be begged or borrowed. In EB, the prior distribution has be created from the observed data itself. When we make inference about a particular case, we use all the observed information from that case and a little bit of information from each of the other cases. We say it is only "borrowed", because the information is given back when we move on to make inference about the next case.

The idea of EB and "information borrowing" is used heavily in statistical genomics, when each "case" is usually a gene or a genomic feature (Smyth, 2004; Phipson et al, 2016).

References

Efron, Bradley, and Carl Morris. Stein's paradox in statistics. Scientific American 236, no. 5 (1977): 119-127. http://statweb.stanford.edu/~ckirby/brad/other/Article1977.pdf

Smyth, G. K. (2004). Linear models and empirical Bayes methods for assessing differential expression in microarray experiments. Statistical Applications in Genetics and Molecular Biology Volume 3, Issue 1, Article 3. http://www.statsci.org/smyth/pubs/ebayes.pdf

Phipson, B, Lee, S, Majewski, IJ, Alexander, WS, and Smyth, GK (2016). Robust hyperparameter estimation protects against hypervariable genes and improves power to detect differential expression. Annals of Applied Statistics 10, 946-963. http://dx.doi.org/10.1214/16-AOAS920

Answer 2

Consider a simple problem like estimating means of multiple groups. If your model treats them as completely unrelated then the only information you have about each mean is the information within that group. If your model treats their means as somewhat related (such as in some mixed-effects type model) then the estimates will be more precise because information from other groups informs (regularizes, shrinks toward a common mean) the estimate for a given group. That's an example of 'borrowing information'.

The notion crops up in actuarial work related to credibility (not necessarily with that specific term of 'borrowing' though borrowing in that sense is explicit in the formulas); this goes back a long way, to at least a century ago, with clear precursors going back to the mid-nineteenth century. For example, see Longley-Cook, L.H. (1962) An introduction to credibility theory PCAS, 49, 194-221.

Here's Whitney, 1918 (The Theory of Experience Rating, PCAS, 4, 274-292):

Here is a risk, for instance, that is clearly to be classified as a machine shop. In the absence of other information it should therefore fake the machine shop rate, namely, the average rate for all risks of this class. On the other hand the risk has had an experience of its own. If the risk is large, this may be a better guide to its hazard than the class-experience. In any event, whether the risk is large or small, both of these elements have their value as evidence, and both must be taken into account. The difficulty arises from the fact that in general the evidence is contradictory; the problem therefore is to find and apply a criterion which will give each its proper weight.

While the term borrowing is absent here the notion of using the group-level information to inform us about this machine shop is clearly there. [The notions remain unchanged when "borrowing strength" and "borrowing information" start to be applied to this situation]

Answer 3

The most commonly known model that "borrows information" is that of a mixed effects model. This can be analyzed in either the Frequentist or Bayesian setting. The Frequentist method actually has an Empirical Bayes interpretation to it; there's a prior on the mixed effects which, based on $\sigma_R^2$, the variance of the random effects. Rather than setting based on prior information, we estimate it from our data.

On the other hand, from the Bayesian perspective, we are not putting a prior on the mixed effects, but rather they are a mid level parameter. That is, we put a prior on $\sigma_R^2$, which then acts as like a hyper-parameter for the random effects, but it is different than a traditional prior in that the distribution placed on the random effects is not based purely on prior information, but rather a mix of prior information (i.e., prior on $\sigma_R^2$) and the data.

I think it's pretty clear that "borrowing information" is not something purely Bayesian; there are non-Bayesian mixed effects models and these borrow information. However, based on my experience playing around with mixed effects models, I think Bayesian approach to such models is a little more important than some people realize. In particular, in a mixed effect model, one should think that we are estimating $\sigma_R^2$ with, at best, the number of individual subjects we have. So if we have 10 subjects measured 100 times, we are still estimating $\sigma_R^2$ from only 10 subjects. Not only that, but we don't actually even observe the random effects directly, but rather we just have estimates of them that are derived from the data and $\sigma_R$ themselves. So it can be easy to forget just how little information based on the data we actually have to estimate $\sigma_R^2$. The less information in the data, the more important the prior information becomes. If you haven't done so yet, I suggest trying to simulate mixed effects models with only a few subjects. You might be surprised just how unstable the estimates from Frequentist methods are, especially when you add just one or two outliers...and how often does one see real datasets without outliers? I believe this issue is covered in Bayesian Data Analysis by Gelman et al, but sadly I don't think its publicly available so no hyperlink.

Finally, multilevel modeling is not just mixed effects, although they are the most common. Any model in which parameters are influenced not just by priors and data, but also other unknown parameters can be called a multilevel model. Of course, this is a very flexible set of models, but can written up from scratch and fit with a minimal amount of work using tools like Stan, NIMBLE, JAGS, etc. To this extent, I'm not sure I would say multilevel modeling is "hype"; basically, you can write up any model that can be represented as a Directed Acyclic Graph and fit it immediately (assuming it has a reasonable run time, that is). This gives a whole lot more power and potential creativity than traditional choices (i.e., regression model packages) yet does not require one to build an entire R package from scratch just to fit a new type of model.

Answer 4

I am assuming, since you tagged machine learning that you are interested in prediction, rather than inference.(I believe I am aligned with @Glen_b 's answer, but just translating to this context/vocabulary)

I would claim in this case it is a buzzword. A regularised linear model with a group variable will borrow information: the prediction at individual level will be a combination of the group mean and individual effect. One way to think of l1/l2 regularisation is that it is assigning a coefficient cost per reduction in total error, since a group variable affects more samples than an individual variable, there will be pressure to estimate a group effect,leaving a smaller deviation from group effect to each individual variable.

For individual points with enough data, the individual effect will be 'strong ', for those with little data, the effect will be weak.

I think the easiest way to see this is by considering L1 regularisation and 3 individuals of same group with same effect. Unregularised, the problem has an infinite numbers of solutions, whereas regularisation gives a unique solution.

Assigning all the effect to the group coefficient has the lowest l1 norm, since we only need 1 value to cover 3 individuals. Conversely,assigning all the effect to the individual coefficients has the worst, namely 3 times the l1 norm of assigning the effect to the group coefficient.

Note we can have as many hierarchies as we want, and interactions are affected similarly: regularisation will push effects to main variables,rather than rarer interactions.

The blog tjmahr.com/plotting-partial-pooling-in-mixed-effects-models. – linked by @IsabellaGhement gives a quote for borrowing strength

"This effect is sometimes called shrinkage, because more extreme values shrinkage are pulled towards a more reasonable, more average value. In the lme4 book, Douglas Bates provides an alternative to shrinkage [name]"

The term “shrinkage” may have negative connotations. John Tukey preferred to refer to the process as the estimates for individual subjects “borrowing strength” from each other. This is a fundamental difference in the models underlying mixed-effects models versus strictly fixed effects models. In a mixed-effects model we assume that the levels of a grouping factor are a selection from a population and, as a result, can be expected to share characteristics to some degree. Consequently, the predictions from a mixed-effects model are attenuated relative to those from strictly fixed-effects models.

Answer 5

Another source I would like to recommend on this topic which I find particularly instructive is David Robinson's Introduction to Empirical Bayes.

His running example is that of whether a baseball player will manage to hit the next ball thrown at him. The key idea is that if a player has been around for years, one has a pretty clear picture of how capable he is and in particular, one can use his observed batting average as a pretty good estimate of the success probability in the next pitch.

Conversely, a player who has just started playing in a league hasn't revealed much of his actual talent yet. So it seems like a wise choice to adjust the estimate of his success probability towards some overall mean if he has been particularly successful or unsuccessful in his first few games, as that likely is, at least to some extent, due to good or bad luck.

As a minor point, the term "borrowing" certainly does not seem to be used in the sense that something that has been borrowed would need to be returned at some point ;-).