What is the Frequentist definition of fixed effects?

Question

Bolker (2015) writes on p. 313 that

Frequentists and Bayesians define random effects somewhat differently, which affects the way they use them. Frequentists define random effects as categorical variables whose levels are chosen at random from a larger population, e.g., species chosen at random from a list of endemic species. Bayesians define random effects as sets of variables whose parameters are [all] drawn from [the same] distribution. The frequentist definition is philosophically coherent, and you will encounter researchers (including reviewers and supervisors) who insist on it, but it can be practically problematic. For example, it implies that you can’t use species as random effect when you have observed all of the species at your field site—since the list of species is not a sample from a larger population—or use year as a random effect, since researchers rarely run an experiment in randomly sampled years—they usually use either a series of consecutive years, or the haphazard set of years when they could get into the field.

Bolker continues on p 315 to state that

The Bayesian framework has a simpler definition of random effects. Under a Bayesian approach, a fixed effect is one where we estimate each parameter (e.g., the mean for each species within a genus) independently (with independently specified priors), while for a random effect the parameters for each level are modeled as being drawn from a distribution (usually Normal); in standard statistical notation, $\textrm{species_mean} \sim {\cal N}(\textrm{genus_mean}, \sigma^2_{\textrm{species}})$.

Bolker's chapter thus provides clear Bayesian and Frequentist definitions of random effects, and a clear Bayesian definition of fixed effects. However, I don't see that it provides any Frequentist definition of fixed effects.

I am aware from this answer that a wide variety of inconsistent definitions of fixed effects exist in the literature. To clarify, I'm looking for a definition that would "complete the set" of Bolker's existing ones, and be demonstrably consistent with the approach that he is taking.

Although related topics are considered at length in this question, I see this question as different and much more specific. I also don't think the answer is present in any of the responses to the other question.

Bolker, B. M., 2015. Linear and generalized linear mixed models. In G. A. Fox, S. Negrete-Yankelevich, and V. J. Sosa (eds.), Ecological Statistics: Contemporary theory and application. Oxford University Press. ISBN 978-0-19-967255-4. In press.

Answer 1

Trying to find single "authoritative" definition is always tempting in cases like this, but the variety of different definitions shows that this term simply is not used in consistent manner. Andrew Gelman seems to have reached same conclusions, you can look as his blog posts here and here, or into his handbook Data Analysis Using Regression and Multilevel/Hierarchical Models written together with Jennifer Hill, where they write (p. 254-255):

The term fixed effects is used in contrast to random effects—but not in a consistent way! Fixed effects are usually defined as varying coefficients that are not themselves modeled. For example, a classical regression including $J − 1 = 19$ city indicators as regression predictors is sometimes called a “fixed-effects model” or a model with “fixed effects for cities.” Confusingly, however, “fixed-effects models” sometimes refer to regressions in which coefficients do not vary by group (so that they are fixed, not random).

A question that commonly arises is when to use fixed effects (in the sense of varying coefficients that are unmodeled) and when to use random effects. The statistical literature is full of confusing and contradictory advice. Some say that fixed effects are appropriate if group-level coefficients are of interest, and random effects are appropriate if interest lies in the underlying population. Others recommend fixed effects when the groups in the data represent all possible groups, and random effects when the population includes groups not in the data. These two recommendations (and others) can be unhelpful. For example, in the child support example, we are interested in these particular cities and also the country as a whole. The cities are only a sample of cities in the United States—but if we were suddenly given data from all the other cities, we would not want then to change our model.

Our advice (elaborated upon in the rest of this book) is to always use multilevel modeling (“random effects”). Because of the conflicting definitions and advice, we avoid the terms “fixed” and “random” entirely, and focus on the description of the model itself (for example, varying intercepts and constant slopes), with the understanding that batches of coefficients (for example, $\alpha_1, \alpha_2, \dots, \alpha_J$) will themselves be modeled.

This is a good advice.

Answer 2

First of all, the 'random effects' can be viewed in different ways and the approaches to them and associated definitions may seem conflicting but it is just a different viewpoint.

The 'random effect' term in a model can be seen as both a term in the deterministic part of the model as a term in the random part of the model.

Basically, in general, the difference between fixed effect and random effect is whether a parameter is considered fixed within the experiment or not. From that point you get all kinds of different practical applications, and the many varying answers (opinions) to the question "When to use random effects?". It might actually be more a linguistic problem (when something is called random effect or not) than something with a problem with modelling (where we all understand the mathematics in the same way).

---

The Bayesian and Frequentist frameworks look in the same way at a statistical model, say: observations $Y_{ij}$ where $j$ is the observation number and $i$ indicates a grouping

$$Y_{ij} = \underbrace{ \alpha + \beta}_{\substack{\llap{\text{mod}}\rlap{\text{el}} \\ \llap{\text{parame}}\rlap{\text{ters}} }}\overbrace{X_{ij}}^{\substack{\llap{\text{indep}}\rlap{\text{endent}} \\ \text{variables}}} +\overbrace{Z_{i}}^{\substack{\llap{\text{ran}}\rlap{\text{dom}} \\ \text{group}\\ \text{term}}} + \overbrace{\epsilon_{j}}^{\substack{\llap{\text{ran}}\rlap{\text{dom}} \\ \text{individual}\\ \text{term}}}$$

The observations $Y_{ij}$ will depend on some model parameters $\alpha$ and $\beta$, which can be seen as the 'effects' which describe how the $Y_{ij}$ depends on the variable $X_{ij}$.

But the observations will not be deterministic and only depend on $X_{ij}$, there will also be random terms such that the observation conditional on the independent variables $Y_{ij} \vert X_{ij}$ will follow some random distribution. The terms $Z_{i}$ and $\epsilon_j$ are the nondeterministic part of the model.

This is the same for the Bayesian and Frequentist approach, which in principle do not differ in their way to describe a probability for the observations $Y_{ij}$ conditional on the model parameters $\alpha$ and $\beta$ and independent variables $X_{ij}$, where $Z_i$ and $\epsilon_j$ describe a non-deterministic part.

---

The difference is in the approach to 'inference'.

• The Bayesian approach uses reverse probability and describes a probability distribution of the (fixed effect) parameters $\alpha$ and $\beta$. This implies an interpretation of those parameters as random variables. With a Bayesian approach the outcome is a statement about the probability distribution for the fixed effect parameters $\alpha$ and $\beta$.

• A Frequentist method does not consider a distribution of the fixed effect parameters $\alpha$ and $\beta$ and avoids making statements that imply such distribution (but it is not explicitly rejected). The probability/frequency statements in a frequentist approach do not relate to a frequency/probability statement about the parameters but to a frequency/probability statement about the success rate of the estimation procedure.

So if you like, you could say that a frequentist definition of a fixed effect is: 'a model parameter that describes the deterministic part in a statistical model'. (ie. parameters that describe how dependent variables depend on independent variables).

And more specifically in most contexts this relates only to the parameters for the deterministic model that describe $E[Y_{ij} \vert X_{ij}]$. For instance, with a frequentist model one can estimate both the mean and variance, but only the parameters that relate to the mean are considered 'effects'. And even more specifically, the effects are most often used in the context of a 'linear' model. E.g. a for a nonlinear model like $E[y] \sim a e^{-bt}$ the parameters $a$ and $b$ are not really called 'effects'.

---

In a Bayesian framework all effects are sort of random and not deterministic (so the difference between random effect and fixed effect is not so obvious). The model parameters $\alpha$ and $\beta$ are random variables.

How I interpret the question's description/definition of the difference in random effect and fixed effect in the Bayesian framework is more as something pragmatic than as some principle.

• the fixed effects $\alpha$ and $\beta$ are considered to be like "where we estimate each parameter ... independently" (the $\alpha$ and $\beta$ are randomly drawn from a distribution, but they are the same for all $i$ and $j$ within the analysis, e.g. the mean of a species is a model parameter that is considered the same for each species)
• and the random effects are like "for a random effect the parameters for each level are modeled as being drawn from a distribution" (for each observation category $i$ a different random effect is 'drawn' from the distribution, e.g. the mean of a species is a model parameter that is considered different for each species)

In a frequentist framework the fixed effect model parameters are not considered as random parameters, or at least it doesn't matter for the inference whether the parameters are a random parameter or not and it is left out the analysis. However, the random effect term is explicitly considered as a random variable (that is, as a nondeterministic component of the model) and this will influence the analysis (e.g. as in a mixed effects model the imposed structure of the random error term).