Does "random effect" really exist in real data when we use random/mixed effect model?

Question

If I understand correctly, here is a standard case when we need the mixed effect model:

We are interested in studying the how drugs influence human health conditions, so we collected information about many patients' drug usage, health condition, and other factors such as demographical information to study. We build a linear mixed model:

$$y = X\beta + Zu + \epsilon$$

where $X$ stands for the covariates about drugs, $Z$ stands for other demographical information, $\beta$ stands for fixed effects, $u$ stands for random effects and $u\sim N(0, \sigma_u^2)$, and $\epsilon$ stands for the residue errors.

My question is about the fact that demographical information does not influence health conditions randomly. In reality, it should be a fixed effect. Since there are many different definitions of fixed and random effects, as given by John Salvatier's answer, let me clarify a little bit: In reality, I believe the data should be: $$y=X\beta + Z\alpha + \epsilon$$ where both $\alpha$ and $\beta$ are fixed, so the var$(y)$ should be as simple as $\sigma_\epsilon^2$.

I do not object to the fact of applying mixed models to these data, but my question is that in many statistic papers I read so far, when researchers talk about the consistency or bias/variance of fitting mixed models, it looks like they mostly assume the underlying data distribution to be a mixed effect model, then since random effects may not even exist in the real world, I don't understand the practical value of these analyses.

1. Of course, all these analyses have research values as they spread new findings/ideas/methods, but does it sound like they are analyzing a problem that does not exist?

2. Are there papers that discuss the statistical properties of the mixed model when they believe the underlying true data distributions are all fixed effects?

Answer 1

In some sense, the OP is correct that one could consider all effects to be "fixed." Yes, each individual is likely have a different response to a drug that could in principle be estimated. Differences in intercepts or of slopes in regression models among individuals, or issues of repeated measures within individuals, could be handled as sets of very many fixed effects. Such analyses would, however, tend to be extremely inefficient and could easily become overfit or poorly determined as the number of predictors considered approaches the number of cases.

"The practical value of these [random-effect] analyses" over fixed effects comes from helping us better examine the hypotheses we are most interested in testing.

Each extra level of a fixed effect that is incorporated into a model uses up a degree of freedom in the analysis, which necessarily limits the power of tests of hypotheses about other coefficients in the model. For example, a reviewer of a survival analysis by my research group asked, quite appropriately, to account for possible outcome differences associated with the institutions at which patients had been treated. With about a dozen different institutions, treating each institution as a level of a fixed effect to be ascertained individually would have used up many of our degrees of freedom.

But we (and the reviewer) weren't interested in the specific hazard associated with each institution; rather, we recognized that institutions might have systematic differences in outcomes that needed to be taken into account.

Modeling the effects of institutions as random effects, in contrast, only used up a single degree of freedom, that associated with the variance of outcomes among the whole set of institutions. This allowed for better tests of the associations with outcome of standard clinical characteristics and of the biomarker we were evaluating. That's one of the types of grouping that modeling with random effects allows.

So even if "demographical information does not influence health conditions randomly," incorporating random effects into models allows for better estimation of coefficients associated with particular features of primary interest. Using a "random" effect means that a decision has been made not to consider the particular individuals or institutions at hand as being of primary concern. They are instead taken to be random samples from a population of individuals or of institutions, with estimating and controlling for the overall variability among members of the population being of primary interest. So even if all effects might be considered "fixed effects," there is much to be gained by treating what might be considered "fixed effects" as random effects.