What does it mean (intuitively) to hold other variables constant in regression?

Question

I am looking for both a 1) mechanical and 2) intuitive explanation for how the effects of individual variables are determined holding other variables constant.

In an example using survey data, what exactly does it mean to say:

"holding constant age, sex, and income, the effect of education is ___"

My understanding is that with regression we are attempting to recreate the experimental setting, and in the example above are trying to compare sub-populations with equal age, sex, income, etc., but with differing levels of education, and estimating the difference in mean of those subpopulations. Questions:

1. Is this intuition correct?
2. Do these subpopulations necessarily exist? What if the survey does not contain respondents with exactly the same values on the controls?
3. How is uncertainty about the estimates of these subpopulations determined?

Answer 1

Intuition is a tricky subject, it depends on the person's background. For instance, I studied statistics after studying mathematical physics. For me the intuition is in partial derivatives. Consider a regression model $$y_i=a+b_x x_i+b_z z_i+\varepsilon_i$$ It can be restated as $$y_i=f(x_i,z_i)+\varepsilon_i,$$ where $f(x,z)=b_x x + b_z z$

Take a total derivative of the function $f()$: $$df=\frac{\partial f}{\partial x}dx+\frac{\partial f}{\partial z}dz$$

This is how partial derivative wrt $x$ is defined: $$\frac{\partial f}{\partial x}=\lim_{\Delta x\to 0} \frac{f(x+\Delta x,z)-f(x,z)}{\Delta x}$$ You hold $z$ constant, and step away from $x$. The partial derivative tells you haw sensitive is $f$ to a change in $x$. You can see that the beta (coefficient) is the slope on the variable of interest: $$\frac{\partial f}{\partial x}=b_x$$

In other words, in the simple linear model your coefficients are partial derivatives (slopes) with regards to the variables. That's what "holding constant" means to me intuitively.

Answer 2

As user122677 answered, the intuition is right: In linear regression every coefficient is the amount of change in the outcome when one variable value is increased by a unit while all other variables remain constant. In other words, coefficients are partial derivatives of model prediction respect to each variable.

Anyway, beware that if our model includes interactions variables can't be changed without changing the interaction and therefore this interpretation of one coefficient can't make sense as a real change. The same happens with polynomial regression, where no term can change without changing other terms.

About existence of those subpopulations, they don't need to exist. In some experimental designs they can exist, but in observational studies with continuous variables they are very unlikely to exist. For example:

• In complete designs of experiments with binary (or discrete finite) variables all combination of values of variables is in the sample.
• In observational studies with continuous variables each observation is very likely to get unique values for all variables and therefore it's not likely to exist two elements with all variables equal except for one.

Answer 3

1. The intuition is correct at its basis. I'll try to answer in brief and intuitive way as well-
2. Those sub populations necessarily exists because you hold them constant by: (a) sampling your subjects with regard to your speculated covariates OR (b) you put a constraint on its variability (i.e. variance = 0). This is done by taking 1 group (e.g. men only, blonds only, etc.) if its categorical variable or by taking an average of a given covariate (age, education, income and so on).