Why are there huge differences in the SEs from binomial & linear regression?

Question

I have data from a simple experiments where people put (a fixed number of) balls either to the left or to the right of them (each ball is just the same with regards to consequences of putting them to left or right). The observation / dependent variable is the number of balls put to the right (out of the total amount of balls given to them).

First, I fitted a linear regression trying to explain this sum of balls put to the right based on some predictors. Now a linear regression is obviously not the best choice, since the data is discrete instead of continuous and has natural lower and upper bounds.

Then I fitted a binomial regression to the data (or a GLM with logit link function). The estimates are pretty similar (at least when judging the direction), but there is a huge difference in standard errors on the predictors. In the binomial regression they are about 26 times smaller than in the linear regression.

I was wondering why that is. As outlined, the linear regression is not the best model for the data – but is it so much worse? Or do I violate critical assumptions and my data actually is not generated by a binomial distribution and hence the GLM with binomial link function can not be trusted?

Answer 1

When you use linear regression, you are violating the assumptions of normality and homoscedasticity (of the residuals). Using a linear regression here cannot be trusted.

The standard errors from the logistic regression will be appropriate. In addition, the way the standard errors are calculated differs. Your linear regression almost certainly used ordinary least squares to fit the model. From there, the standard errors are the square roots of the main diagonal of $\ \hat\sigma^2\bf (X'X)^{-1}$. The logistic regression will have been fit through maximum likelihood, and the Wald standard errors are determined from the degree of curvature at the maximum of the likelihood function. If the response distribution was specified as normal, these would be the same, but since the response distribution is binomial there is no reason they should be the same.

Answer 2

$\newcommand{\fractionBallsRight}{\text{fractionBallsRight}} \newcommand{\BallsToRight }{\text{BallsToRight }} \newcommand{\TotalBalls}{\text{TotalBalls}}$ Let us analyse what happens in both cases. First, your data is in a data table (let's call it Data) with columns: $\BallsToRight, \TotalBalls, x_1, x_2, \ldots, x_n$, where the $x$'s are explanatory variables.

Let's add an extra column: $\fractionBallsRight = \frac{\BallsToRight}{\TotalBalls}$.

Linear regression

In the linear regression case you try to estimate $\fractionBallsRight = \sum_i \beta_i x_i + \beta_0 + \epsilon$ where some assumptions must be fullfilled. In R this can be estimated by:

lm(fractionBallsRight ~ x1 + x2 + .. + xn + 1, data=Data)

Logistic regression for grouped data

Probably it is good to first take a look at Interesting Logistic Regression Idea - Problem: Data not currently in 0/1 form. Any solutions?

What is estimated in this case is the equation $\fractionBallsRight = \frac{1}{1+e^{-(\sum_i \beta_i x_i + \beta_0)}}$. Clearly this is a completely different equation than the one with the linear regression, therefore the coefficients and their standard errors will be different.

The 'logistic regression equation' can be 'linearised', after some manipulations you will find that $\log\!\left( \frac{\BallsToRight}{1-\BallsToRight}\right)=\sum_i \beta_i x_i + \beta_0$, which again shows that the equation for logistic regression and the one for linear regression are completely different (the right hand side is linear in both cases, but the dependent variable is completely different), and therefore the estimated coefficients and their standard errors will also be different.

The R-code for the logistic regression for grouped data is

glm(cbind(BallsToRight, TotalBalls-BallsToRight) ~ x1+x2+ ... + xn+1, 
    data=Data, family=binomial)

Note that, as @gung correctly said (+1), the estimation method for the linear model is OLS and for logistic regression it is maximum likelihood.

It depends on the data, but at first glance I would suggest to use the logistic regression variant for the reasons you mentioned (OLS assumptions are violated).