Could logistic regression be used to detect large errors in least squares regression?

Question

I have the following linear model:

$$w^*=\text{arg min}_w\sum_{i=1}^N \bigg(Y_i-\sum_{j=1}^M X_{i,j}\times w_j\bigg)^2$$

Let $T \in N^*$ and $e_i=|Y_i-\sum_{j=1}^M X_{i,j}\times w_j|$.

It's possible using logistic regression to predict which errors will be less than $T$ (i.e., $e_i<T$) and greater or equal with $T$ (i.e., $e_i \ge T$)?

Here is more information to make the question clearer:

$N$ represent the number of observations. My data has the following property: the histogram of errors using multiple linear regression has a Laplace distribution. My data come from digital images represented on 8 bits. The $Y_i$ are current pixels and $X_{ij}$ are neighborhoods pixels. I want to predict which pixels produce errors less than $T$. I want to know what R functions can I use to make a test? $T$ is not very large, it has the values between 1 and 15 in general.

Answer 1

I think this would only work if the logistic regression model had access to relevant covariates that are missing from the OLS model. In such a situation (i.e., model misspecification), there could be regions in which the observed response values diverge strongly from the predicted values and the logistic regression model would have the requisite information to detect them.

Answer 2

You are trying to use logistic regression to find a possible structure to your residuals. Residuals should be unstructured. If logistic regression picks up something the model is misspecified. Great.

Important: logistic regression looks for a very specific structure. Your method works only if residuals are broken in a very specific way.

For this sort of detection use non-linear regressions. Kernel or Additive regression on the squared/log residuals are far more useful.
Even better: they will allow you to model the whole variance. This answers your question of where the large errors really are.

Answer 3

It would be possible in the case where the original model was heteroskedastic and the heteroskedasticity was related to the covariates. For example,

$y_i \sim \text{N}(x_i^T\beta, \sigma^2x_{i,1}^2)$

where the variance of the $i^{th}$ observation is proportional to the square of the first covariate.

One can imagine, in non-Normal regression situations, similar structures that don't require heteroskedasticity per se. However, the default assumptions of regression models in general involve the errors being independent of the regressors, and the variance of the errors being constant.

On the other hand, if doing, for example, Poisson regression, you're out of the linear model world, but since the variance of the "error" is proportional to the mean, it follows that it is related to the covariates, and such a logistic regression would work - although it would convey no information not already conveyed by the results of the Poisson regression, which fully specify the conditional distributions of the $y_i | x_i$. In the generalized linear / additive model framework, where the likelihood is fully specified, the only way you'll be able to add information to the initial regression by using the logistic regression you suggest is if the initial regression has misspecified (usually by ignoring) the structure of the residuals, e.g., ignored the heteroskedasticity in the linear model presented above.

Nonetheless, your suggestion might reveal something about the structure of the residuals in an exploratory analysis. I suspect, though, that effectively discretizing the residuals by $< T$ or $\ge T$ would usually decrease the information content of them more than it would help clarify the analysis - unless it was an outlier analysis, perhaps.

Answer 4

There are circumstances where such a thing might be made to work - assuming you can supply suitable predictors to your logistic regression, but

(i) dichotomizing the spread may be less useful than leaving it continuous

(ii) I think formal hypothesis testing is a bad idea for assessing model assumptions, since it doesn't answer a useful question (imagine, in large samples, a very small trend in spread - it might be highly significant, but really quite unimportant in terms of its impact on the inference in the original model). More important to think about the effect-size (how much impact on our inference?) rather than the significance (is our sample size large enough to detect it?). We almost never satisfy the assumptions exactly, and we don't gain anything from testing what we already know - either we reject, which tells us nothing more than what we knew before, or we don't, which only tells us our sample size was to small to detect what we already know to be the case. Neither tells us how bad the failure of the assumptions might be for us.