8

histogram of my data

I'm trying to see if variables x and y together or separately significantly affect Q_7 (the histogram for which is above). I've run a Shapiro-Wilk normality test and got the following

shapiro.test(Q_7)
## data:  Q_7
## W = 0.68439, p-value < 2.2e-16

With this distribution, will the following regression work? Or is there another test I should be doing? 

lm(Q_7 ~ x*y)
kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
  • 1
    Look up the Gauss Markov Theorem. – G. Grothendieck Nov 26 '16 at 13:25
  • Try log-transforming `Q_7`. At the moment it is heavily right-skewed. Check the distributions of the predictors as well. – Joe Nov 26 '16 at 13:16
  • 7
    check residuals, not data – Zheyuan Li Nov 26 '16 at 12:52
  • Try with the square root transformation. If you have many zeros the log transformation may not work well. Also, since you are dealing with counts, Poisson negative binomial regression are more natural choices. – utobi Nov 26 '16 at 13:59
  • 1
    What does "non data" mean? – Silverfish Nov 26 '16 at 18:46
  • @SiIverfish I'm hoping that's a stray, leftover from an edit of "non normal data" on the way to "data that isn't normally distributed". If it isn't that then I'm as mystified and curious as you are. – Glen_b Nov 27 '16 at 02:31
  • As you appear to be using R for your analysis, I'd suggest you run the regression and then the standard diagnostic suite. A basic walk-through can be found here: http://data.library.virginia.edu/diagnostic-plots/ – AkselA Nov 27 '16 at 11:07
  • it may be interesting to note that AIC and Cp are in a certain sense robust to the normality assumption: see e.g. https://arxiv.org/pdf/1308.2766.pdf – user795305 Dec 03 '16 at 21:11

2 Answers2

17

A regression analysis assumes that the data is normally distributed conditioned on the variables in the regression model. That is, if this is the regression model: $$y=X\beta+\varepsilon$$ where $X$ is your matrix of regressor variables, $y$ is the (vector of) data to be explained, $\beta$ is a vector of coefficients on the regressors and $\varepsilon$ is random variability (typically considered noise), then the assumption of Normality applies strictly to $\varepsilon$, not to $y$ (edit: well, strictly speaking it applies to the conditional distribution $y|X$ (which is the same as the distribution of $\varepsilon$), but not to the marginal distribution of $y$). In other words, the data should be Normally distributed once the effects of the regressors have been accounted for, but not (necessarily) before.

What you're testing here is the distribution of $y$, where what you want to test is the distribution of $\varepsilon$. Of course you don't know $\varepsilon$, but you can estimate it by running the regression and examining the distrbution of the residuals $\hat\varepsilon=y-X\hat\beta $ (where $\hat\beta$ are the estimated coefficents from the regression). These residuals $\hat\varepsilon$ are an estimate of $\varepsilon$, and so their distribution will be an approximation of the distribution of $\varepsilon$.

Ruben van Bergen
  • 6,511
  • 1
  • 20
  • 38
  • This is a good brisk summary of standard stuff but seems to miss a key feature of this question, which is that with a skewed but non-zero response this functional form is unlikely to be a good idea. To avoid negative predictions, and on other grounds, Poisson regression seems a better starting point. – Nick Cox Aug 22 '17 at 12:00
9

The short answer is yes.

First of all (as is pointed out by Ruben van Bergen), the distribution of $y$ (or $X$, for that matter) is not relevant. If you were to make a distributional assumption, it would be on your residuals $\varepsilon$, so that is what you should check.

But more importantly, you don't need the normality assumption at all for your estimation to work. You are using R's lm function, which estimates your model using ordinary least squares (OLS). That method will give you a correct estimation of the expectation of $Y$ conditional on $X$ as long as:

  • $\mathbb{E}[\varepsilon|X] = 0$ (there is no external factor affecting both your outcome and your explanatory variables).
  • $\mathrm{Var}(\varepsilon) < \infty$ (your residuals have finite variance).

If you further make the assumption that you residuals are uncorrelated and that they all have the same variance, then the Gauss-Markov theorem applies and the OLS is the best linear unbiased estimator (BLUE).

If your residuals are correlated or have different variances, then OLS still works but it can be less precise, which must be reflected in the way you report the confidence intervals of your estimates (using, say robust standard errors).

If you also make the assumption that your residuals are normally distributed, then OLS becomes asymptotically efficient because it is equivalent to maximum likelihood.

So the regression may work better if your data are normally distributed, but it will still work if they aren't.

thomas
  • 173
  • 1
  • 5