regression assumptions and OLS assumptions, are they different?

Question

Question

1. Is there difference between assumptions of regression and assumptions of Ordinary Least Squares (OLS)?
2. To apply OLS (the only regression method that I can use) on data, should my data and OLS meet both assumptions (assumptions of regression and assumptions of OLS)?
3. Since 'non-linear' regression is also one kind of a regression, if it meets the assumptions of regression, can I apply non-linear regression? No matter whether it is a linear regression or non-linear regression, if the "assumptions of regression" are met, can I use either of them? Or Should there be another set of assumptions for non-linear regression?

More Details

• I am studying linear regression, covering Ordinary Least Squares(OLS) so far.

• I have learned the assumptions of regressions which should not be violated. The number of assumptions varies from book to book, but let me refer to that of the popular statistics youtuber "zedstatistics". The 6 assumptions of regression are as follows:

  1. Linearity
  2. Constant Error Variance
  3. Independent Error Terms
  4. Normal Errors
  5. No multi-collinearity between predictors
  6. Exogeneity

• I thought if my data and model meets the six assumptions mentioned above, I could apply OLS on the data freely. However, I found on the Internet "assumptions of OLS" on the famous book website 'Econometrics with R'

OLS performs well under a quite broad variety of different circumstances. However, there are some assumptions which need to be satisfied in order to ensure that the estimates are normally distributed in large samples.

1. The Error Term has Conditional Mean of Zero
2. Independently and Identically Distributed Data
3. Large Outliers are Unlikely

• Now I am puzzled because it seems like there are two streams of assumptions to be met, in order to use OLS. The 1st assumption is the 'assumptions of regression (the 6 assumptions)' and we also need to check if the 2nd assumption (the 3 assumptions mentioned right above) is met as well.
• Am I correct? Or are the two streams of assumptions saying the same thing, but expressed in a different way?

Answer 1

The OLS assumptions are incomplete, since they are missing the constant error variance and lack of multi-collinearity.

Moreover, the assumptions of OLS follow from the six assumptions listed by zedstatistics:

1. Error term with conditional mean of zero, means the same as exogenity
2. IID follows from the independent errors (independence) and from the normal errors (identically distributed).
3. Large outliers are unlikely, follows from the normally distributed errors, as the tails of the normal distribution decay quickly.

Answer 2

The assumptions for the regression model are model assumptions to specify the behaviour in the model. By contrast, use of the OLS estimator is based on specification of the estimation method (which is not so much an assumption as a decision of how you want to do your estimation). You can either directly specify that you want to use OLS as your estimation method, or you can specify use of the MLE and this reduces down to OLS in the Gaussian regression model, or you can specify another estimator entirely.

Answer 3

There are the following 5 standard assumptions:

• A0 (Linearity): The data generating process is given by $y_i = x_i\cdot\beta + u$.

Here $y$ is the dependent variable and $x$ is a vector of $k$ explanatory variables. The parameter $\beta$ determines the effect of the explanatory variables on the dependent variable. Deviations from this relationship are captured by the error term $u$.

• A1 (No perfect multikolinearity): None of the $k$ explanatory variable is a perfect linear combination of the other.

This assumption ensures that the OLSE exists. If one variable was a perfect linear combination of the other (i.e. we can write one variable $a$ as $a = \lambda b + \delta c$, where $\lambda$ and $\delta$ are real numbers and $b$ and $c$ are other explanatory variables), it would be impossible for a linear estimator to distinguish between the three variables. Heuristically, the variables $a,b$ and $c$ would have to be estimated but since the variables are a perfect linear combination, there are only two independent equations. That is, we would have to solve for three variables with only two equations.

• A2 (Mean independence): The conditional mean of $u$ given the explanatory variables $x_1, x_2, \dots, x_n$ in the sample is zero, i.e. $\mathbb E[u\mid x_1, x_2, \dots, x_n] = 0$.

This assumption ensures that the (conditional and unconditional!) mean of the OLSE is given by $\beta$, i.e. the OLSE is (conditionally and unconditionally) unbiased. Consequently, the average effect of the explanatory variables is given by $\hat\beta$.

• A3 (Constant variance): The conditional mean of $u^2$ given the explanatory variables $x_1, x_2, \dots, x_n$ in the sample is a non-zero finite constant, i.e. $\mathbb E[u^2\mid x_1, x_2, \dots, x_n] = \sigma^2$ for some finite $\sigma > 0$.

This assumption ensures that the variance of the OLSE does not vary with the explanatory variables and that $\sigma^2$ can be easily estimated with the usual variance estimator $(n-k)^{-1}\sum_{i=1}^n(y_i - x_i\cdot\hat\beta)^2$, provided that assumption A2 holds. Furthermore, the (conditional and unconditional!) variance of the OLSE is then given by $\sigma^2(\sum_{i=1}^n x_ix_i')^{-1}$

• A4 (Normality): The conditional distribution of $u$ given the explanatory variables $x_1, x_2, \dots, x_n$ in the sample is normal with mean $0$ and variance $\sigma^2$.

This assumption allows statistical inference in finite samples since the OLSE $\hat\beta$ will be also normal with mean $\beta$ and variance-covariance matrix $\sigma^2(\sum_{i=1}^n x_ix_i')^{-1}$. Assuming $\sigma$ is known, one could test whether the $\ell$th component of $\beta$ is equal to zero by checking whether $$\frac{e_\ell\cdot\beta}{\sqrt{\sigma^2e_\ell\cdot(\sum_{i=1}^n x_ix_i')^{-1}e_\ell}}$$ is larger than 1.96. Here $e_\ell$ denotes the $\ell$th basis canonical basis vector of the Euclidean space.

Another important consequence of assumption A4 is that $\hat\beta$ is not only BLUE (best linear unbiased estimator) but BUE (best unbiased estimator).

---

Note that we don't need the assumption that $u_1, u_2, \dots, u_n$ are independent and identically distributed (assumption A4 implies it in the above setting though). If we added the assumption, it would suffice to condition on $x_i$ instead of the full sample, as the full sample would be, by assumption independent, i.e. we would have, for example, $\mathbb E[u\mid x_1, x_2,\dots,x_n] = \mathbb E[u\mid x_i]$.

The sampling assumption is also very specific to the use-case. For example, independent errors works well for most cross section applications, but it is a very strong assumption for time series or panel data analysis.

---