I was trying to understand the role of the bias term in linear regression which is given by,
y=w^T. phi(x)+b
From what I understand it allows for any fixed offset in the data. To be more specific, it compensates for the difference between the averages (over the training set) of the target values and the weighted sum of the averages of the basis function values.
What are other uses for this bias term?
Put it simply, we want bias in situation: setting default value for hypothesis function. For example, when all predictors for house price are zero, we want the default house price will be 1000. That's when bias comes to play.
If you are estimating a model you typically are looking for the minimum variance unbiased estimator (MVU). Hence, you intend to find the one model whose predictions have the lowest variance possible while maintaining a bias $(y_{true} - y_{estimated})$ of zero.
The thing is, there is a trade-off between the variance of a model and its bias. Ideally you want both values as close as possible to zero, which then would guarantee you correct predictions. However, by reducing the bias on the training-data you are raising the variance on the test-data and vice versa.
Hence, there are certain cases where the MVU may not be what is wanted. You then need the bias term to be different from the differences between the averages. Which then would be a different use case for your bias term.
