1

I am in the seemingly unusual situation of having practically unlimited data. In this case, will linear regression choose as good of a model as any other algorithm in the case where the number of samples approaches infinite?

To clarify, I have two cases in mind. With the same set of features, will ordinary least squares produce as good a prediction on out-of-sample data as algorithms such as lasso, ridge, or elastic net - algorithms meant to increase the generalization of a prediction. My intuition says that with enough data, OLS will do just as well. Similarly, given that the underlying relationship is linear, with enough data, will simple linear regression produce as good of a prediction as a boosting algorithm such as gradient boosting regression, or as a bagging algorithm such as random forests?

user023049
  • 35
  • 3
  • 2
    Not a duplicate, but the points in this thread are relevant: another algorithm like RF might be sufficiently robust to detect all kinds of relationships which must otherwise be fully specified to the regression model. This is true no matter how much data you have. http://stats.stackexchange.com/questions/164048/can-random-forest-be-used-for-feature-selection-in-multiple-linear-regression/164068#164068 – Sycorax Aug 17 '15 at 19:02
  • 3
    If the underlying relationship is not exactly as specified by your model, then the answer is clear. This pushes the burden back on you: how sure are you that your model is *the one and only correct* one? – whuber Aug 17 '15 at 19:09
  • I should have been more clear. I meant to ask the question under the assumption that the underlying relationship between the dependent and independent variables is linear. – user023049 Aug 17 '15 at 19:18
  • 2
    "linear regression" is not a form of model selection. So linear regression won't "choose" a model at all. What comes under "any other algorithm"? There's many other aspects of your question unspecified or unclear. – Glen_b Aug 17 '15 at 19:20

1 Answers1

2

I think what you meant to ask was, how much value do LASSO, ridge, or elastic net add when you are in an information-rich environment? By information rich, I mean that you have enough rows of data to make up for all but the worst collinearity problems (see my post that begins with "The collinearity problem is way overrated!") such that you can estimate all coefficients with very high precision. Then I think the answer is, not much. All three of these methods work by adding information that your coefficients should not be that big, but by definition, you have all the information you need. LASSO and elastic net offer the added benefit of variable selection, but as the null hypotheses of the form $H_{0i}: \beta_i = 0$ are unlikely to be exactly true in the first place, as your sample size gets ridiculously large, these methods will zero out fewer and fewer coefficients.

Community
  • 1
Ben Ogorek
  • 4,629
  • 1
  • 21
  • 41