2

I refer you to Breiman's paper Statistical Modeling - A Tale of Two Cultures where he illustrated some examples of descriptive modelling. Under section 11.1, 100 runs of regression were performed, each time leaving randomly selected 10% of the data. May I know what is the purpose of performing 100 runs and getting the average error rate and relationship between the outcome and explanatory?

I understand the objective of creating tests sets in predictive modelling is as to prevent overestimating the predictive accuracy of the model. But I do not quite understand in the context of descriptive modelling.

tatami
  • 695
  • 1
  • 7
  • 24
  • The question appears clear at first: "I do not quite understand [the objective of creating tests sets] in the context of descriptive modelling." But then, the bounty goes to the response which focuses on the context of predictive modeling. Maybe your question is not so clear after all. – g3o2 May 29 '17 at 12:20
  • @g3o2 care to explain on why the response is more appropriate for predictive modelling? I am still confuse on the differences between predictive and descriptive modelling. The link you gave is helpful. – tatami May 30 '17 at 13:47
  • It would be useful if you considered either opening another question or extending your current question to ask explicitly for the difference between predictive modeling and descriptive modeling. This [paper](https://www.stat.berkeley.edu/~aldous/157/Papers/shmueli.pdf) provides decent explanations in its sections _1.2 Predictive modeling_ and _1.3 Descriptive modeling_. Furthermore, I refer you to the book [Data Science for Business](http://data-science-for-biz.com/DSB/Home.html), a more "official" resource, which also describes the said differences. – g3o2 May 30 '17 at 14:16

4 Answers4

3

The holdout is considered the validation set, not the test set. The test set gives you an idea of how well your model performs on unseen data, ie how well your model represents reality. That said, ideally you can catch overfitting before you expose your model to the test set.

This is where the validation (holdout) set comes in. By systematically setting aside a validation set to compare, in this case, 90% of the training data to, you benefit in at least two ways:

  • If there is large fluctuation in error between validation sets, there may be overfitting
  • Using the average of each training/validation split provides you with a better sense of the model's true accuracy. Ultimately, this gives you an overfitting-proof assessment, as your total model accuracy is less dependent on the arbitrary choice of your training set
redress
  • 740
  • 1
  • 7
  • 16
  • Would this validation process replaces method to assess descriptive model's performance (e.g. goodness-of-fit test & R^2)? – tatami May 28 '17 at 11:46
  • Cross validation is not a statistical method, it is a data practice. Using R^2 doesnt exactly make sense in this context. – redress May 29 '17 at 23:58
3

In the paper you have referenced, the author uses cross-validation in the context of a predictive modeling approach, e.g. to perform (automatic) variable selection. In descriptive modeling, variable selection is performed based on theory first and only then confronted with the model output and data. See this post for a discussion on predictive vs descriptive.

In a purely descriptive setting, model quality assessment is also not solely focused on quantitative indicators but also relies on theoretical assumptions, expectations and the modeling process itself. This is unlike a setting which includes a predictive purpose, where being able to put a confidence interval around quantitative quality measures such as R squared can be considered indispensable.

In descriptive modeling and more particularly for OLS, cross-validation or most often bootstrapping can rather be used to identify or measure the impact of potential leverage points and outliers. As such, you will gain further knowledge about the underlying population or sample and be able to decide on the approach to take, such as for example whether to resort to more robust regression methods or to flatten certain observations in the sample.

g3o2
  • 506
  • 4
  • 10
  • With reference to your third paragraph, does it mean cross-validation is used to remove outliers? By detecting outliers and influential observation, how does it help to understand the underlying population? These outliers does not represent the population. – tatami May 30 '17 at 13:54
  • The detection of such points is crucial in descriptive modeling because they can introduce significant bias, hampering the quality of the (ideally general) conclusions drawn from the model. Among many other methods, you can also use resampling methods to _detect_ such points; how you _deal_ with them is another story. When using bootstrapping (or, admittedly more rarely, LOO-CV), scrutinize the iterations for instances of major model performance drops/spikes (incl. significance of coefficients and major variations in the coefficient values in the case of regression). – g3o2 May 30 '17 at 14:55
1

Think of it like this -

Suppose you have a population of 1000 students and you want their average height ̅X. So, you go and take a sample of 20 students and measure their height and take the average (μ). Chances that μ will be true representative of the population average height (X) is low. But if you repeat the process 100 times and take the average of all the 100 averages (average of the sampling distribution) then it will be unbiased estimation of the population height i.e.true representative of the sample population.

Now coming to your question - The objective of creating test set is to get an idea of error rate on unseen data to determine if the model is over-fitting. Similar to my above example, if you run just one model and use that to estimate error on test set, it might not be true representative of the actual test error. So, you run many models and calculate the average error. This error will be true representative of the actual test error and whether or not this error is large you can determine whether the model is over-fitting or not.

saurav shekhar
  • 391
  • 1
  • 2
  • 11
0

If your "descriptive" model doesn't predict well, how can you trust anything it has to say about your data/process? If you don't use something like cross-validation, how can you have a reasonable idea of how well your model predicts?

The fact that your model predicts well doesn't guarantee that you can pull any useful knowledge out of it, but if it can't predict well how can you have any confidence in what you believe it says?

Wayne
  • 19,981
  • 4
  • 50
  • 99