2

I am working on an xgboost model using caret. I'm using cross validation, but don't know if I'm understanding it correctly. As I understand, it creates multiple training and test sets. Does this mean that the data doesn't have to be splitted to training and test sets before the modeling?

If so, how do I obtain the mean absolute errors from the test set? I am using R.

I tried to get them using the following code:

postResample(pred = pred, obs = obs)[3]

but the MAEs seem rather optimistic. Is it possible to obtain the MAEs just for the test set?

I am also using hyperparameter tuning, if that matters. The model is also linear.

kkz
  • 205
  • 1
  • 5

1 Answers1

1

Your understanding is correct; the data used from the train function essentially act as train and validation sets; they do not reflect "true performance" on unseen data.

Yes, we will have to use a separate test set to assess the performance of the particular algorithm on unseen data. To that extent, the use of nested cross-validation might be beneficial for this, this post on Nested cross validation for model selection gives an extensive discussion on the topic.

In any case, we can obtain MAE on the test-set by simply using predict on the test explanatory variables and then comparing the predicted response to the actual response from the test set. As you correctly noticed, if we do not use a separate test-set the performance metrics will (very likely) be optimistic.

usεr11852
  • 33,608
  • 2
  • 75
  • 117
  • And this is all the case _because_ OP is doing hyperparameter tuning, correct? If OP had a single model being trained on the training folds and scored on the testing folds, then the average score would be an unbiased estimator? – Ben Reiniger Jul 31 '19 at 23:05
  • Yes, correct is because of the tuning. Yes it will be (practically) unbiased but do read this mega-thread on: https://stats.stackexchange.com/questions/61783/ to get the full idea! Variance can really hurt us! ("Practically" because we will not use all of our data in the training set, so we do have minor bias there.) – usεr11852 Aug 01 '19 at 00:01