0

I have an unbalanced dataset and would like to apply SMOTE to the training data. I can either do one of the following:

  1. Inside trainControl() add sampling = "smote" and then run train()
  2. First sample the training data using SMOTE(), NOT include sampling in trainControl() and then run train(). For SMOTE() I used the default parameters as in the documentation: SMOTE(form, data, perc.over = 200, k = 5, perc.under = 200, learner = NULL, ...)

However, I end up with different results because the training dataset are different sizes, the first option maintains the number of observations, but the second option reduces the number of observations.

I would like to know why and which one would be the correct way of doing it. Thanks.

Fernando Han
  • 15
  • 1
  • 6
  • Are you sure you need balancing? Whats the ratio of your class sizes? What's the size of your data set? How are you evaluating your model performance? I'm sorry to redirect the question, but these balancing techniques are vastly overused, and often a different approach that does not mangle your training data is better. – Matthew Drury Jan 30 '19 at 17:15
  • This is a fraud dataset, being 99% not fraudulent and 1% fraudulent, very unbalanced. The dataset is composed 4500 observations. I will build models using logistic regression, classification tree and random forest. To compare the models I will be using Kappa since it is more appropriate for unbalanced classes. – Fernando Han Jan 30 '19 at 18:57
  • 1
    Since you're predicting fraud, you should almost certainly NOT be viewing it as a classification problem, instead you almost certainly want to think of this as a risk estimation (what is the probability this is fraudulent) or a ranking (what is the most likely to be fraudulent) problem. It's generally not correct in this circumstance to try to classify each observation as fraud or not fraud. Class balance is not a very relevant concern in risk estimation or ranking problems, so I'll stick to the point that SMOTE is not useful in this problem. – Matthew Drury Jan 30 '19 at 20:51
  • Thank you Matt for the explanation. So SMOTE would only be used if it was a purely classification model, correct? Or even then there are exceptions? – Fernando Han Jan 30 '19 at 21:40
  • 1
    I haven't found class balancing techniques to ever be useful, personally (and I've worked with some ostensibly unbalanced data sets, half a percent or so of the positive class). I've always found it better to fit the model optimizing for probabilistic measures like log-loss (for which class balancing is not really relevant), and then set a classification threshold when I need to construct a decision rule. There may be situations where one class is exceedingly rare that it is important, but that's outside my experience. – Matthew Drury Jan 30 '19 at 21:42
  • Thanks so much for your input. This is my first major project in machine learning and your insights and experience were really helpful! – Fernando Han Jan 31 '19 at 12:03
  • You're welcome Fernando! – Matthew Drury Jan 31 '19 at 17:26
  • 2
    Possible duplicate of [SMOTE data balance - before or during Cross-Validation](https://stats.stackexchange.com/questions/346534/smote-data-balance-before-or-during-cross-validation) – G5W Jan 31 '19 at 17:54

1 Answers1

1

See the documentation.

You can optimistically bias the results when sampling outside of resampling. A synthetic version of a training set point can end up in the test set.

topepo
  • 5,820
  • 1
  • 19
  • 24
  • Great documentation, it explains exactly my question. Thank you so much! The reason I am asking this is because I want to run a Chi-square test (goodness of fit) on different logistic regression models of different complexity (less variables) but the `anova()` function does not work with caret models built with train. So I am having to sample outside and then run the `glm()` function. As a follow-up question, is there a way either to use smote in resampling using `glm()` or do chi-square test using caret model? – Fernando Han Jan 30 '19 at 19:33