Model selection using nested cross validation

Question

I am working on a school project using remote sensing data, for classification purposes. And I am trying to select the best model (models) for my data. The approach that I adopted is the following:

1. Looping over different models and using K-fold cross-validation (k=10).
2. Select the best performing model (models)
3. Parameter grid search (or random search) for each of the models I chose in (2)
4. Cross-validation using the best parameters I found in (3)
5. Perform prediction

Does this approach make sense? I am still in step 1 and it is already taking up so much time. I am new to machine learning, so I wanted to get some insight into how people in this domain usually tackle projects. Any advice would be highly appreciated.

Answer 1

You should optimize the hyperparameters of each class of model first and then compare the different classes of model to each other. Otherwise, the performance of the different model types will be heavily influenced by the arbitrary selection of hyperparameters. For example, with the initial parameters, it may be the case that a random forest performs better than SVM. However, once both have the best hyperparameters, it may be the reverse case. If you had chosen the random forest and then optimized, you'd end up with a worse than optimal model because you never even tested the optimal SVM model.

Using loops where you can and reusing old code will speed things up. Also, depending on your program, there are functions which implement CV and hyperparameter selection, so you don't need to code that by hand. In Python you can use GridSearchCV.