Say we have a binary classification problem with mostly categorical features. We use some non-linear model (e.g. XGBoost or Random Forests) to learn it.
Multi-collinearity will not be a problem for certain models. Such as random forest or decision tree. For example, if we have two identical columns, decision tree / random forest will automatically "drop" one column at each split. And the model will still work well.
In addition, regularization is a way to "fix" Multi-collinearity problem. My answer Regularization methods for logistic regression gives details.
Late to the party, but here is my answer anyway, and it is "Yes", one should always be concerned about the collinearity, regardless of the model/method being linear or not, or the main task being prediction or classification.
Assume a number of linearly correlated covariates/features present in the data set and Random Forest as the method. Obviously, random selection per node may pick only (or mostly) collinear features which may/will result in a poor split, and this can happen repeatedly, thus negatively affecting the performance.
Now, the collinear features may be less informative of the outcome than the other (non-collinear) features and as such they should be considered for elimination from the feature set anyway. However, assume that the features are ranked high in the 'feature importance' list produced by RF. As such they would be kept in the data set unnecessarily increasing the dimensionality. So, in practice, I'd always, as an exploratory step (out of many related) check the pairwise association of the features, including linear correlation.
If the non-linear model is tree-based model, then you shouldn't consider it serious. Different tree model will have different deal method, such as the random forest will keep them both (because they build the tree independently, and random select the feature for every trees), but it have no effect about the prediction performance, even you remove the redundant one. But for xgboost, it will choose anyone of them, and use it until the last tree build.
It just about the interpretation meaning, so remove the highly correlation variable is suggested.
Multi-collinearity is always a possible problem. Variables that are predictors in the model will affect the prediction when they are linearly related (i.e., when collinearity is present).
