It is apparently a common practice to use ROC as a feature selection method in my new job. They test the variables one by one against the response and anything with ROC<=54 is tossed aside.
Would you say that this is a good practice? I'm quit skeptical as I would have rather used ensemble learning in order to incorporate feature selection. Or used a regularization method like lasso/elastic net.
Univariate feature selection is generally a poor method.
This question is deftly answered by silverfish in the context of correlation, but all his arguments apply to your case as well. In short, there is no reason to believe that univariately checking how each individual variable $x$ is related to your response $y$ reveals anything about the multivariate nature of the relationship between $X$ and $y$. It's quite possible that you end up screening out many of your good predictors.
As you point out, LASSO, ridge, or glmnet are much preferred methods for feature selection in a multiple regression model, as they:
You should carefully and respectfully start pointing your team towards a more modern and disciplined approach.
(*) You also don't mention if your team is testing for non-linear relationships when fitting these univariate models. At the very least, these univariate models should be based on some basis expansion of the feature, like cubic splines. Clearly if they are only testing for univariate linear relationships, there there are some issues there as well.
Echoing Matthew's response from another light: many markers have a concept of stratified predictive accuracy. In this sense they provide extremely good predictive accuracy in a subgroup or in-tandem with another marker. Two examples from the health sciences:
Suppose for instance two types of breast cancer grow in women who are premenopausal and post-menopausal. The majority of women who are diagnosed with cancer are post-menopausal, yet the cancers diagnosed in pre-menopausal women are extremely aggressive, difficult to treat, and their genetic markers are unknown. If you naively presume that all cancers have the same genotype and genetic markers, you will show a low predictive accuracy for a genetic marker that yields a 100% Area-Under-The ROC in premenopausal women.
Another example is how a dyad of conditions might be necessary for disease. For instance, in nephrology, people are typically at risk and subsequently are diagnosed with Stage 3 Chronic Kidney Disease once they develop both hypertension and diabetes. CKD is known to progress to ESRD, which required chronic renal replacement therapy, and is generally very bad. Interventions to halt progress of CKD are poorly known, so earlier diagnostics are needed, but the manifestations of the disease are coincident with many other conditions, it is only a specific spectrum of conditions or a combination of markers that may inform clinicians that the patient has CKD.
Just the same, a stepwise approach to evaluating a sequence of markers does not hold the promise of identifying an optimal ROC in any scientific sense, albeit perhaps in a statistical sense it would. In my experience of evaluating markers, the best use for ROC is evaluating a pre-specified hypothesis about a specific marker rather than a huge list thereof. The CIs for ROCs and their AUCs tend to be quite wide since they deal with empirical functions, and after correction for multiple testing, the risk of Type II error is just too high to justify evaluating 100 markers or more.
The exception to this might be the understanding that the nature of the analysis is a hypothesis generating study. That means, however, that none of the previously collected data may serve to confirm this hypothesis. A separate study must be done. Many clinicians are disheartened at how irreproducible results can be from such fishing expeditions.
Since you are using SAS, I thought I'd share this. I'm not sure what model you are using, but if you are using logistic regression this may be a useful resource.
Sample 54866: Logistic model selection using area under curve (AUC) or R-square selection criteria
Under "Details", it reads:
In addition to the AIC and BIC criteria available in PROC HPLOGISTIC, the SELECT macro can also choose models using the area under the ROC curve (CHOOSE=AUC), the R-square statistic (CHOOSE=RSQUARE), or the max-rescaled R-square statistic (CHOOSE=RSQUARE_RESCALED).
Under "Limitations", it reads
The best subsets selection method (SELECTION=SCORE) available in PROC LOGISTIC is not available in the SELECT macro. The area under the ROC curve criterion (CHOOSE=AUC) is not available with nominal, multinomial (LINK=GLOGIT) models.
