I am working on a project where we perform non-response adjustment by weighting survey respondents by their probability of response. In order to do this, we need to estimate each respondents probability of response using a model (typically a logistic regression). Essentially, after receiving all of our survey responses we have a subset who responded (1s) and a subset that didn't (0s), the goal of the model is to take this information and change the 1s and 0s into probabilities.
My colleague was describing the current method of creating the model where they use step-wise regression to select the model without any sort of cross-validation or holdout procedure involved. I was going to mention how step-wise methods are generally frowned upon and they are data dredging, but then I thought, maybe it doesn't matter? If all I want to do is estimate the probabilities of my training data (I don't care to use this model for future data and I don't care about analyzing the coefficients) does it matter if I am over-fitting the training data? Further, maybe that is actually my goal? Maybe I actually want to over-fit?