0

I have topic models probability obtained using LDA topic models method. I’d like to use these probabilities for 5 topics to predict an outcome (0/1). I’d like to find an optimum cutoff point of each topic to the outcome (0/1). I was thinking about ROC but it seems like it may not work since I don’t have the predicted outcome. Are there any methods that allows me to find the cutoff points for each variable, or does it make sense to do it?

Background

  1. We collected open ended responses from participants and performed the topic modeling.
  2. we collected outcome of choice from these participants e.g., intent to leave, performance, etc
  3. hypothesis is that the response the participants enter is indicative of the outcome.

The question is: at what points in each of these topic model probabilities the outcome value changes e.g., from 0-1 or 1-0.

Kuni
  • 145
  • 1
  • 7
  • Please provide more information, in particular what you mean by "I was thinking about ROC but it seems like it may not work since I don’t have the predicted outcome." Do you know the actual `outcome` that you're trying to be able to predict for each case? Are the 5 topics mutually exclusive and comprehensive, so that for any one case the probabilities add up to 1? How does the `outcome` differ from the topic probabilities that you've already modeled? – EdM Aug 18 '20 at 21:55
  • @EdM, really good questions. I’ve updated my question to give a brief background. The topics are not mutually exclusive or independent. Hence my problem with using classification. But I’ve been trying to find a way to analyze these kinds of data. One thing I’m curious about is to see if it’s possible to get the optimum cutoff points where if the topic probabilities for one topic is higher or lower than the cut point, then gives the outcome. Much like sensitive/specificity. – Kuni Aug 19 '20 at 11:37

1 Answers1

1

It's seldom that there is a single "point" at which the "outcome value changes." There's usually a gradation with respect to a predictor in terms of the probability of a particular outcome.

So what usually works best for a binary outcome is a probability model for the outcome that takes as much information into account without overfitting. Furthermore, with binary outcomes there's a particular risk of omitted-variable bias; if you leave out any predictor associated with outcome it can make it harder to identify other predictors associated with outcome.

In that context, looking for separate cutoffs for each of the predictors based on ROC is throwing away the detailed information about how probability of outcome changes with each topic's probability. It's also treating the topics separately instead of together.

So instead of looking at your topic probabilities separately with respect to outcome, it would seem to make the most sense to combine them into a single model. A simple model might be a logistic regression model that includes each topic probability as a predictor, which in R might be written for a 0/1 outcome like:

glm(outcome ~ T1 + T2 + T3 + T4 + T5, family = "binomial")

where T1 etc represent the probabilities found for the corresponding topic mappings. That provides a probability model for outcome that uses all the probability information about each of the topics together at once. In principle that could be extended to a mixed model in which you allow for differences among individuals.

Then you can get an ROC that combines information from all of your topic mappings at once. You then use your knowledge of the subject matter and the goals of your project (e.g., how much more costly are false negatives than false positives for you) if you need to choose a probability cutoff that best represents the "point" at which the "outcome value changes."

EdM
  • 57,766
  • 7
  • 66
  • 187
  • thank you for an awesome answer. It is really helpful. One question: can I do logistic regression even if the topic probabilities may not be independent? – Kuni Aug 20 '20 at 10:10
  • 1
    @Kuni that's a standard issue in regression. Predictors, like your topic probabilities, are often correlated. That can make the precision of individual regression-coefficient estimates worse than you might want, but it doesn't necessarily make the regression model overall much worse at prediction. The big problem is if the set of predictors is close to linearly dependent, which is why I asked earlier whether all the probabilities add up to one. If they do, or come close to that, you might want to omit one predictor or use a penalized approach like ridge regression to handle the problem. – EdM Aug 20 '20 at 12:42
  • Got it! super helpful. Thank you – Kuni Aug 20 '20 at 15:16