2

I already referred this post post and post. Don't mark it as duplicate.

I am working on a binary classification problem using logistic regression. Loan default or not.

I have a requirement where I am told to predict the outcome classes for our unseen records and also report the confidence of predictions generated.

I am novice data scientist. So, am not really sure what is the difference between likelihood and confidence?

If I run logistic regression, I get likelihood measure like 70% probability of belonging to class 1 and 30% of probability of belonging to class 0.

My questions are as follows

a) Does likelihood and confidence mean the same? Is there any simple explanation that ordinary layman like me can understand.

b) Is there any tutorial that you can suggest which has on how to report confidence of predictions? When I use scikit-learn for logistic regression, I don't know how can I report confidence for the predictions?

c) Any idea on how can we generate the interval? In scikit-learn logistic regression tutorials that I find online, I only see probability/likelihood of an instance becoming label 1 or label 0. Can you guide me on how can we generate interval?

The Great
  • 1,380
  • 6
  • 18
  • 1) You’re asking about the “likelihood” of an event, but that does not appear to be part of your requirements. Did your boss ask for a “likelihood” of an event? // 2) Consider asking your boss for clarification on what exactly is needed. // 3) Do you have an explicit requirement to give a confidence interval? I can think of a way to report the results without giving a confidence interval, yet I would answer the questions about predicted class membership and confidence about loan default. – Dave Jan 18 '22 at 01:30
  • 1
    4) This seems like a perfect situation to [predict tendency, not membership.](https://www.fharrell.com/post/class-damage/) Making a [hard classification](https://www.fharrell.com/post/classification/) requires knowledge of what’s at stake. What do you lose when someone you predicted as safe winds up defaulting? What do you lose when someone safe is falsely regarded as risky and not approved for a loan? Those will influence how you use the probability output of your logistic regression. You don’t have to use the software default of $0.5$ as your decision threshold. – Dave Jan 18 '22 at 01:35
  • Hi @Dave - Yes, I wanted to compute/predict the likelihood and the confidence of my predictions. For ex: I can say that person A has a 90% likelihood of his loan being approved. But how confident am I about the inference/conclusion that we make from the model results. That's what they wish to know. Likelihood and confidence – The Great Jan 18 '22 at 02:28
  • 3) Yes, I do have an explicit requirement to report results with confidence (interval) etc. Can direct me to any resources that involves likelihood determination as well as confidence of the results? – The Great Jan 18 '22 at 02:30
  • A confidence interval is different from a colloquial “confidence” in the model. – Dave Jan 18 '22 at 02:39
  • Okay, is there any way to find the confidence of the model output? – The Great Jan 18 '22 at 05:37
  • That would be "model validation". – Dave Jan 18 '22 at 16:39
  • Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/133327/discussion-between-the-great-and-dave). – The Great Jan 18 '22 at 22:08
  • If I do cross validation, I get average accuracy across k folds. So, does it qualify as confidence in the model? – The Great Jan 18 '22 at 22:08
  • 1
    It's typical to chat in the chat after you continue the discussion in chat. Let's migrate there. – Dave Jan 18 '22 at 22:17

1 Answers1

3

My guess is that Bayesian (Logistic) Regression is what you're looking for.

In a regular Logistic Regression you find the (model) parameters $\beta$ that maximise the likelihood of your data (see Maximum Likelihood estimation). In a binary classification you use those parameters to calculate the probability $p$ of a given (test) sample $X_{test}$ of belonging to a positive class.

In Bayesian regression, on the other hand, you're not trying to find single best parameters $\beta$ that fit your data. Instead, you're trying to find the distribution of $\beta$s given your prior belief about the parameters (e.g. you can assume that they're coming from a normal distribution centered at $0$ with some standard deviation $\sigma$*) and using your data to update your prior belief, arriving at a posterior distribution of $\beta$s. You can use the means of those posterior distributions to do predictions - this should give you results similar (or equivalent**) to predictions done using estimates from a normal logistic regression. However, having an entire distribution of $\beta$s (as opposed to a having a single value) gives you an idea about the variability of those estimates. A posterior distribution with high variance indicates that your estimate of $\beta$ has lots of variability, i.e. given a slightly different training dataset, the MLE estimate of $\beta$ can change a lot. You can use the standard deviation of the posterior distribution to calculate the confidence interval for the distribution of $\beta$. Now instead of reporting $\beta = 0.1$ you can say that e.g. 95% confidence interval for $\beta$ is e.g. $[0.05, 0.15]$. I believe that this is what your requirements refer to as confidence.

Remark You can perform predictions using both $\beta$s corresponding to the lower and upper bound of the confidence interval and this will give you two distinct predictions).

Here is a Python example of Bayesian Logistic Regression performed using PyMC3 package.

* What consitutes a good prior is a topic for a separate debate. Gaussian distribution centered at 0 is a good default choice. It also has a regularization effect as it forces the parameters to be close to 0.

** Logistic Regression with uniform prior will give results equivalent to MLE estimates.

Tomasz Bartkowiak
  • 1,249
  • 12
  • 20
  • 1
    thanks for the help. upvoted – The Great Jan 18 '22 at 10:09
  • the machine always predicts “yes” with a probability between 0 and 1. So, is that called as our confidence score? – The Great Jan 18 '22 at 10:13
  • I found that from here - https://medium.com/voice-tech-global/machine-learning-confidence-scores-all-you-need-to-know-as-a-conversation-designer-8babd39caae7 – The Great Jan 18 '22 at 10:15
  • If Likelihood is same as confidence score of our predictions, then that is what I require. But is it same or different? – The Great Jan 18 '22 at 10:16
  • If you have time, can help me with this related question - https://stats.stackexchange.com/questions/560895/how-to-handle-hierarchical-categorical-variables-retain-or-drop – The Great Jan 18 '22 at 10:34
  • 1
    The confidence (interval) in this case would refer to $\beta$s, not the predictions themselves (at least not directly). Regular logistic regression gives you only one (MLE) estimate of $\beta$. Using this $\beta$ you can make a prediction, e.g. say that for $X_{test}$ it says that the probability of class 1 is 60%. In Bayesian Logistic Regression you get a range of $\beta$s (corresponding to the 95% confidence interval). If you use those two extreme $\beta$s to do predictions on the same test sample you'll get a range of probabilities of belonging to the class 1 (e.g. 55% - 65%). – Tomasz Bartkowiak Jan 18 '22 at 10:36
  • Yes, understand your point on CI. But does confidence Interval and confidence mean the same? – The Great Jan 18 '22 at 10:38
  • 2
    @TheGreat It is on you to define what confidence means. From what you wrote in your comments to me, it seems like you are interested in model validation to show a colloquial “confidence” that the model is a good one, rather than formal confidence intervals. – Dave Jan 18 '22 at 10:44
  • 1
    @TheGreat I don't know, although that would be my guess. Or it could be simply the difference between the $P(X = 1)$ and $P(X = 0)$, i.e. how _confident_ your model predictions are for a given sample (i.e. 51%/49% is not confident, 99%/1% is confident). – Tomasz Bartkowiak Jan 18 '22 at 10:44
  • Note: As others have said you are incorrect in labeling this as a classification problem. The Bayesian approach is most suited for communicating uncertainties. Short of that, computing 0.95 confidence intervals for predicted probabilities for selected covariate settings will be helpful. This uses standard logistic regression methods, with a normal approximation for the predicted log odds of the outcome. – Frank Harrell Jan 18 '22 at 13:32