Prediction using a logistic regression model

Question

Given a logistic regression model:

$y \in \{0, 1\}$

$ P(y=1|x;\theta) = h_{\theta}(x) = \frac{1}{1+\exp(-\theta^T x)}$

And given the value $\theta^*$ which maximises the conditional likelihood $P(y|X; \theta)$:

It seems to me that, given a new training example $x$, I should calculate the predicted value as:

$ y^*|x; \theta^* = \textbf{1} \{\frac{1}{1+\exp(-\theta^{*T} x)} > 0.5 \} $

However a well known online ML course (page 3) purports that the prediction rule is:

$ y^*|x; \theta^* = \textbf{1} \{\theta^{*T}x > 0 \} $

These two rules don't agree on e.g. the trivial case $x \in \mathbb{R}, x =0$. Which is correct?

Answer 1

They do agree. One deals with a probability of $p=0.5$. The other deals with a log-odds of $0$.

$$ \log\bigg(\dfrac{p}{1-p}\bigg)=\log(1)=0 $$

Importantly, though, logistic regression alone is not a classification method, there’s nothing special about using $0.5$ probability as a cutoff threshold, and methods like logistic regression are best-evaluated on their probability outputs rather than threshold-based metrics (e.g., accuracy, sensitivity, specificity, F1 score).

https://www.fharrell.com/post/class-damage/ https://www.fharrell.com/post/classification/ https://stats.stackexchange.com/a/359936/247274

Proper scoring rule when there is a decision to make (e.g. spam vs ham email)

Answer 2

Of course, they agree.

We know that always, $\theta ^{*T} \in R $.

Your condition is:

$$ \frac{1}{1+e^{-\theta^{*T} x}} > 0.5 $$ $$ \implies \frac{1}{1+e^{-\theta^{*T} x}} > \frac{1}{2} $$ $$ \implies 2>1+e^{-\theta^{*T} x} $$ $$ \implies 1>e^{-\theta^{*T} x} $$ $$ \implies e^{0}>e^{-\theta^{*T} x} $$ Since $ y=e^{x} $ is always strictly increasing, we can conclude from the above inequality that $$ \implies 0>-\theta^{*T} x $$ $$ \implies 0<\theta^{*T} x $$ Therefore $$ \theta^{*T} x>0 $$

And yes, as others pointed out, it is not advisable to have thresholds and include the decision-making in your model. Because, you would just be losing data(say, of uncertainty) by doing that. For example, if you get the $P(y=1)=0.990$, you are not only classifying it as 1, but you are very(99%) sure that it is 1. This is not exactly the case if you get $P(y=1)=0.510$, because you are probably in a dilemma here. But if you only consider only the value of the final classification and not bother about the probability as long as it is greater than 0.5(or some other threshold), you would lose some information you get from the model.