I want to derive the decision rule for the local constant logistic regression:
Consider the log-likelihood for the GLM (general linearised model)
\begin{equation} l( \beta_{0}, \beta_{1})= \sum_{i=1}^{n}(y_{i}(\beta_{0}+\beta_{1}^{\tau}X_{i})-\log(1+\exp(\beta_{0}+\beta_{1}^{\tau}))) \end{equation}
To bring in some locality we bring in some kernel weights into the objective function.
The local model at $\boldsymbol{x}$ is based on the minimiser of
\begin{equation} \sum_{i=1}^{n}k(\frac{\boldsymbol{x}-X_{i}}{h}) (y_{i}(\beta_{0}(\boldsymbol{x})+\beta_{1}(\boldsymbol{x})^{\tau}X_{i}) - \log(1+\exp(\beta_{0}(\boldsymbol{x})+\beta_{1}(\boldsymbol{x})^{\tau}X_{i}))) \end{equation}
The Nadaraya–Watson version would be a "local constant" estimator, arising from $P(y=1|X=\boldsymbol{x}) = \frac{\exp(\hat{\beta_{0}}(\boldsymbol{x}))}{(1+\exp(\hat{\beta_{0}}(\boldsymbol{x})))}$, with $\hat{\beta_{0}}(\boldsymbol{x})$ the solution to
\begin{equation} \sum_{i=1}^{n}k(\frac{\boldsymbol{x}-X_{i}}{h})(y_{i}\beta_{0}(\boldsymbol{x}) - \log(1+\exp(\beta_{0}(\boldsymbol{x}))) \end{equation}
This is not a closed-form solution. In the case of binary regression above we do not need to find $\beta_{0}(\boldsymbol{x})$ exclusively.
Consider the local constant likelihood objective for binary classification above.
I want to derive an expression for the decision rule for the corresponding classifier. That is, find a condition which determines whether or not the estimated $P(Y = 1|X = \boldsymbol{x})$ is above or below 0.5.
Attempt:
\begin{align*} & \operatorname{min} \sum_{i=1}^{n} k \left( \frac{x-X_i}{h}\right) \left(y_i \beta_0(x) -\operatorname{log}(1+\operatorname{exp}(\beta_0(x))\right)) &\\ l(\beta_0(x)) & = \sum_{i=1}^{n} k \left( \frac{x-X_i}{h}\right) \left(y_i \beta_0(x) -\operatorname{log}(1+\operatorname{exp}(\beta_0(x))\right)) &\\ \frac{\partial l(\beta_0(x))}{\partial \beta_0(x)} & = \sum_{i=1}^{n} [(0)](y_i\beta_0(x) - \operatorname{log}(1+\operatorname{exp}(\beta_0(x))) + k \left( \frac{x-X_i}{h}\right) \left[y_i - \frac{1}{1+\operatorname{exp}(\beta_0(x))} \times \operatorname{exp}(\beta_0(x))\right] &\\ 0 & = \sum_{i=1}^{n} k \left( \frac{x-X_i}{h}\right)\left[y_i - \frac{\operatorname{exp}(\beta_0(x))}{1+\operatorname{exp}(\beta_0(x))}\right] &\\ \text{where} &\\ & \frac{\operatorname{exp}(\beta_0(x))}{1+ \operatorname{exp}(\beta_0(x))} = p(x) &\\ \text{For class 1:} &\\ & \sum_{i=1}^{n} k \left( \frac{x-X_i}{h}\right)\left[y_i - \frac{\operatorname{exp}(\beta_0(x))}{1+\operatorname{exp}(\beta_0(x))}\right] \geq 0 &\\ \text{For class 2:} &\\ & \sum_{i=1}^{n} k \left( \frac{x-X_i}{h}\right)\left[y_i - \frac{\operatorname{exp}(\beta_0(x))}{1+\operatorname{exp}(\beta_0(x))}\right] < 0 &\\ \end{align*}
I have to write the decision rule in terms of the probability, but am unsure how to go about it.
