Variance covariance matrix of parameters in logistic regression?

Question

The given below image is taken from book Introduction to Linear Regression Analysis (Douglas C Montgomery) My apologies in advance for not typing , I just want to understand the concept.

(1) First Question
We are estimating the parameters in logistic regression by Maximum Likelihood Estimation concept. The thing I am not able to get in this image is - what is $n_{i}$ and how the expression is obtained.( Doubt already marked by violet arrow; the highlighted yellow expression is the same as in above equation.)

$\ $

(2) Second Question
This $n_i$ expression is also involved in the variance of parameters(see below picture). But in these links - Link 1 and Link 2, $n_i$ is not mentioned anywhere in the discussion.

Could someone help me to understand this thing?

score 3 · Accepted Answer · answered Jul 06 '21 at 01:42

I agree with EdM that the notation is a bit confusing, but this is because the book describes two uses of logistic regression. The first equation describes the log-likelihood when there is a single trial for each unit in the data set; for example, did the patient die or not. This is by far the most common use of logistic regression. This is really the only thing you need to focus on and likely the only use of logistic regression you'll ever encounter.

The second equation describes the log-likelihood for a more esoteric use of logistic regression, which is when each individual has multiple trials; for example, how many answers did a student get correct on a test. Each item on the test is a trial for each observation. So, for the analysis of a 5-item test taken by a whole classroom of students, the students are the observations, and each student has 5 trials, so each student's value of $y_i$ could be a number from 0 to 5, and for all students $n_i=5$ (but in principle, if you gave each student a test with different numbers of problems then the $n_i$s would vary). If each unit has a single trial, then $n_i=1$ and the two expressions for the logistic regression log-likelihood are equivalent. Wikipedia distinguishes between binomial regression, which can have $n_i \ge 1$, and binary regression, which has $n_i=1$ for all $i$; this is exactly the distinction made in the textbook.

The reason you are unlikely to see $n_i$ in any equations for logistic regression is that the scenario with multiple trials for each individual is usually not estimated with standard logistic regression. Instead, one might use a mixed model to allow each individual to have their own intercept, which adds flexibility to the model. The form of logistic regression with $n_i \ne 1$ is extremely uncommon. I would not worry about it, and if you are trying to understand logistic regression in its most typical case, assuming $n_i = 1$ is sufficient 99.9% of the time (and all most people ever learn).

Could you clear my [this doubt](https://stats.stackexchange.com/questions/532955/mean-accuracy-and-standard-error-of-the-accuracy-for-knn-classification-algorith). Its a different questions but didnt hear from anyone. — Girish Kumar Chandora, Jul 06 '21 at 04:16
Glad this helped. I can't help with your other problem as I have no knowledge of that area. — Noah, Jul 06 '21 at 06:39

score 2 · Answer 2 · answered Jul 05 '21 at 21:27

The first display is a bit confusing, because it uses the symbol $n$ in two different ways. For the upper limit of the products and sums, it's the total number of "observations." But the symbol $n_i$ is later meant to represent the number of trials on "observation" $i$.*

I find it easier to work backward from Equation 13.9. If there were $n_i$ total trials on "observation" $i$ and $y_i$ of them had positive outcomes with an underlying probability $\pi_i$ (based e.g. on covariate values), then $n_i-y_i$ had negative outcomes with a corresponding probability $1-\pi_i$. That directly gives the result for the log-likelihood shown in Equation 13.9. If you then set $n_i = 1$, recognize that

$$y_i \ln \left( \frac{\pi_i}{1-\pi_i} \right)= y_i \ln \pi_i - y_i \ln(1-\pi_i)$$

and figure out that the bare $\pi$ in your first highlighted region should be the "observation"-specific $\pi_i$ (misprints do happen), then you should be able to show that the (corrected) equation at the top of page 425 is for the special situation of Equation 13.9 in which each "observation" only has a single trial.

With respect to the second part of your question, this answer shows that iteratively reweighed least squares is equivalent to a standard numerical search method in the case of a binomial model with a logit link. With $n_i$ still representing the number of trials on "observation" $i$, the formula $V_{ii} = n_i \hat \pi_i (1-\hat\pi_i)$ is the plug-in estimate of the binomial variance based on the estimate $\hat \pi_i$ for the true probability $\pi_i$ when there are $n_i$ trials. If there is only a single trial per "observation," that's just $\hat \pi_i (1-\hat \pi_i)$, the form shown in the links you provide (which implicitly assume one "trial" per "observation" in the terminology used by this cited text).

*The cited text seems to use "observation" to represent all cases that have the same set of covariate values. I tend to think of an "observation" as being the result of a single trial. I would have found it easier to follow had the word "case" been used instead.

Variance covariance matrix of parameters in logistic regression?

2 Answers2