1

A linear regression model has been created based on a dataset with observations which can be sorted into several different categories.

I have been asked to assess how well this regression model (which was created using the entire dataset) fits the observations by the category subsets. Correct me if needed, but this seems to render R-squared useless - (1) because the means of predictions and observations inside a specific category are not equal (sum of $\hat{Y_i}$ does not equal sum of $Y_i$), and the relationship that SSE + SSR = SSTotal no longer holds within each category.

Calculating the correlation coefficient between the overall model's fits for a single category, and those category's observations, would be equivalent to fitting a new least-squares model specific to that category and calculating an R-squared value, but that would not be assessing the fit of the OVERALL model to that category. Does that make sense?

What other options would you recommend here? Would some kind of likelihood ratio test perhaps be appropriate? I confess I'm not sure what else to do.

Thanks for your help.

Greg
  • 123
  • 1
  • 5

1 Answers1

1

You can compute an $R^2$ value on subsets of your dataset. This will give you a measure of fit (let's call it $R_c^2$) for each category.

Here is how to do it:

Let $S_c$ be a set containing the indices of all your data points that belong to category $c$. Then you can compute the corresponding $R_c^2$ for this set:

$R_c^2 = 1 - \frac{RSS_c}{TSS_c}$

where you compute the residual sum of squares $RSS_c$ and the total sum of squares $TSS_c$ on data points that belong to category $c$. In formulas:

$RSS_c = \Sigma_{i \in S_c} (ŷ_i - y_i)^2$

$TSS_c = \Sigma_{i \in S_c} (y_i - \overline y_S)^2$.

Notation:

$y_i$ is the true value for data point $i$.

$ŷ_i$ is your prediction for data point $i$ (using the full model).

$\overline y_S = \frac{1}{|S_c|} \Sigma_{i \in S_c} y_i $ is the mean over all true value for $y$ in $S_c$.

mcb
  • 121
  • 2
  • 1
    This approach makes sense but it seems to me it could be problematic, as illustrated in my post at https://stats.stackexchange.com/a/13317/919. The issue lies in what the intended meaning of "how well ... fits" is. The use of $R^2$ masks any changes in the actual sizes of residuals within each category, potentially eliminating the most important information of all. It is possible for responses in one category to be beautifully fit (relative to other categories) but for the $R_c^2$ to be zero. – whuber Jul 31 '18 at 11:35
  • 1
    @whuber I think the idea behind this question might deserve more attention. Often times models or features seem to be evaluated based on a loss metric for the whole dataset. When doing feature selection or evaluating feature importance, features may be deemed unimportant if they don't drastically improve the overall loss metric even if they improve the loss metric on a small subset of the data. You are absolutely correct that it's important to take into account the amount of variability there is within the subset and this subset-based $R^2_c$ does not take that into account. – jsk Jun 20 '19 at 00:25
  • @whuber Not sure if it would be better to add a bounty to this question, or reframe it a little and post a new question. – jsk Jun 20 '19 at 00:40