What is the cost function in cv.glm in R's boot package?

Question

I'm doing a cross validation using the leave-one-out method. I have a binary response and am using the boot package for R, and the cv.glm function. My problem is that I don't fully understand the "cost" part in this function. From what I can understand this is the function that decides whether an estimated value should be classified as a 1 or a 0, i.e the threshold value for the classification. Is this correct?

And, in the help in R they use this function for a binomial model: cost <- function(r, pi = 0) mean(abs(r-pi) > 0.5). How do I interpret this function? so I can modify it correctly for my analysis.

Any help is appreciated, don't want to use a function I don't understand.

Feng Mai · Answer 1 · 2018-06-10T17:25:27.380

9

r is a vector that contains the actual outcome, pi is a vector that contains the fitted values.

cost <- function(r, pi = 0) mean(abs(r-pi) > 0.5)

This is saying $cost = \sum|r_i - pi_i|$. You can define your own cost functions. In your case for binary classification you can do something like this

mycost <- function(r, pi){
    weight1 = 1 #cost for getting 1 wrong
    weight0 = 1 #cost for getting 0 wrong
    c1 = (r==1)&(pi<0.5) #logical vector - true if actual 1 but predict 0
    c0 = (r==0)&(pi>=0.5) #logical vector - true if actual 0 but predict 1
    return(mean(weight1*c1+weight0*c0))
  }

and put mycost as an argument in the cv.glm function.

edited Jun 10 '18 at 17:25

answered Jan 29 '13 at 00:20

Feng Mai

222
1
8

Shouldn't $cost$ be something like $$\sum \Bigl\lfloor \frac{|r_i-p_i|}{0.5}\Bigr\rfloor$$ (with the exception, when $|r_i-p_i|=1$, then the term is $1$, not $2$)? – Mooncrater Aug 08 '17 at 16:06
@feng-mai pi==0 or pi < 0.5? ( and pi==1 or pi > 0.5?) if using 0.5 as the decision boundary. Are not the pi the predicted *probabilities*? – PM. Jun 04 '18 at 09:33
1

@PM Yes you are right. $pi$ are the responses from the glm model. Thanks for the correction. – Feng Mai Jun 10 '18 at 17:26

score 1 · Answer 2 · answered Feb 14 '14 at 06:03

cost <- function(r, pi = 0) mean(abs(r-pi) > 0.5)

First, you have set a cut-off as 0.5. Your r is 0/1, but pi is probability. So individual cost is 1 if absolute error is greater than 0.5, otherwise 0. Then, this function calculates the average error rate. But remember, the cut-off has been set before you define your cost function.

Actually, I think it makes more sense if the choice of cut-off is determined by cost function.

Alex · Answer 3 · 2015-04-07T10:17:37.270

The answer by @SLi already explains very well what the cost function you have defined does. However, I thought I would add that the cost function is used to calculate the delta value from cv.glm, which is a measurement of the cross validation error. However, critically delta is the weighted average of the error of each fold given by the cost. We see this by inspecting the relevant bit of the code:

for (i in seq_len(ms)) {
    j.out <- seq_len(n)[(s == i)]
    j.in <- seq_len(n)[(s != i)]
    Call$data <- data[j.in, , drop = FALSE]
    d.glm <- eval.parent(Call)
    p.alpha <- n.s[i]/n # create weighting for averaging later
    cost.i <- cost(glm.y[j.out], predict(d.glm, data[j.out, 
        , drop = FALSE], type = "response"))
    CV <- CV + p.alpha * cost.i # add previous error to running total
    cost.0 <- cost.0 - p.alpha * cost(glm.y, predict(d.glm, 
        data, type = "response"))
}

and the value returned by the function is:

  list(call = call, K = K, delta = as.numeric(c(CV, CV + cost.0)), 
    seed = seed)

What is the cost function in cv.glm in R's boot package?

3 Answers3