Adding weights to logistic regression for imbalanced data

Question

I want to model a logistic regression with imbalanced data (9:1). I wanted to try the weights option in the glm function in R, but I'm not 100% sure what it does.

Lets say my output variable is c(0,0,0,0,0,0,0,0,0,1). now I want to give the "1" 10 times more weight. so I give the weights argument weights=c(1,1,1,1,1,1,1,1,1,1,1,10).

When I do that, it will be considered in the calculation of the maximum likelihood. Am I right? misclassification of "1" is just 10 times worse then missclassifying a "0".

score 16 · Answer 1 · edited Apr 08 '19 at 21:39

16

Weighting is a procedure that weights the data to compensate for differences in sample and population (King 2001). For example, in rare events (such as fraud in credit risk, deaths in medical literature) we tend to sample all the 1’s (rare events) and a fraction of 0’s (non events). In such cases we have to weight the observations accordingly.

Example: Let us say, In a population of 500,000 transactions there are 50 fraud transactions. In this case you would

Sample all 50 frauds transaction (100% of the fraud)
10% of the good transactions (10% of 500,000 is 50,000 good transactions)

In this case you would assign a weight of 1 for fraud transactions and a weight of 10 for good transactions. This is called the Weighted Maximum Likelihood method. The important takeaway is that the weighting is related to sampling proportions

Refer: Logistic Regression in Rare Events Data (King 2001)

edited Apr 08 '19 at 21:39

Community

1

answered Aug 05 '15 at 01:38

subra

791
3
8

1

hi subra!!! thank you very much for the King approach!! haven't heard of it! in my case i have 1 million transactions! ( 900.000 are "0", and 100.000 are "1"). so should i sample 10% of my "0"? then i have almost a balanced data set. then i have to weight the "0" ten times more than the "1" right? and the function in R glm() in the MASS package exactly does that right? if i weight my observations, i'll calculate the weighted maximum likelihood? thank you! really appreciate your answer and help – ching Aug 05 '15 at 13:44
i really think a lot about this problem. what if i say: now use all my data to build a logit model (with the 9:1 unbalanced data). and then then i weigh my "1" ten times, even tho in reality i don't have more data and it is NOT 10% of my data. it's just like, i act like i have..... so now when R calculates the model, it thinks i only use 10% of my "1" and considers it in the calculation of the likelihood. does that make any sense? – ching Aug 05 '15 at 14:36

score 15 · Accepted Answer · edited Jul 24 '17 at 07:44

Ching, You do not have to make your data set balanced in terms of 1’s and 0’s. All you need is sufficient number of 1’s for the maximum likelihood to converge. Looking at the distribution of 1’s (100,000) in your dataset, you should not have any problems. You can do a simple experiment here

Sample 10 % of the 1’s and 10% of the 0’s and use a weight of 10 for both
Sample 100% of the 1’s and 10% of the 0’s and use a weight of 10 for the 0’s

In both cases, you will get identical estimates. Again the idea of weighting is related to sampling. If you are using the whole data set you should not weight it. If I were you I would just use 10% if 1's and 10% of 0's.

In R, you would use glm. Here is a sample code:

glm(y ~ x1 + x2, weights = wt, data =data, family = binomial("logit"))

In your dataset there should be a variable wt for weights.

If you use 10% of both 0's and 1's, your wt variable will have a value of 10.

If you use 10% of the 0's and 100% of 1's: wt variable will have a value of 10 for observations with y=0 and 1 for observations with y=1

Adding weights to logistic regression for imbalanced data

2 Answers2

Linked