Understanding the output of LIME - How is the contribution of features related to the predicted output?

Question

How is the contribution of features extract from explaining a regression model with LIME locally related to the predicted output of the surrogate model?

I thought that LIME is additive (some blog post as source), but wasn't able to get this additiveness in my example. I'll illustrate my tries:

I explain a Random Forest model via LIME using:

# Create explainer for the Random Forest model
rf_explainer = lime.lime_tabular.LimeTabularExplainer(rf_X_train.values, feature_names=config['rf_features'], class_names=['duration'], mode='regression')

# Explain values
rf_exp = rf_explainer.explain_instance(rf_X_sc.values[0], rf_regressor.predict)
feature_importance = [x[1] for x in sorted(rf_exp.__dict__['local_exp'][0], key=lambda tup: tup[0])]
y_surrogate = rf_exp.__dict__['local_pred'][0]
result = [scenario, 'rf', rf_y_sc.iloc[0], rf_prediction[0], y_surrogate] + feature_importance

The result is:

scenario     sc_1
method       rf
y_real       390.0
y_rf         312.910836
y_surrogate 1846.915013   # <- Surrogate seems to be pretty bad
feature_0    -21.599091
feature_1    -27.415115
feature_2     13.378463
feature_3   -199.55607
feature_4     -7.411741
feature_5   -194.997414
feature_6     -5.433271
feature_7   -334.37682
feature_8    -19.342806

Because LIME uses a linear model as a surrogate, I expected that

-21.599091 + -27.415115 + 13.378463 + -199.55607 + -7.411741 + -194.997414 + -5.433271 + -334.37682 + -19.342806

is y_surrogate. Unfortunately, the sum is -796.753865, which is unequal to 1846.915013.

My next thought was, that I have to subtract -796.753865 from 1846.915013 to come to something like the base value of the surrogate model; but I checked with a second sample that uses the same explainer - it showed a different value.

What am I understanding/doing wrong? Thanks for your help!

Edit: Here is an example that you can easily reproduce - the behavior is the same.

 from sklearn.datasets import load_iris
 from sklearn.model_selection import train_test_split
 import xgboost as xgb
 import lime
 import lime.lime_tabular     
 

 def main():
     random_state = 42
     df = load_iris(as_frame=True)['data']     

     # XGBoost regressor that predicts 'sepal length (cm)'
     target = ['sepal length (cm)']
     features = ['sepal width (cm)', 'petal length (cm)', 'petal width (cm)']
     X_train, X_test, y_train, y_test = \
         train_test_split(df[features], df[target], test_size=0.33, random_state=random_state)     

     regressor = xgb.XGBRegressor(n_estimators=30, random_state=random_state)
     regressor.fit(X_train, y_train)     

     sample_to_explain = X_test.iloc[0:1,:]
     y_test_p = regressor.predict(sample_to_explain)  # Predict first sample in X_test
     xgb_explainer = lime.lime_tabular.LimeTabularExplainer(X_train.values, feature_names=features,
                                                            class_names=target, verbose=True, mode='regression')
     exp = xgb_explainer.explain_instance(sample_to_explain.values[0], regressor.predict)
     feature_importance = [x[1] for x in sorted(exp.__dict__['local_exp'][0], key=lambda tup: tup[0])]
     y_surrogate = exp.__dict__['local_pred'][0]
     print(f'xgb,\n'
           f'Real y: {y_test.iloc[0].values[0]},\n'  # 6.1
           f'Predicted by XGBoost: {y_test_p},\n'  # 6.2832994
           f'Predicted by LIME surrogate: {y_surrogate},\n'  # 6.127684969038174
           f'Feature importances: {feature_importance},\n'  # [0.03934168590785398, -0.5762293534314864, -0.08231280097169107]
           f'Values from sample: {sample_to_explain.values[0]}\n')  # [2.8 4.7 1.2]     

     # 2.8*0.03934168590785398 + 4.7 * -0.5762293534314864 + 1.2 * -0.08231280097169107 = -2.6968966017520244 != 6.127684969038174     
 

 if __name__ == '__main__':
     main()

Adam Kells · Answer 1 · 2021-09-28T14:36:11.967

1

The numbers you get out from LIME will follow a linear model such that:

\begin{equation} y_{surrogate} = x_0+f_1x_1+...+f_nx_n \end{equation}

The equation you have in your question is:

\begin{equation} y_{surrogate} = f_1+...+f_n \end{equation}

I can't say why the $y_{surrogate}$ value is so bad without knowing more about your problem though. It seems like it might be a bug in the code.

edited Sep 28 '21 at 14:36

answered Sep 28 '21 at 10:33

Adam Kells

908
1
12

Previously, I also tried that, but the result was even worse: `6.000000 * -21.599091 + 25.000000 * -27.415115 + 1.000000 * 13.378463 + 119.000000 * -199.55607 + 17.000000 * -7.411741 + -73.943733 * -194.997414 + 40.817074 * -5.433271 + -73.949699 * -334.37682 + 40.822285 * -19.342806 = 13459.748259382872`, so, I didn't even wrote the result in the question. The explanations generated make sense and the model seems to have learned the right thing, I checked that with around 120 other samples. The model predicts seconds and the mean absolute error of the random forest model is around 185s. – So S Sep 28 '21 at 11:43
I think the first step would be to debug why the $y_{surrogate}$ is so bad. It's hard to say why as I can't access any of the inputs of the line which generate that quantity. That's where I would start though. – Adam Kells Sep 28 '21 at 13:14
Thanks! I built some reproducible results and shared the source code in the original question. The behavior is the same. So most likely, I have the same error in both blocks of code. – So S Sep 28 '21 at 14:09
Okay that's great, at least in this example, $y_{surrogate}$ is close to the expected value. I made an edit to my answer as I realised the linear fit will also have some intercept coefficient. As far as I can tell, this number isn't output by LIME. – Adam Kells Sep 28 '21 at 14:37

Understanding the output of LIME - How is the contribution of features related to the predicted output?

1 Answers1