Questions tagged [confusion-matrix]

A confusion matrix is a contingency table used to evaluate the predictive accuracy of a classifier. Confusion matrix is the 2x2 frequency table with counts "True positive", "True negative", "False positive", "False negative", relating classifying to a class of interest vs. else class. But in a broader sense, any frequency kxk crosstabulation "Predicted" x "Actual" classes can be called a confusion matrix, in the context of evaluation of a classifier.

A confusion matrix is a special contingency table used to evaluate the predictive accuracy of a classifier. Predicted classes are listed in rows (or columns) & actual classes in columns (or rows), with counts of the cases in each combination listed in each cell. All cases represented along the main diagonal are accurately classified, while the off-diagonal elements are misclassified. Inspection of the confusion matrix can identify which classes tend to be 'confused' for each other. The confusion matrix also allows the calculation of model performance metrics such as sensitivity and specificity, precision and recall, positive and negative predictive value, etc.

Here is an example confusion matrix from a model of Fisher's iris data with good accuracy. The model occasionally confuses versicolor and virginica, but never misclassified either as setosa.

                      Actual:
             setosa versicolor virginica 
Predicted:
  setosa         50          0         0  
  versicolor      0         47         3  
  virginica       0          4        46 
267 questions
34
votes
3 answers

How can I interpret a confusion matrix

I am using confusion matrix to check the performance of my classifier. I am using Scikit-Learn, I am little bit confused. How can I interpret the result from from sklearn.metrics import confusion_matrix >>> y_true = [2, 0, 2, 2, 0, 1] >>> y_pred…
user3378649
  • 1,107
  • 4
  • 13
  • 22
30
votes
2 answers

FPR (false positive rate) vs FDR (false discovery rate)

The following quote comes from the famous research paper Statistical significance for genome wide studies by Storey & Tibshirani (2003): For example, a false positive rate of 5% means that on average 5% of the truly null features in the study…
25
votes
4 answers

Given true positive, false negative rates, can you calculate false positive, true negative?

I have values for True Positive (TP) and False Negative (FN) as follows: TP = 0.25 FN = 0.75 From those values, can we calculate False Positive (FP) and True Negative (TN)?
Simplicity
  • 535
  • 1
  • 7
  • 12
16
votes
3 answers

How to build a confusion matrix for a multiclass classifier?

I have a problem with 6 classes. So I build a multiclass classifier, as follows: for each class, I have one Logistic Regression classifier, using One vs. All, which means that I have 6 different classifiers. I can report a confusion matrix for each…
16
votes
2 answers

Using the caret package is it possible to obtain confusion matrices for specific threshold values?

I've obtained a logistic regression model (via train) for a binary response, and I've obtained the logistic confusion matrix via confusionMatrix in caret. It gives me the logistic model confusion matrix, though I'm not sure what threshold is being…
Black Milk
  • 356
  • 1
  • 3
  • 12
15
votes
3 answers

Relationship between the phi, Matthews and Pearson correlation coefficients

Are the phi and Matthews correlation coefficients the same concept? How are they related or equivalent to Pearson correlation coefficient for two binary variables? I assume the binary values are 0 and 1. The Pearson's correlation between two…
14
votes
1 answer

Confidence interval of precision / recall and F1 score

To summarise the predictive power of a classifier for end users, I'm using some metrics. However, as the users input data themselves, the amount of data and class distribution varies a lot. So to inform users about the strength of the metrics, I'd…
9
votes
3 answers

What is no ' information rate ' algorithm?

I plan to implement ' no information rate ' as part of summary statistics. This statistic is implemented in r (Optimise SVM to avoid false-negative in binary classification) but not in Python (at least I cannot find a reference) . Is there a…
blue-sky
  • 609
  • 1
  • 7
  • 17
8
votes
1 answer

What does it imply when the sensitivity = 1.000 and specificity = 0.000?

What adjustments do I need to make when I have extreme values in the confusion matrix as stated above i.e sensitivity = 1 and specificity = 0?
Moses
  • 247
  • 7
8
votes
1 answer

Is there any difference between Sensitivity and Recall?

In most of the places, I have found that sensitivity=recall. In terms of the Confusion Matrix, the formula for both of these is the same: $TP/(TP+FN)$. Is there any difference between these two metrics? If not, then why does the same thing has a…
7
votes
2 answers

Is there something like a confusion matrix for a probabilistic score rather than categories?

Imagine we have pictures of three animals: dogs, cats, and horses. We train our image classifier and get a confusion matrix, noticing that the model tends to predict that dogs are horses. But then we read Cross Validated and learn that…
7
votes
1 answer

Can F1-Score be higher than accuracy?

I'm using sklearn's confusion_matrix and classification_report methods to compute the confusion matrix and F1-Score of a simple multiclass classification project I'm doing. For some classes the F1-Score that I'm getting is higher than the accuracy…
crash
  • 299
  • 2
  • 8
7
votes
2 answers

How to threshold multiclass probability prediction to get confusion matrix?

Lets say my multinomial logistic regression predict that a chance of a sample belonging to a each class is A=0.6, B=0.3, C=0.1 How do I threshold this values to get just binary prediction of a sample belonging to a class, taking in to an account…
6
votes
3 answers

How can I derive confidence intervals from the confusion matrix for a classifier?

I have am using k-fold cross validation to generate a confusion matrix for a classifier. I need to calculate 95% confidence intervals for the number of times each class is predicted when run against a bunch of input data. So if my output after…
5
votes
0 answers

Confusion matrix for multilabel classification

I know that a similar subject was treated here, but my question is a little bit different. I have a result of multilabel classification, like this (2 observations, 3 labels in the example, in practice I have 10k observations and 300 labels): >…
Tau
  • 81
  • 1
  • 5
1
2 3
17 18