0

I am implementing machine learning algorithm for multiclass classification problem in R programming. The problem is that when I predict the accuracy I am getting around 90% accuracy but when I calculate the AUC score I am getting more than 1. Will the AUC support multi-class classification?

Which is the better approach to consider the evaluation metric of machine learning model for multi class classification?

Below is my code 

library(dplyr)
smp_size <- floor(0.80 * nrow(New_data))
set.seed(123)
train_ind <- sample(seq_len(nrow(New_data)), size = smp_size)
train <- New_data[train_ind, ]
test <- New_data[-train_ind, ]

train_features <- train[,-ncol(train)]
train_labels <- train[,ncol(train)]
test_features <- test[,-ncol(test)]
test_labels <- test[,ncol(test)]

library(e1071)
library(rpart)
library(mlbench)
svm.model <- svm(as.factor(train$labels) ~ ., data = train,cross = 10)
class(svm.model)
summary(svm.model)
print(svm.model)
svm_pred <- predict(svm.model,test_features)

length(svm_pred)
length(test_labels)
table(pred = svm_pred, true = t(test_labels))
conf_matrix <- table(svm_pred, test_labels)
conf_matrix 

library(MLmetrics)
Accuracy(svm_pred, test_labels)
AUC(svm_pred, test_labels)

Output:

> conf_matrix
        test_labels
svm_pred    0    1    2    3    4
       0  896   75   50   18   28
       1   71  919   28   13   16
       2    7    6  112    1    9
       3   31   44   12 1023    2
       4    1    5    9    0   55
> library(MLmetrics)
> Accuracy(svm_pred, test_labels)
[1] 0.8758379
> AUC(svm_pred, test_labels)
[1] 1.047074
Tim
  • 108,699
  • 20
  • 212
  • 390
vinaykva
  • 389
  • 1
  • 4
  • 19
  • Please refer to the documentation: https://cran.r-project.org/web/packages/MLmetrics/MLmetrics.pdf the `AUC` functions calculates AUC for **binary** problems, while you have multiple categories. It is a bug that the function does not throw an error. – Tim Sep 11 '17 at 19:02
  • @Tim How can this be off topic. If AUC has a bug I was looking for an alternative approach for considering the metric evaluation of my machine learning model which has a multi class problem. – vinaykva Sep 11 '17 at 19:07
  • Please check https://stats.stackexchange.com/help/on-topic Software or programming related questions are off-topic in here. – Tim Sep 11 '17 at 19:11
  • @Tim My question is related to machine learning on how to evaluate the performance of the machine learning model which is part of the machine learning. – vinaykva Sep 11 '17 at 19:15
  • Please read carefully the help pages. This question in this form is off-topic. – Tim Sep 11 '17 at 19:27
  • @Tim I updated the question – vinaykva Sep 11 '17 at 19:33
  • Let us [continue this discussion in chat](http://chat.stackexchange.com/rooms/65443/discussion-between-vinay-and-tim). – vinaykva Sep 11 '17 at 19:33
  • 2
    The question is still off-topic. The previous version was more relevant: you got AUC > 1 since you used your software incorrectly. This is not a statistical issue, but it is just about using your software. AUC can be implemented for multi-label problems, see https://stats.stackexchange.com/questions/21551/how-to-compute-precision-recall-for-multiclass-multilabel-classification – Tim Sep 11 '17 at 20:08
  • 1
    If you set aside the code related issues, is your question answered by [How to plot ROC curves in multiclass classification?](https://stats.stackexchange.com/q/2151/7290) – gung - Reinstate Monica Sep 11 '17 at 20:24
  • yes I got the answer from Tim's shared link – vinaykva Sep 11 '17 at 20:27

0 Answers0