9

I plan to implement ' no information rate ' as part of summary statistics. This statistic is implemented in r (Optimise SVM to avoid false-negative in binary classification) but not in Python (at least I cannot find a reference) .

Is there a canonical reference that I can refer to so as to implement this algorithm ?

I've searched Wikipedia and various Google searches but have not found a reference.

Update :

Reading caret doc https://cran.r-project.org/web/packages/caret/caret.pdf

" The overall accuracy rate is computed along with a 95 percent confidence interval for this rate (using binom.test) and a one-sided test to see if the accuracy is better than the "no information rate," which is taken to be the largest class percentage in the data. "

blue-sky
  • 609
  • 1
  • 7
  • 17
  • 8
    In R's `caret` package `confusionMatrix` function (which is what was used in the example you linked to), the No Information Rate is just the largest class percentage in the data (this is explained in the help for the function, which you can access by typein `?confusionMatrix` in the R console). The idea is that a useful model should do better than you could do by always predicting the most common class. – eipi10 May 19 '17 at 17:26
  • 1
    @eipi10 thanks , does this mean ' no information rate ' term originated with r ? – blue-sky May 19 '17 at 17:44

3 Answers3

8

Suppose that you have response $y_i$ and covariates $x_i$ for $i = 1 ...n$, and some loss function $\mathcal{L}$. The no information error rate of a model $f$ is the average loss of $f$ over all combinations of $y_i$ and $x_i$:

$${1 \over n^2} \sum_{i=1}^n \sum_{j=1}^n \mathcal{L}\left(y_i, f(x_j)\right)$$

If you have a vector of predictions predicted and a vector of responses response, you can calculate the no info error rate by generating all the combinations of predicted and response and then evaluating some function loss on these resulting vectors.

In R, supposing RMSE loss, (using the tidyr library) this looks like:

predicted <- 1:3
response <- 4:6
loss <- function(x, y) sqrt(mean((x - y)^2))

combos <- tidyr::crossing(predicted, response)
loss(combos$predicted, combos$response)

In Python this looks like

import numpy as np

predicted = np.arange(1, 4)
response = np.arange(4, 7)

combos = np.array(np.meshgrid(predicted, response)).reshape(2, -1)

def loss(x, y):
    return np.sqrt(np.mean((x - y) ** 2))

loss(combos[0], combos[1])
alexpghayes
  • 465
  • 5
  • 14
  • 2
    Although implementation is often mixed with substantive content in questions, we are supposed to be a site for providing information about statistics, machine learning, etc., not code. It can be good to provide code as well, but please elaborate your substantive answer in text for people who don't read this language well enough to recognize & extract the answer from the code. – gung - Reinstate Monica May 02 '18 at 20:15
  • Finally had some time to update the answer. Let me know if this still needs changes? – alexpghayes May 20 '18 at 02:11
  • expand.grid can be used in place of tidyr::crossing for a base solution – Maxwell Chandler Jan 21 '19 at 18:45
  • 1
    Further discussion of this equation (and topic) can be found in "Elements of Statistical Learning, 2nd edition", see equation 7.58. https://web.stanford.edu/~hastie/ElemStatLearn/ – Brian D May 22 '19 at 20:33
1

The no information error rate is the error rate when the input and output are independent. You can compute it by evaluating the prediction rule on all possible combinations of the target and the features, i.e as

$$\hat \gamma = \frac{1}{N}\sum_{i=1}^N\sum_{j=1}^NL\left(y_i, \hat f(x_j)\right).$$

Simon Boge Brant
  • 595
  • 2
  • 13
0

No information rate is Naive classifier which needs to be exceeded in order to prove that model we created is significant. We calculate accuracy and then compare it with Naive classifier. accuracy should be higher than No information rate (naive classifier) in order to be model significant.

Roshan
  • 1