2

I m newbee in deep learning and I try to understand algorithms and their statistical logic. I could not understand let us we have two algorithms A and B with 75% and 78% test set accuracies respectively.
Is that mean Algorithm A is significantly better than Algorithm B if the test set contains 100 instances?

Daniel
  • 21
  • 1
  • Set aside the source of your proportions: you want to compare 75/100 and 78/100. What statistical methods do you know for comparing proportions? (We’ll have another discussion someday about the dangers of using accuracy as your performance metric, but perhaps it’s okay for getting your feet wet when you’re first learning.) – Dave Jun 06 '20 at 13:18
  • so algorithm A is significantly better than B ? – Daniel Jun 06 '20 at 14:30
  • *significant* is a loaded word in statistics. What do you mean by it? – Aksakal Jun 08 '20 at 00:23

1 Answers1

1

Please see Test if two binomial distributions are statistically different from each other

The keywords for your problem, if you want to search other sources are:

  • binomial test - which is the usual statistical test for proportions (in your case the accuracy)

  • 2 sample - because you want to compare two proportions, and the binomial test usually compares only one sample to a fixed proportion.

Jacques Wainer
  • 5,032
  • 1
  • 20
  • 32