1

A problem I encountered a few times and I always had problem answering my boss's question:"Are you sure it is fixed?"

I would really like to be able to give an answer along the lines :

"Before the fix, the app would crash after an average time of 5 minutes over 10 run. I ran 5 tests of 10 minutes without crash, so I am 99.99% confident that I fixed it"

I would prefer to have hard data than gut feeling... How can I compute such a statement?

Context

I had an app crash after 1-5 minutes sitting idle. I found and fixed a race condition causing a pointer to be freed twice. How long and how many times should I run my app to be pretty sure that I fixed the problem and not that I'm just being lucky.

Vincent Hubert
  • 111
  • 1
  • You can justify such a statement (assuming only that the tests were independent) provided you replace "99.99%" by "76%." The latter approximates the number $1 - (1 - (5/10)^2)^5.$ This is a consequence of [Chebyshev's Inequality](https://en.wikipedia.org/wiki/Chebyshev%27s_inequality) and the variance inequality obtained at https://stats.stackexchange.com/questions/45588. – whuber Nov 08 '18 at 16:21

0 Answers0