2

“The variation achieved a 20% increase in conversions with 90% significance.” Unfortunately, this isn’t equivalent to, “there is a 90% chance of a 20% increase.” So what does it actually mean?

20% is the increase we observed in our sample, and, if we had to guess, it’s the increase we’d expect to see if we continued the test indefinitely. It’s the most likely outcome. But there is not a 90% chance of a 20% increase, or even a 90% chance of an increase of at least 20% or approximately 20%.

90% is the chance that there would be a difference at all. In other words, if we had ten tests producing this result, and we decided to continue all ten tests indefinitely, we’d expect one to end up with the original and variation converging to identical conversion rates. Of the remaining nine tests, some might end up showing an increase of far less than 20%. Others might end up showing an even bigger increase.

I somehow feel that the 2nd paragraph is a poor explanation or maybe even wrong. The way I look at it, in simple words, it means there is only a 10% probability that the 20% variation was due to chance.

Original Link here

Siddharth Gopi
  • 1,395
  • 1
  • 12
  • 22
  • 1
    (+1) Very closely related--it probably already has some good answers--is http://stats.stackexchange.com/questions/26450/why-does-a-95-ci-not-imply-a-95-chance-of-containing-the-mean. The third paragraph is amazingly wrong. – whuber Mar 23 '15 at 19:17

1 Answers1

2

Indeed, the second paragraph is wrong.

Probably a better way to rephrase the initial sentence would be: "The variation achieved a 20% increase in conversions. This increase was significant at the 0.1 level."

This means that if there was no real difference at all (increase = 0%), you would still expect 10% of the times you repeat your sample to get an increase at least as large as that 20% by chance.

Aghila
  • 657
  • 5
  • 18