4

If the results of a Chi-Square test give a P-Value of 0.01 then can we say that the confidence level in their being a difference is (1-0.01) = 99% confidence.

Background. We are conducting website landing page tests for our clients. We want to be able to say "We are 99% confidence that version B performed better than version A".

The folks who run this website use his type of terminology: http://getdatadriven.com/ab-significance-test

Is it right to say this with the example I gave?

Glen_b
  • 257,508
  • 32
  • 553
  • 939
Doug Fir
  • 1,218
  • 1
  • 13
  • 27
  • 4
    No, no, no! Check the [definition of p-values](http://stats.stackexchange.com/questions/31/what-is-the-meaning-of-p-values-and-t-values-in-statistical-tests). – Tim Jan 19 '15 at 21:25
  • @Tim useful link I read over that question and it makes sense. What I'm really looking for is a definition of "Confidence" and when am I allowed to use that word. I'm guessing this word is over used and abused in stats. Using the example above, if one were to weave the string "We are __ confident in the result" or something similar, is this even feasible? Could we say "We are 99% confident that the variation observed is real and not due to chance"? – Doug Fir Jan 19 '15 at 21:40
  • not with frequentist statistics... no. Search on here for interpretation of confidence interval. – John Jan 19 '15 at 21:59
  • I will do that. I'm getting work pressure to formulate a sentence that uses "confidence" sounds like I cannot – Doug Fir Jan 19 '15 at 22:16

2 Answers2

8

Is "Confidence Level" just 1 minus P-Value?

No.

If the results of a Chi-Square test give a P-Value of 0.01 then can we say that the confidence level in their being a difference is (1-0.01) = 99% confidence.

This is misusing the terms. Words like "confidence" and "p-value" have a specific meaning in frequentist statistics.

If you consider their meanings, it becomes clear why the phrasing you want to use would be wrong.

The p-value is the probability of a result at least as extreme as the observed one, given the null hypothesis is true.

1-pvalue would then be "the probability of a result less extreme than the observed one, given the null hypothesis is true.

That is not remotely the same thing as the probability that the alternative is true given the observed result, which is presumably the kind of thing you are trying to talk about.

We are conducting website landing page tests for our clients. We want to be able to say "We are 99% confidence that version B performed better than version A".

You can make a more-or-less similar statement to that from a Bayesian framework, but its size won't be related to p-values in a direct way.

The folks who run this website use his type of terminology: http://getdatadriven.com/ab-significance-test

Don't accept everything you read on the internet uncritically (that would include answers on stats.stackexchange.com, but at least there's some degree of critical peer-review, which should help with feeling confident that mostly it isn't nonsense). However, from what I see on the page you link to, they don't misuse statistical terms quite in the way you suggest, but misuse them in a somewhat different way (using '99% certain' rather than '99% confident', presumably making it more explicitly a probability statement that's intended, and without any suggestion of a relation to confidence intervals).

Statistical terms are misused all the time on the internet. [I'm fairly sure one could find links to support almost every single wrong idea raised in questions here.]

Is it right to say this with the example I gave?

Not by any reasonable interpretation of the word 'right', given the context.

Glen_b
  • 257,508
  • 32
  • 553
  • 939
  • 2
    Thank you for the detailed answer. I think there's a persistent pressure in the work place to misuse statistical terms for ease of understanding. This is why I posted, to clarify things and this has helped. – Doug Fir Jan 20 '15 at 03:33
  • Unfortunately, no matter how much we might wish to, p-values simply don't give us any basis at all for making statements of the desired form. If there's a strong imperative to make statements of that form, you don't want to use the usual form of hypothesis testing, since it doesn't tell you what you need. Bayesian statistics comes quite a lot closer to it (but of course, there's the issue that many people are less familiar with it). – Glen_b Jan 20 '15 at 04:52
3

With Bayesian statistics, you could conceivably turn it into a statement about "99% confident that version B performed better than version A." However, neither the p-value nor the 99% confidence interval will let you conclude that. See Why does a 95% Confidence Interval (CI) not imply a 95% chance of containing the mean? for a discussion of the meaning of a confidence interval.

With a p<.01, it would be more precise to say that "chance alone would create these differences between version A and B less than 1% of the time." Of course, I realize that's probably not as persuasive in the real-world.

Community
  • 1
Anthony
  • 1,564
  • 12
  • 24