2

I'm new to CV, so please excuse any faux pas. I've collected sample data from a population and would like to discern whether or not the experimental group outperforms the control group. Let's say I have 10,000 observations within the experimental group and saw 4500 successes. Let's also say that I observed a 47% success rate in my control group (so if my control group is 5% of the total sample size, that would be 235 successes out of 500 trials).

Now, am I right in my usage and interpretation of the following?:

prop.test(4500,10000,.47,alt="less")

data:  4500 out of 10000, null probability 0.47
X-squared = 15.9776, df = 1, p-value = 3.205e-05
alternative hypothesis: true p is less than 0.47
95 percent confidence interval:
 0.0000000 0.4582455
sample estimates:
   p 
0.45

Since the p-value is far less than .05, I can safely reject the null hypothesis that that the experimental group will perform as well or better than the control group. Also, I know that 95% of the time, the probability of success for the experimental group will fall between 0% and ~45.8%.

Also, is the prop.test function a good indicator of sufficient sample size? If not, what is?

Thanks!

userNaN
  • 241
  • 1
  • 2
  • 9
  • 1
    Consider these questions: (1) If someone else were to independently sample $500$ members of the control population, would they likely have observed exactly $235$ successes? (Probably not, so how much do you suppose their number of successes would deviate from your number?) (2) Where in your code have you accounted for that sampling variability? – whuber Jun 19 '14 at 19:05
  • Really good questions. I suppose I don't really account for sampling variability, right? I drew what I thought was a fairly large random sample, but I'm not certain if I can now draw any statistically significant/accurate conclusions from it. I'd still like to know if I'm using `prop.test` correctly, but if I'm missing some important prerequisites please point me in the right direction. Thank you! – userNaN Jun 19 '14 at 20:10
  • It seems your real question is less about `prop.test` and more about understanding how to test for a difference in proportions, with a hope to then use `prop.test` or something like it once you figure out what you're trying to do. The hard part isn't the '*how do I do it in prop.test*?' part. – Glen_b Jun 20 '14 at 00:55

1 Answers1

3

Your interpretation is incorrect.

"Since the p-value is far less than .05, I can safely reject the null hypothesis that that the experimental group will perform as well or better than the control group. "

No. What you did was this: you "tested" if the p from the experimental group is greater or equal 0.47. But you don't know if the p of the control group is actually 0.47. So you could say that the evidence favors the hypothesis that the proportion in the experimental group is less than 0.47, but you could not say that the evidence shows that it is less than the proportion of the control group, which is uncertain.

Actually, just for illustration, assuming the values stated on the question, you wouldn't reject the hypothesis that the proportions are different with prop.test:

success <- c(4500, 235) ## successes in experimental and control
n <- c(10000, 500) ## total observations in experimental and control
prop.test(success, n)

2-sample test for equality of proportions with continuity correction

data:  success out of n
X-squared = 0.6907, df = 1, p-value = 0.4059
alternative hypothesis: two.sided
95 percent confidence interval:
 -0.06587065  0.02587065
sample estimates:
prop 1 prop 2 
  0.45   0.47

Notice that once you account for the uncertainty in the other proportion the p-value raises to 0.4.

As for the second sentence:

"Also, I know that 95% of the time, the probability of success for the experimental group will fall between 0% and ~45.8%."

This is a common misconception. What you can say is that the procedure you used constructs confidence intervals that would cover the true value of p 95% of the time. But this is not the same as claiming that 95% of the time you will estimate p on the range of the confidence interval, neither that the true p is inside the confidence interval with 95% probability. This other question may help on this topic.

Community
  • 1
Carlos Cinelli
  • 10,500
  • 5
  • 42
  • 77
  • Thanks for your explanation. So how do I discern the `p` for the control group so I can use it in my analysis? Or perhaps a better question is: Is the `prop.test` function really what I want here, or is there a simpler method that I should start with? – userNaN Jun 20 '14 at 14:29
  • We need more information about the experiment to make a better decision, but it seems that `prop.test` would be an adequate first approximation to the problem. You just have to use the information of the successes and sample size of both the control and experiment group, as illustrated on the answer above. – Carlos Cinelli Jun 20 '14 at 16:23