4

I tried searching the site but nothing came up.

I have a simple situation for a business problem - We have a population of 50 million files. We want to review a sample and see whether a file contains or does not contain "x". So it's binary. We will calculate a proportion of yes:no.

We want to be able to quantify how accurate our estimate is of the true population. "We are 95.. 99% confident that the true proportion is "x/y" +/- x%. From my research, confidence intervals cannot be interpreted in such a way. Rather, confidence intervals are supposed to be interpreted as "in the long-run, if we conduct many experiments, we are 95% confident that the CIs calculated for those samples will contain the true population parameter". Do we just take an arbitrary "large enough" random sample, calculate the proportion and assume that is the true proportion? I came across the binomial credible interval that can be interpreted as the former, but I know nothing of Bayesian inference unfortunately.

DataNoob7
  • 71
  • 6
  • 2
    Do a standard binomial CI for starters. Also, 50M files does sound a lot so yeah, not going to be a 10" `grep`, but maybe you can still brute-force it intelligently within an hour or two. Why do inference when we can know an exact answer? If the files have time-stamps it might be more relevant to even look at the proportion per month/week/day and compare. That would be more relevant variations (e.g. "on any given day we saw Y% of file having the pattern of question, with an IQR of [y_1,y_2]" a statement someone with minimal Stats background can follow). – usεr11852 Dec 21 '21 at 16:14
  • (also how many files out of the 50M are examined? Realistically, unless our prior is *really* strong, after the first couple of thousand sampled files it will become mostly irrelevant; I would focus more on the idea that we don't bias our sample by accident more than anything else) – usεr11852 Dec 21 '21 at 16:22
  • We don't know if each file contains "x" without manually going into the files. We want to be able to take a sample and estimate the proportion of "yes:no" for the entire population. – DataNoob7 Dec 21 '21 at 16:22
  • Sample calcs online say we need to review 384 files, which seems off (probably due to the nature of the population being so large). I figure it will be a random sample of a couple of thousand. Also my knowledge of sample size calcs is when the DV is continuous, not binary. But the other issue is trying to provide some level of assurance that the proportion we observe for the sample is an estimate of the population within X%, x range, etc. For this, I understand confidence intervals are commonly misinterpreted to say this, so I need another method (if it exists). – DataNoob7 Dec 21 '21 at 16:27
  • This sounds like primarily a technology problem to be honest, having tens of millions of samples and being able to check them only manually seems highly problematic, what are these, papyri? (I kid) Check the R package `LearnBayes` to get an idea how a Bayesian binomial proportion CI would behave like. As I said, if you are planning to do "*a random sample of a couple of thousand.*" then you don't have to worry about misinterpretation. The paper you link to is focusing on samples sizes in the lower hundreds (at best). – usεr11852 Dec 21 '21 at 16:34
  • Sorry, when I say misinterpretation, I am referring to frequentist confidence intervals, not credible intervals. I guess what it boils down to is if we do a random sample of 3000, is the assumption that the 50M population has that same sample proportion reasonable? Is there a theorem I can point to give this rationale of 3000 some validity? – DataNoob7 Dec 21 '21 at 16:54
  • 1
    No worries, we are good! :) See Greg's answer below. "*we do a random sample of 3000, is the assumption that the 50M population has that same sample proportion reasonable?*" If it is a random sample yes. – usεr11852 Dec 21 '21 at 16:59
  • You seem to have replaced your *objective* by a *method.* Wouldn't it be correct to state that what you *want* to do is *estimate the proportion of $x$ in the population to within a specified precision with a specified level of confidence* and that your proposed method for doing that will be to obtain some kind of sample of that population? When the sample is a simple random sample (with or without replacement as you choose) then this sounds like a classic binomial or hypergeometric estimation problem. A key fact is that the needed sample size has little to do with the population size. – whuber Dec 21 '21 at 22:32
  • @Whuber sorry I missed your comment. I think where my struggle lies in how to interpret the "frequentist confidence interval" vs the "Bayesian credible interval" based on other threads on this site and other professionals I have worked with. The former appears to be "in the long-run, if we took many (infinite) samples, we can be 95% that the CI for those samples will contain the true population parameter". Where as the latter is "we can be 95% confident that the population (in this case proportion) is within x%. Is the binomial estimation a confidence interval or credible interval? – DataNoob7 Dec 22 '21 at 02:05
  • The first thing you need to do is formulate a clear problem. That admits of multiple types of solutions. A Bayesian solution is available, and to be preferred, when you have an objective, defensible proposal for a prior distribution on the parameter. When such an assumption might be disputed or difficult to justify, you can compute a confidence interval. That does *not* require the long-run interpretation you have described. In short, resist confusing a statistical method with a philosophy of probability. – whuber Dec 22 '21 at 14:33
  • I think this is what I was grappling with. Is it correct to state that the "long-run" interpretation is the definition of probability from the frequentist pov? This does not change the fact that we are still able to say "with 95% confidence, the population parameter is within the L and U calculated from the sample" as you mentioned above. – DataNoob7 Dec 22 '21 at 15:05

5 Answers5

7

This type of problem is well-suited to a confidence/credible interval. It is unclear to me why you think a confidence interval is unsuitable. You ought to be able to get a reasonable inference for the true proportion using a Wilson score interval. You can compute this confidence interval (for a finite or infinite population) using the CONF.prop function in the stat.extend package in R. Here is an example where I use some randomly generated mock data to get the interval:

#Set significance level and population size
#In this case the population size is large so it could be ignored if desired
N     <- 50000000
ALPHA <- 0.05

#Generate some (mock) data
p <- 0.03
n <- 3000
set.seed(1)
DATA <- sample(c(0,1), size = n, replace = TRUE, prob = c(1-p,p))

#Compute the confidence interval
stat.extend::CONF.prop(alpha = ALPHA, x = DATA, N = N)

        Confidence Interval (CI) 
 
95.00% CI for proportion for population of size 5e+07 
Interval uses 3000 binary data points from data DATA with sample proportion = 0.0303 

[0.0247716637647717, 0.0370961961781321]
Ben
  • 91,027
  • 3
  • 150
  • 376
  • Agreed. (+1). I like `binomial::binom.confint` more cause it has all the different types too. (Thus you can really see that with 2000+ samples pretty much any way you cut them the CIs are very similar.) – usεr11852 Dec 21 '21 at 21:23
  • @Ben I am referring to the Frequentist definition of confidence interval. From my understanding, the Frequentist definition of a CI relates to the methodology - That is, if I conduct my CI methodology on 100...1000...100000 random samples, I can expect 95% of the CIs calculated will contain the true population parameter. What I cannot say is: "I am 95% certain that the CI I calculated for the sample contains the true population parameter." So, from my understanding as others have pointed out, I need to use a Bayesian approach to develop a credible interval (not confidence interval). – DataNoob7 Dec 21 '21 at 22:21
  • @DataNoob7: (cause you said "others") My position from the start was "*Do a standard binomial CI for starters.*" If anything I argued that the Bayesian posterior CI, given the absence of strong priors, will be dominated by the likelihood of the data (i.e. the "frequentist" part) so any difference will be immaterial. The general point that both me and Greg had was that you need to learn some Bayesian Inference - especially given you try delve into the interpretation more.) – usεr11852 Dec 21 '21 at 22:29
  • Yes. My apologies for glossing over your opinion in broad strokes. If a frequentist approach is what I need to do, and makes more sense, then my issue lies in the interpretation of said CI as I noted in my comment above. If I am mistaken regarding the interpretation of the CI please let me know. This is also discussed [here](https://stats.stackexchange.com/questions/2356/are-there-any-examples-where-bayesian-credible-intervals-are-obviously-inferior-t) – DataNoob7 Dec 21 '21 at 22:52
  • @DataNoob7, I have amended my answer in this thread to address your concern about interpreting a confidence interval. I also recommend you read my answer in the thread you linked to. – Geoffrey Johnson Dec 22 '21 at 00:45
  • @usεr11852: It is nice to have the other method options, but Wilson score is already pretty solid. The main advantage of ```stat.extend::CONF.prop``` is that it allows you to specify a finite population size and it also lets you choose if you want to make inferences about the full population or only the unsampled part. – Ben Dec 22 '21 at 01:01
  • Thanks GeoffreyJohnson, I will take another look. @Ben what are your thoughts on my reply to your initial comment? – DataNoob7 Dec 22 '21 at 01:23
  • Yeah, of course. It definitely has its merits. As said, I use `binomial::binom.confint`mostly to ensure that all methodology give qualitatively similar results, as soon as I notice some divergence I start thinking of the why and if/how I need to account for that. (e.g. follow-up with the use of priors/posteriors, etc.) – usεr11852 Dec 22 '21 at 03:16
  • @usεr11852: That's a very sensible approach --- I guess I am just too much of a Wilson partisan! – Ben Dec 22 '21 at 11:54
5

The short and best advice is for you to learn Bayesian inference.

The linked paper makes some very good points and gives some good recommendations, but those are not the only possible recommendations. Rejecting any use of confidence intervals based on the paper is like "throwing out the baby with the bathwater". There are definite problems with how some confidence intervals are interpreted, and using the wrong confidence interval procedure can lead to meaningless intervals (but using the wrong Bayesian credible interval procedure can also lead to meaningless inference).

The common approximation methods for confidence intervals for a proportion do not work very well when the true proportion is near 0 or 1. My preferred interval in that case (and even when not near 0 or 1) is a Bayesian credible interval (or even full posterior), but you should understand what assumptions are being made and what the interval means, not just plug numbers into a formula.

You can investigate the various confidence/credible/compatibility interval procedures that work well for your likely situation and choose the one that works best (but understand what they are really saying, what assumptions are being made, and what their limitations are).

Greg Snow
  • 46,563
  • 2
  • 90
  • 159
  • 1
    I agree Greg (+1) just my commentary is that given there is no prior here (or a weak one in any case from what OP says) we are gilding the lily. With a few thousand samples any non-really strong prior will be washed away from the likelihood and what we are left is a superficial conversation if a minor difference between confidence and credible intervals is meaningful or just arguing for arguments sake. – usεr11852 Dec 21 '21 at 17:04
  • Thanks you both for the information. Obviously, I am very green when it comes to this. I am unsure where to even begin to calculate credible intervals, or which priors or posteriors to even use. I did stumble across this [calculator](https://www.causascientia.org/math_stat/ProportionCI.html) that notes the distributions used, so maybe this is what I should use for this problem. – DataNoob7 Dec 21 '21 at 17:49
  • [Here](https://stats.stackexchange.com/a/557379/307000) is a related thread. – Geoffrey Johnson Dec 21 '21 at 18:18
  • @usεr11852, my concern is more that a canned "credible interval" routine will be used, then interpreted using frequentist definitions. Even using proper Bayesian definitions there can be some subtle but important things to think through. For example consider if the sample results in 0 cases of X, now compare a beta-binomial highest posterior density interval from a Beta(1,1) prior to an equal tail credible interval from a Beta(2,2) prior. The difference may be small, but there is a very important implication to inference in that difference. – Greg Snow Dec 21 '21 at 19:50
  • Hmm.. I see your point "somewhat" because I would have very little concerns. Those posteriors will be mighty similar... B(1,1) or B(2,2) as soon as we top it up with a 0 success/2000 failures... Done deal. After all that B(2,2) is pretty much an Agresti-Coull (which is my go-to choice). – usεr11852 Dec 21 '21 at 21:32
  • 1
    @usεr11852, yes, the 2 intervals will be pretty similar, but one will include 0% and the other one will not; one posterior distribution will have its mode at 0%, the other one will be 0 at 0%. One will have the observed percentage as the most likely parameter value, the other one will exclude the observed percentage as an impossible value for the parameter value. That is a huge impact from a small change in the prior. – Greg Snow Dec 22 '21 at 17:46
  • "*huge impact from a small change in the prior.*" is the single most important reason why we wouldn't trust anyone who is not expert on the subject to pick priors; ergo arguing for B(1,1) or B(2,2) is somewhat irrelevant. (I mean, true points (+1) just really, I don't see these differences as very impactful, I guess they might be resonant for some genome-wide study just not for the kind of industry projects I have seen - Thanks for this commentary though, my understanding on the matter is not a finished article at all). – usεr11852 Dec 22 '21 at 23:01
4

Confidence intervals can absolutely provide the inference you are interested in. A confidence interval is a set of plausible hypotheses in the parameter space given the observed data. The confidence level is based on the frequency probability of the experiment. This allows you to make the claim, "I am 95% confident that the CI computed from my sample covers the unknown fixed true population parameter." It's analogous to knowing the bias of a coin is 0.95 in favor of heads (95% of the time the coin lands heads) and the confidence this knowledge of the long-run proportion imbues regarding the outcome of a single flip. If asked how confident you are that the coin will land heads (or has already landed heads), you would say you are 95% confident based on its long-run performance.

Often times papers are published with examples that do not use all of the available information in the likelihood when constructing a confidence interval and this is presented as evidence against using a confidence procedure. This is analogous to using only a fraction of the information in the likelihood when constructing a posterior credible interval. Both the credible interval and the confidence interval reside in the parameter space. The choice between constructing a Bayesian credible interval or a frequentist confidence interval is a matter of what you want to measure using probability, the experimenter or the experiment.

The most precise solution to your problem is to construct a confidence interval for the population proportion by inverting the CDF of a binomial distribution. Here are some related threads that discuss frequentist confidence intervals for a Bernoulli proportion compared to Bayesian credible intervals when the unknown fixed true proportion is near 0 or 1 [1], [2].

Geoffrey Johnson
  • 2,460
  • 3
  • 12
  • 1
    OK, I'll soften my first sentence. – Geoffrey Johnson Dec 21 '21 at 23:28
  • Based on many threads I have read or "professionals" I work with, it seems confidence intervals do not allow you to say: "I am 95% confident that the true population parameter resides within the confidence interval I created for this sample", but rather "if I do this procedure enough times, 95% of the CI for those samples will contain the true population parameter". – DataNoob7 Dec 22 '21 at 01:34
  • You're saying the former. From what I read, if I want to have the interpretation of the first sentence, I will need to construct a Bayesian credible interval. I must admit, your detailed answers (which I extremely appreciative of) in the threads you linked are over my head. I will have to revisit this idea once I become more informed/take additional coursework. – DataNoob7 Dec 22 '21 at 01:34
  • A confidence interval is not created "for" a sample but "from" a sample. Including this minor correction, I am saying your former statement is a consequence of your latter statement. Both statements are correct under the frequentist paradigm. Please see the discussion regarding a biased coin in my answer above. Some people with ulterior motives will spread misinformation about frequentist inference claiming, "it doesn't give you what you want," or, "there is no confidence in confidence intervals." – Geoffrey Johnson Dec 22 '21 at 12:37
  • Be certain to talk with a wide range of professionals and read a wide range of papers. I recommend papers by Bradly Efron, Min-ge Xie, D.A.S. Fraser, and of course myself. [Here](https://www.researchgate.net/publication/350055507_Decision_Making_in_Drug_Development_via_Inference_on_Power) is a paper in the context of clinical drug development highlighting that Bayesian belief is objectively viewed as confidence based on frequency probability of the experiment. – Geoffrey Johnson Dec 22 '21 at 12:45
  • [Here](https://stats.stackexchange.com/a/555548/307000) is a simple example comparing Bayesian and frequentist inference involving a COVID screening test. – Geoffrey Johnson Dec 22 '21 at 13:01
  • It is just very confusing. Ex:These threads [1](https://en.wikipedia.org/wiki/Confidence_interval), [2](https://math.stackexchange.com/questions/1757954/interpretation-of-confidence-interval), [3](https://stats.stackexchange.com/questions/26450/why-does-a-95-confidence-interval-ci-not-imply-a-95-chance-of-containing-the) say my point above. Many Google searches (and even other papers I came across chastising other researchers) say "you cannot say that with 95% confidence the true population parameter lies within the L and U you calculated from the sample. This is a common misinterpretation!". – DataNoob7 Dec 22 '21 at 13:34
  • Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/132540/discussion-between-geoffrey-johnson-and-datanoob7). – Geoffrey Johnson Dec 22 '21 at 13:38
  • I was having internet problems for the last 45 minutes, so I am uncertain if you received my other messages in the chat. – DataNoob7 Dec 22 '21 at 14:55
0

We want to be able to quantify how accurate our estimate is of the true population. "We are 95.. 99% confident that the true proportion is "x/y" +/- x%. From my research, confidence intervals cannot be interpreted in such a way.

You are correct. Our confidence (or belief) is undefined unless we start with a Bayesian prior belief. In that case, we can update to a posterior belief based on the experiment. But the posterior will depend on the prior. For example, suppose our prior belief is the proportion is about 50%. And suppose our confidence interval is [30%, 40%]. We still might not believe the true parameter is in that low range. Instead we believe the experiment is a fluke.

The confidence-interval approach avoids making any claims about what we should believe post-hoc. However, it does give a useful guarantee: no matter what the true proportion is, with 95% probability (or 99.9%, etc), it will be in the confidence interval. In other words, if you make a claim "the true proportion is in this interval", that claim will only be wrong with probability 5% (or 0.1%, etc).

But when you see the actual interval resulting from your experiment, you might use prior knowledge to decide whether the 5% event (or 0.1%, etc) happened or not. That would be a Bayesian approach as I mentioned above.

usul
  • 532
  • 2
  • 5
0

I don’t think you should rely on the article “The fallacy of placing confidence in confidence intervals” as the last word on frequentist confidence intervals as the description is both limited and one-sided. Find a standard text first.

Let’s be specific to your problem. There is an unknown population proportion p. Suppose I construct a standard binomial 95% confidence interval for p then we know in advance that 95% of the random samples that could be drawn will produce CIs that contain p. We also know that the CI from a particular sample is either right or wrong. For example, suppose a random sample is drawn and the observed CI is 0.22 < p < 0.29. Now we know that P( the statement “0.22 < p <0.29” is correct) = 0 or 1. This is standard classical statistics. In such a situation, I would state 95% confidence that 0.22 < p <0.29 in the sense that this particular CI is the outcome of a process that produces correct CIs 95% of the time. Sure, I don’t know whether the observed CI is correct or not, but the 95% accuracy of the process that generated it is somewhat reassuring. Either the observed CI contains p or we have suffered a rare event [Note the past tense here.] In particular, my 95% confidence claim is neither a probability, nor a posterior probability, nor a belief. The genius of Neyman was to use a different word, confidence. Confidences should not be manipulated as if they are probabilities.

There is an important caveat regarding such confidence claims. Confidence would be undermined if it was known that the confidence interval procedure that I was using had poor conditional properties. In such circumstances, it is better to base confidence claims on appropriate conditional probabilities rather than on the unconditional probability. As far as I know, the standard binomial CI procedure does not have any dramatic shortcomings in this regard.

Regarding Bayesian credible intervals, suppose for some prior you were to generate a 95% credible interval, say 0.23 < p < 30, for example. That is, for your given prior, the posterior probability that 0.23 < p < 30 is 0.95. That form of outcome may seem better than a CI to you, but don’t forget that P( the statement “0.23 < p < 30” is correct) = 0 or 1, just like for confidence intervals.

Finally, if you are considering such large sample sizes, then perhaps you should think about raising the desired confidence level.

Graham Bornholt
  • 51
  • 1
  • 5
  • Thanks for the comment. Then it is similar to as @GeoffreyJohnson stated. It's the fact that because the procedure for which we created this CI from the sample produces CIs that contain the true population parameter 95% of the time is what allows us to say "we are 95% confident that the true population parameter is within (a

     – DataNoob7 Dec 23 '21 at 18:20
  • Yes, it is perfectly acceptable to say you are 95% confident that the parameter lies in the CI in that sense. – Graham Bornholt Dec 23 '21 at 19:29