Probability that $n$ trials will succeed given that $k$ succeeded

Question

I'm not sure exactly how to ask this, or if there is such a thing. I'm new to statistics and have just studied confidence intervals and confidence levels of survey data, such as the confidence of population mean being within the certain interval around the sample mean.

I'd like to measure some confidence in data accuracy.

I'm doing some data munging at work, basically copying some data from a source, manipulating it a bit, and then storing it in a database. I'd like to be able to tell my manager that I'm confident that the data is accurate to a certain level.

I plan on verifying the data with another tool, sampling the calculated data vs. original data (say randomly picking 10000 entries of the 62000000 and recalculating to verify), and assuming the entire sample matches, report that I am x% confident that the entire set of calculated data is correct.

However, I'm not sure what I'm measuring. I don't want to measure a mean, but instead the accuracy of n% of the population, and infer if that means that m% is indeed accurate.

Can I only be confident that if I verify n% of the population is accurate, that only n% is accurate?

Answer 1

One way to think of this is as repeated Bernoulli trials with an equal probability of success $θ$. Suppose you observe 10,000 trials, all of which succeed, and you want to know the probability that an additional 62,000,000 - 10,000 = 61,990,000 trials would also all succeed. I'm at a loss to formulate a frequentist approach to this problem, using a confidence interval. But a Bayesian approach seems natural: you want the posterior probability of all 61,990,000 trials succeeding given your prior beliefs about $θ$ and the 10,000 observed successes.

Let $θ$ have a $\operatorname{Beta}(α, β)$ prior. Then its posterior distribution is $\operatorname{Beta}(10,000 + α, β)$. Then the posterior predictive probability for success of a single future trial happens to be just the posterior mean of $θ$, which is

$$\frac{10,000 + α}{10,000 + α + β} \quad .$$

For the probability that 61,990,000 independent trials succeed, just raise this quotient to the 61,990,000th power.

It is interesting to note that because 61,990,000 is so large, this probability will be very small unless $α$ is much larger than $β$; that is, unless you were already virtually certain that your method was extremely accurate before you even observed those 10,000 successes.

Answer 2

I will build on the answer by @Kodiologist but show that frequentist answers are possibe. What we need is the concept of predictive likelihood, see https://projecteuclid.org/download/pdf_1/euclid.ss/1177012175. The idea is to build a likelihood function for an unobserved random variable. There are multiple ways of doing this.

Let $X=(X_1, \dotsc, X_n)$ be the observed variables (in the exmple $n=10000$) and $Y=(Y_1, \dotsc, Y_m)$ be the future (unobserved) variables, so we want a predictive likelihood for $\sum_j Y_y$. $X,Y$ have independent bernoulli distributions with the same parameter $\theta$. We write $x=\sum_i X_i, y= \sum_j Y_j$, and the maximum likelihood estimator of $\theta$ based on $x, y$ is $\hat{\theta}_y = \frac{x+y}{n+m}$.

Then the profile predictive likelihood for $y$ is $$ L_p(y | x) = \sup_\theta f_\theta (y,x) = L_y(y,\hat{\theta}_y) $$ where $L_y$ is the usual likelihood function based on $x,y$. For our example model this becomes $$ L_p(y | x) = \sup_\theta \binom{n}{x} \theta^x (1-\theta)^{n-x} \binom{m}{y} \theta^y (1-\theta)^{m-y} $$ and when we have observed $x=n$ this reduces to $$ L_p(y | x) = \binom{m}{y} \left( \frac{n+y}{n+m} \right)^{n+y} \left( \frac{m-y}{n+m}\right)^{m-y} $$ (I will come back and extend this answer)