Following this question, I am trying to learn the math behind checking whether a coin is fair.
While user @BruceET provides an outline of the overall math to get there, I am still stuck on the basic concepts, which I have pinned down to not understand the "Binomial proportion confidence interval."
In his answer he writes:
Addendum per comment. Argument for the Wald CI (which was originally intended only for very large $n.)$ Begin with $E(\hat p) = p; SE = SD(\hat p) = \sqrt{\frac{p(1-p)}{n}}.$ Assuming $\hat p$ is normal, standardize to get $Z = \frac{\hat p - > p}{SE} \stackrel{aprx}{\sim}\mathsf{Norm}(0,1),$ so that $P(-1.96 < Z > < 1.96)\approx 0.95.$ Use algebra to transform the event, obtaining $P(\hat p - 1.96SE < p < \hat p + 1.96SE)\approx 0.95.$ Then provided we knew SE, an approximate 95% CI for $p$ would be of the form $(\hat p - 1.96SE,\, \hat p + 1.96SE).$ However $SE = > \sqrt{\frac{p(1-p)}{n}}$ contains unknown $p.$ For sufficiently large $n,$ one has $\hat p \approx p,$ so approximate $SE$ as $\widehat{SE} = \sqrt{\frac{\hat p(1-\hat p)}{n}}.$ Then pretend $\left(\hat p - 1.96\widehat{SE},\, \hat p + 1.96\widehat{SE}\right)$ is 95% CI for $p.$
So from this question and answer, I think I understand the first step. To quote the answer:
So, let the items that you pick be represented by the random variables $X_i, 1\le i \le n$, each of them identically distributed with variance $\sigma^2$. They are independently sampled, so the variance of the sum is just the sum of the variances. $$ > \text{Var}\left(\sum_{i=1}^n X_i\right) = > \sum_{i=1}^n\text{Var}\left(X_i\right) = \sum_{i=1}^n\sigma^2 = > n\sigma^2 $$
Next we divide by $n$. We know in general that $\text{Var}(kY)=k^2 > \text{Var}(Y)$, so putting $k=1/n$ we have
$$ \text{Var}\left(\frac{\sum_{i=1}^n X_i}{n}\right) = \frac{1}{n^2} > \text{Var}\left(\sum_{i=1}^n X_i\right) = \frac{1}{n^2} n\sigma^2 = > \frac{\sigma^2}{n} $$
Finally take the square root to get the standard deviation $\dfrac{\sigma}{\sqrt{n}}$. When the population standard deviation isn't available the sample standard deviation $s$ is used as an estimate, giving $\dfrac{s}{\sqrt{n}}$.
So if I make a random variable that represents the average value of a bunch of coin flips (where heads is assigned a 1 and occurs with probability p; tails is assigned a 0 and occurs with probability q=1-p), then the expected value of that random value for n coinflips is n*p and the standard deviation is $\dfrac{\sigma_c}{\sqrt{n}}$ where $\sigma_c$ is the standard deviation of flipping a single coin (which is p(1-p)).
So now for any number of coins, I know the expected value of how many will flip heads. I can also now know the frequency of heads vs tails represented as a number from 0-to-1 by dividing by the total amount of flips, which is represented by $\frac{\sum_{i=1}^n E\left(X_i\right)}{n}$. I have also now calculated the variance of this random variable.
Now from what I understand from @BruceETs explanation of the "Binomial proportion confidence interval", we can use this information to estimate our distribution for this random variable as a normal distribution. We find a section that has 95% of the area under this normal distribution that we estimate to be centered around the frequency of heads that we measured $\hat{p} = N_{Heads}/N_{Total}$.
And it's around this point that I am conceptually lost. The goal is to get an estimate for the probability of the weighted coin. So in this case does confidence interval of 95% represent the following: "95% of the time a fair coin would observe this"?
I am really struggling to interpret what this confidence interval exactly represents. My goal is to be able to quantify exactly what my "confidence" is in a certain weighted parameter. So for example if I perform 100 different tests of flipping 1000 fair coins, how often will my 95% CI test fail? Is it expected to fail 5% of the time?
Ideally an answer that connects to the details of the math for the "Binomial proportion confidence interval" would be preferred. (I think I understand conceptually that there are some definition problems with assigning a probability weight as a random distribution when it is predetermined, as answered here. My question is more about just trying to actually understand how this confidence interval is generated and how the math actually explains what it is.)
