6

This question is about a kind of bias that is slightly more subtle than doing multiple tests without caution (related to multi-hypothesis testing).

Experimenters are willing to reject an hypothesis $H_0$ that a coin is fair, expecting to find a higher probability for heads. When flipping 100 coins, significance of $\alpha=5$% is obtained with 59 heads.

An obvious bias consists in forgetting negative results and only publishing positive ones. At worst: do 100 flips multiple times until you get more than 59/100 heads and publish only the last experiment.

Instead, the experimenter does multiple flips and stops when "he feels like stopping" and publish all the previous flips. If he is secretly willing to reject $H_0$ this might create a bias: stopping at a point the results are rather positive.

At worst, he might stop when the test for all previous flips is positive. Actually, I'm quite sure the stopping time $T:$"first time the test is significant on all previous flips" is almost surely finite. Do you confirm this?

Is there a name for this kind of bias? Do you know of any studies or texts about it?

Benoit Sanchez
  • 7,377
  • 21
  • 43
  • 3
    You may be interested in this thread and the simulations in there: https://stats.stackexchange.com/questions/244646/why-is-it-wrong-to-stop-an-a-b-test-before-optimal-sample-size-is-reached/244664 – Bernhard Jan 30 '18 at 10:11

2 Answers2

2

Some relevant posts here and here.

According to an answer from the second post, it seems that as the number of flips goes to infinity, at some point the significance test will be positive (almost surely), which is to say, there exists some finite number of samples after which it will almost surely happen.

According to the first post, the number of flips it will actually take has a finite median but infinite expectation. The median grows very quickly as a function of the required $z$-score to pass the test, so it may be that this sort of bias can be effectively mitigated by demanding a lower $\alpha$ threshold.

shimao
  • 22,706
  • 2
  • 42
  • 81
  • Thanks a lot. You say: "there exists some finite number of samples after which it will almost surely happen.". Would it be rather " almost surely, there exists some finite number of samples after which it will happen"? – Benoit Sanchez Jan 30 '18 at 09:06
  • Formally, the statement is $\lim \sup_k \frac{S_k}{\sqrt{k}} = \infty$, which is by definition $\lim_{n \rightarrow \infty} \sup_{k \geq n} \frac{S_k}{\sqrt{k}} = \infty$ hence the strange wording I used, but I am pretty sure this would imply $\sup_{k \in \mathbf{N}} \frac{S_k}{\sqrt{k}} = \infty$. So yes, I think it works out. – shimao Jan 30 '18 at 09:11
1

I did some simulations (under $H_0$: a fair coin $p=0.5$). I limited the number of flips to $n_\max$ because the raw stopping time $T$ has such a huge tail that sometimes the computer wouldn't stop in a reasonable time. Anyway it's more realistic with a limit.

The experiment is:

  • do some first $100$ flips to initialize
  • do a z-test with $\alpha=5$%. If it is significant or you flipped more than $n_\max$, stop.
  • otherwise flip one more time and go back to the previous step

The false discovery rate (type I error) differs a lot from $\alpha$:

  • For $n_\max=1000$ : 26%
  • For $n_\max=10000$ : 40%

However something happens because of the optional stopping theorem: when "meta-anlyszing" several of these experiments (simply merging them into one big flipping session and do a z-test), the bias on false discovery rate tends to disappear:

enter image description here

It might sound a bit paradoxical: we have many experiments where 26% are falsely significant on average, but the global experiment still has the right type I error of 5%. And the global estimator $\hat p$ is still (asymptotically) unbiased. It can be explained by the fact that the longest experiments, having more weight, are the least favourable to rejecting $H_0$.

As a conclusion, optional stopping can cause a strong bias for tests on each single experiment, but the bias tends to disappear when doing several experiments.

Benoit Sanchez
  • 7,377
  • 21
  • 43