I am trying to understand the basis of hypothesis testing and I came up with a paradox. Let me explain with an example. Consider the case of trying to determine whether is coin is fair. The null hypothesis is that the coin is fair and we would like to test whether the data supports he null hypothesis. One test is to flip a coin n times and see if the number of heads is in an interval centered on n/2 coins such that 95% of the outcomes would be in that interval provided the null hypothesis were true. If the outcome is outside of the interval, we reject the null hypothesis.
The basic structure of this test is that there are two sets of events, Set1 is the set where the null hypothesis rests and Set2 is the set of all other events. In the previous example, the 95% confidence interval centered on the hypothesis, the coin is fair, is Set1. One way to argue for this test is to say "Because we would only get a result in Set2, outside the confidence interval, with less than 5% probability, if we arrive in that set, we should reject the null hypothesis which is in Set1." However this argument breaks down with a different choice of Set1 and Set2. Consider the case where we toss the coin 10000 times. Set2 is the set of {21 heads out of 10000, 41 heads,... 20i+1,...,9981}, i.e. every 2oths head count starting at 1. Set1 is the complement of Set2 and Set1 contains the null hypothesis that the coin is fair. The probability for the 10000 tosses to be in Set2 is about 1/20 regardless of what the bias of the coin is, provided that the coin isn't deterministically always heads or always tails. According to this test, we should reject the null hypothesis even though the test tells us nothing about how likely we are to have a fair coin. The argument by low probability fails, e.g. "we should reject the null hypothesis because the experiment is in Set2 which does not contain the null hypothesis and happens with low probability".
What additional conditions/criteria, other than low-probability, should be used in designing a hypothesis tests?
