2

For example for this problem:

You have a coin that when flipped ends up head with probability p and ends up tail with probability 1−p. (The value of p is unknown.)

Trying to estimate p, you flip the coin 14 times. It ends up head 10 times.

Then you have to decide on the following event: "In the next two tosses we will get two heads in a row."

Would you bet that the event will happen or that it will not happen?

(http://www.behind-the-enemy-lines.com/2008/01/are-you-bayesian-or-frequentist-or.html)

Could one flip a bunch of different coins a million times and see what happens? (Or write a program to do so?)

EDIT: Some people were asking, I think, how probability theory could possibly be relevant when empirical data was available. For the above problem, I wrote a script which would flip a coin with a 1% chance of landing heads, a 2% coin, a 3% coin, etc. fourteen times; and if it came up heads ten out of those fourteen times, then it would flip the coin twice more and see whether it came up heads both times. This "experiment" is repeated a bunch of times and the percent of the time you get two heads, given that you've gotten ten out of fourteen heads, is around 48%, which is exactly the Bayesian answer given on the website above.

I tried not to make any assumptions about the nature of probability and just run the experiment the way it would happen "in the wild", if it were possible to run a million trials of this kind in real life, but I'm slightly worried that I made some inherently Bayesian assumption in the writing of the code.

Thoughts?

Community
  • 1
user77463
  • 31
  • 3
  • 5
    if we knew what reality was there would be no need for a model, frequentist or bayesian. – bdeonovic May 18 '15 at 17:25
  • 9
    Why would an empirical test be needed? *All* statistical procedures can be evaluated with perfect accuracy using mathematical reasoning. Any empirical test would only be a test of whether various underlying *assumptions* may be true, not of the statistical procedures. – whuber May 18 '15 at 19:22
  • Here is an [author](http://jakevdp.github.io/blog/2014/06/12/frequentism-and-bayesianism-3-confidence-credibility/) who wrote Python scripts to do simulations, in order to show the difference between frequentist and bayesian methods. – Pieter Hogendoorn Feb 03 '17 at 13:24

3 Answers3

3

In Bayesian approach as the sample size grows the influence of the prior diminishes. With an infinite sample you'll get the same results in frequentist and Bayesian approaches. With a finite sample, or small sample, there is no way to tell which approach is more accurate.

Aksakal
  • 55,939
  • 5
  • 90
  • 176
  • 5
    Counterexample: the parameter of interest $y$ is a mean. The prior is uniform on the positive reals but the true value of $y$ is negative. The Bayesian solution will never coincide with the frequentist estimate. – Sycorax May 18 '15 at 17:37
  • 1
    @user777, should we can start a thread on dangers of improper priors? :) – Aksakal May 18 '15 at 18:16
  • 2
    @user777 - not a fair comparison. The parameter space for the frequentist estimator should be $(0, \infty)$ if the investigator is absolutely certain that the true parameter is $ > 0$, and of course both frequentist and Bayesian estimates would go to 0 from above as the sample size increases. After all, we don't treat the mean of the Poisson as if negative values were possible in frequentist analysis (see any reporting of confidence intervals on same.) – jbowman May 18 '15 at 19:57
  • 1
    @Aksakal This isn't an issue with improper priors. Some gamma prior has the same "problem," but is itself proper and strictly positive. I point this out to illustrate that there are some assumptions underlying your post which are not explicit. – Sycorax May 18 '15 at 20:22
  • 1
    Isn't the Frequentist "approach" just the Bayesian approach with an unstated prior? Why present them as two different approaches? Why not just say the Bayesianism makes the prior explicit? – Neil G Jun 06 '15 at 08:30
  • 1
    @Neil G: NO, its not so. Some frequentist analysis do not have *any* corresponding prior. To proove your atatement, you would have to give a procedure for "unmixing" the underlying prior from some given frequentist analysis. Can you do that? – kjetil b halvorsen Feb 03 '17 at 13:39
  • It is worth mentioning that it is not only about *size* of the sample, but also about [informativeness of the likelihood and prior](http://stats.stackexchange.com/questions/200982/do-bayesian-priors-become-irrelevant-with-large-sample-size/201059#201059). Because "it depends" it makes things even more complicated. – Tim Feb 03 '17 at 14:06
  • @kjetilbhalvorsen You can discover the assumed prior by calculating the frenquentist belief for any question on a data set and then doing a pointwise quotient of densities between the belief and the likelihood induced by the data. (The likelihood is incontrovertible.) Feel free to ask a question if this is unclear and I'll be happy to provide some examples. – Neil G Feb 03 '17 at 17:35
  • @ Neil G: That assumes that there is something called "frequentist beleif". What is it, say, if the frequentist reported a p-value, or "reject at 5% level", or a confidence interval?? You would also need to prove that the ratio you calculated actually *is* a probability, that is, gives a total probability of 1. For instance, for the usual t-test there is no prior giving that procedure, because the resulting probability is improper. – kjetil b halvorsen Feb 03 '17 at 17:54
  • @kjetilbhalvorsen There is always a frequentist belief because you can ask questions that determine it. For example, "Given a set of coin flips, what gambling odds would I be willing to accept on the next flip." Frequentism can answer such questions and answering such questions decodes the "frequentist belief" about the coin. – Neil G Feb 03 '17 at 19:08
  • Also, the usual t-test emerges from a standard Bayesian Gaussian model since the t-distribution is the posterior predictive of the Gaussian model. Ask a question, and I'll work through an example for you. – Neil G Feb 03 '17 at 19:10
  • @Neil G: I have no problems with the theory, and can work out examples itself. My problem it with your claim that you **always** can do this. I doubt it, and no matter of examples can prove that ... Anyhow, such is a formal relationship, the logic behind frequentist and Bayesian procedures are quite different! – kjetil b halvorsen Feb 03 '17 at 20:47
3

In comparing Bayesian and frequentist methods note that the frequentist approach is indirect and doesn't really give an "answer" in the same sense as a direct probability model such as a Bayesian model. Notes above have concentrated on a point estimate of the population mean, which is fine; also look at inferential quantities.

When the person viewing the results is using the prior that was used in the calculations, the Bayesian results have to be right.

See http://www.fharrell.com/2017/01/p-values-and-type-i-errors-are-not.html for more about problems caused by the indirectness of the frequentist approach.

Frank Harrell
  • 74,029
  • 5
  • 148
  • 322
0

sometimes it is just imposible to do that much experiments, unlike the example of flipping a coin , it could be some medical issues, which is unaffordable to be repeated for many times, in this case frequencist based methods not work, the only choice is Bayesian

crazyminer
  • 79
  • 5