8

Can you help me understand the frequentist point of view in the bayesian vs frequentist debate? I have read a lot and all the sources I found are either filled with complex equations or written from a bayesian point of view, or both. I have not found a single sample problem where the frequentist approach would produce more useful output than the bayesian approach. I feel like I only understand one side of this debate and I would like to understand the other side as well. I do not have any background in statistics, so I would appreciate simple examples of cases where frequentist methods produce more value than bayesian methods.

A nice example would be a betting scenario where a frequentist and bayesian bet against each other about some future outcome and the frequentist has positive expected value.

John
  • 21,167
  • 9
  • 48
  • 84
Atte Juvonen
  • 1,199
  • 1
  • 10
  • 18
  • 3
    Surely you could find a few tens of thousands of such examples just by browsing through this site. In light of this, just what kind of answers are you looking for? – whuber Jan 26 '16 at 17:31
  • 7
    After 2 hours of googling I found 0 examples where the frequentist approach is more useful than the Bayesian. If you have 10 000 examples, can you provide 1 of them? Thanks. – Atte Juvonen Jan 26 '16 at 17:35
  • 2
    I do not know if this is at thew level you would like, but you can find a relevant discussion in L. Wasserman's book which is avaliable online too. http://read.pudn.com/downloads158/ebook/702714/Larry%20Wasserman_ALL%20OF%20Statistics.pdf . If you go to page 216, you will find an example concerning confidence intervals where the frequentist approach outperforms the Bayesian. – JohnK Jan 26 '16 at 18:05
  • Either you are trying to be polemical--which is not appropriate for this site--or you seem to have a different impression of "useful" than most people who ask questions here. Could you tell us more specifically what you mean by that? By the way, here is a site search for upvoted accepted answers that are non-Bayesian. Although not all these 20K+ hits will concern "frequentist" procedures, a great many will: http://stats.stackexchange.com/search?q=isaccepted%3Ayes+score%3A1+-bayes. It seems to me this is one objective way to assess what "useful" might mean to people. – whuber Jan 26 '16 at 18:33
  • 2
    @whuber: I don't believe your definition of "useful" will differ from mine in a way where it's meaningful to discuss about it. I am not here to infer that bayesian > frequentist. I have very recently learned about these subjects and I feel like I only understand one side of the debate. I would like to understand the other side as well. I find it easiest to grasp new concepts via practical examples; in this case an example where frequentism provides something of value (where bayesian methods fall short) – Atte Juvonen Jan 26 '16 at 18:44
  • 2
    I appreciate that clarification and would like to suggest you consider including it explicitly in the statement of your question. This sort of question risks getting out of control by devolving into a vexing, unresolvable debate, so it's important to keep your readers focused. – whuber Jan 26 '16 at 18:48
  • 3
    I vote to reopen. @whuber, the fact that 20k+ people came here to ask a question about frequentist techniques and got a useful answer does not imply that frequentist techniques were more appropriate than Bayesian ones in those specific cases; it just means that they are widespread. – amoeba Jan 26 '16 at 22:03
  • @amoeba Please be careful with the use of words. I have made no claims about appropriateness, only about the apparent *usefulness* of answers formulated using frequentist methods and principles. The search includes only answers that have been both upvoted and accepted, which is *prima facie* evidence they were useful to the original proposers of the questions. – whuber Jan 26 '16 at 22:46
  • @whuber In the current version the Q only asks about situations where frequentist methods are *more* useful than Bayesian ones, and no amount of upvoted and accepted answers about frequentist methods can serve as an evidence that they are ever *more* useful. However, I now see that the original formulation of the Q included a sentence about frequentist methods being "useful" [on their own], as opposed to "more useful". You were probably replying to that sentence, removed in a later edit. In any case, I certainly meant no offense! – amoeba Jan 26 '16 at 22:59
  • @Amoeba no offense taken; I apologize if it sounded like that. I wanted to make sure my previous comments were understood as I had intended them to be. To be sure, I am diffident about just what any body of questions and answers might really be telling us about usefulness, but I wanted to make it clear that the question's claim (which is still there) that "I have not found a single sample problem where the frequentist approach would produce more useful output than the bayesian approach" appears to be without foundation. This is important because that claim regrettably sets a polemical tone. – whuber Jan 26 '16 at 23:03
  • 2
    It's not clear to me how a "superior" approach could be defined - Bayesian and frequentist approaches each give the "right" answer, according to their own precepts. (And a particularly partisan Bayesian might argue their approach to be superior over the frequentist one, because the frequentist's answer is "wrong", and vice versa.) I appreciate the betting example but I think this question would benefit from being pinned down more. – Silverfish Jan 27 '16 at 00:14

1 Answers1

6

A nice example would be a betting scenario where a frequentist and Bayesian bet against each other about some future outcome and the frequentist has positive expected value.

I will not give you this example because such an example would favor a Bayesian approach unless the Bayesian chooses a bad prior which is a cop-out example not really worth writing about.

The frequentest approach is not designed to obtain the highest expected value in betting scenarios (luckily the world of statistics and probability is much more broad than just that). Rather, frequentist techniques are designed to guarantee certain desirable frequency properties, particularly that of coverage. These properties are important for parameter estimation and inference in the context of scientific research and inquiry.

I encourage you to check out this link here to a blog post by Dr. Larry Wasserman. In it he talks about frequency guarantees in more depth (see the examples he gives).

Suppose we had some data $Y$ and we think it is distributed according to some conditional distribution $Y \sim f(Y|\theta^*)$ (if you like you can imagine that $Y$ is normally distributed and $\theta^*$ is the mean and\or variance). We do not know the value of $\theta^*$ , so we have to estimate it. We can use either a frequentist or Bayesian approach to do so.

In the frequentist approach we would obtain a point estimate $\hat \theta$ and a confidence interval for that estimate. Assuming $\theta^*$ exists and the model is valid and well behaved, the frequentist $(1-\alpha)$ confidence interval is guaranteed to contain $\theta^*$ $(1-\alpha)$% of the time regardless of what $\theta^*$ actually is. $\theta^*$ could be 0, it could be 1,000,000, it could be -53.2, it doesn't matter, the above statement holds true.

However, the above does not hold true for Bayesian confidence intervals otherwise known as credible intervals. This is because,in a Bayesian setting, we have to specify a prior $\theta \sim \pi(\theta)$ and simulate from the posterior, $\pi(\theta|Y) \propto f(Y|\theta)\pi(\theta)$. We can form $(1-\alpha)$% credible intervals using the resulting sample, but the probability that these intervals will contain $\theta^*$ depends upon how probable $\theta^*$ is under our prior.

In a betting scenario, we may believe that certain values are less likely to be $\theta^*$ then others, and we can assign a prior to reflect these beliefs. If our beliefs are accurate the probability of containing $\theta^*$ in the credible interval is higher. This is why smart people using Bayesian techniques in betting scenarios beat frequentist.

But consider a different scenario, like a study where you are testing the effect of education on wages, call it $\beta$, in a regression model. A lot of researchers would prefer the confidence interval of $\beta$ to have the frequency property of coverage rather than reflect their own degrees of belief regarding the effect education on wages.

From a pragmatic standpoint, it should also be noted that in my earlier example, as the sample size approaches infinity, both the frequentist $\hat \theta$ and Bayesian posterior $\pi(\theta|Y)$ converge onto $\theta^*$. So as you obtain more and more data, the difference between the Bayesian and frequentist approach becomes negligible. Since Bayesian estimation is often (not always) more computationally and mathematically rigorous than frequentist estimation, practitioners often opt for frequentist techniques when they have "large" data sets. This is true even when the primary goal is prediction as opposed to parameter estimation/inference.

Zachary Blumenfeld
  • 3,826
  • 1
  • 14
  • 21
  • +1 but regarding your regression example (testing the effect of education on wages), while I agree that "a lot of researches" (myself included!) do prefer to use frequentist procedures, there are many people, statisticians included, saying that this whole approach is misguided and does not work properly or even as intended. This is not a place to debate it, but it should be mentioned that this point of view exists too. – amoeba Jan 26 '16 at 21:56
  • @amoeba, pretty much all of those arguments are not about properly used frequentist approaches per se but about overuse, misuse, and misunderstanding of them. – John Feb 11 '16 at 03:50
  • Zachary, as this thread is closed, would you mind or perhaps prefer if your answer were moved in the http://stats.stackexchange.com/questions/194035 ? This can be done if this thread is "merged" into that one (i.e. closed as duplicate and all answers are moved). I think this could be helpful. – amoeba Feb 11 '16 at 10:31
  • @amoeba sure, if you think that would be helpful. – Zachary Blumenfeld Feb 11 '16 at 10:47
  • I cannot do it, I can only flag this post for moderator attention and suggest merging. But I wanted to ask you first. E.g. your answer starts with a quote and if it's moved then the quote will become a bit confusing. But the answer itself seems to fit in there and I think it will be better visible in that very popular open thread than in this closed and unpopular one. – amoeba Feb 11 '16 at 10:50
  • @amoeba thank you for asking. Whatever, the moderators feel is best I will go along with that. I did think about answering that question separately when it was posted, but there was already a lot of answers and I didn't feel like getting into a Bayesian vs Frequentist debate and I was afraid I would start to rant my opinions...which would not have been conducive of anything. But if this post does improve the discussion then maybe that is the better place for it.... – Zachary Blumenfeld Feb 11 '16 at 11:45
  • 1
    "I will not give you this example because such an example would favor a Bayesian approach unless the Bayesian chooses a bad prior which is a cop-out example not really worth writing about." I strongly disagree with this. This is the **fundamental** reason for considering frequentist statistics in the first place: good priors are hard to come by. Bayesian results are trivially better with a good prior, but a obtaining a good prior is **very** non-trivial. – Cliff AB Oct 20 '16 at 21:54