6

Recently I am reading a paper in 2001, Michael D. Ernst, Jake Cockrell, William G. Griswold, David Notkin Dynamically Discovering Likely Program Invariants to Support Program Evolution TSE 2001, in this paper, it says,

Learning approach such as Bayesian and PAC learning, assume there is noise in the input data and hence, inaccuracies in classification are acceptable or even beneficial.

I never knew Bayesian will be benefited by noisy data. So, my questions are:

  1. Noisy data will really benefit Bayesian? If so, it will improve accuracy or simply speed up the model?
  2. What does "noisy data" here really mean? Because I tried to check some resources, one paper said, noisy data will speed up EM, and noisy data there means latent data or missing data. I felt EM has some connection with Bayesian, so I am wondering whether there is any connection...
kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
Cherry Wu
  • 289
  • 2
  • 11
  • please give a proper reference (author(s), title, year, etc); links can disappear without warning – Glen_b Feb 21 '17 at 23:09
  • 1
    Michael D. Ernst, Jake Cockrell, William G. Griswold, David Notkin Dynamically Discovering Likely Program Invariants to Support Program Evolution TSE 2001 – Cherry Wu Feb 22 '17 at 00:57
  • 1
    This is good, thanks, but it belongs in the question. Please edit the question to insert it. – Glen_b Feb 22 '17 at 00:59

1 Answers1

2

Adding noise reduces the quality of Bayesian results as it does for Frequentist and Likelihoodist methods. It will also slow down the model. This can be seen with a simple, degenerate example.

Consider a case of data consisting of five points (1,1), (2,2), (3,3), (4,4) and (5,5). The slope is 1 and the intercept is zero. There is 100% certainty as to the parameters, if the model is valid. The posterior will be the Dirac delta function. Now adding noise creates an ordinary posterior with less certainty as a necessity. Furthermore, anything which spreads the uncertainty increases computation time.

Where increases in variability do improve Bayesian methods is when it identifies signal rather than noise. Imagine a training set that only had green and brown eyed individuals. How would it handle its first blue-eyed person outside the training set? By having a blue-eyed person in the data set, this increase in natural variability improves the degree to which the model matches reality. This will speed up processing speed. It will narrow the variability.

Dave Harris
  • 6,957
  • 13
  • 21