0

The brute force Maximum A Posteriori estimation involves computation of posterior probability for all values of $\theta$ and then we choose the value of $\theta$ that maximizes $P(\theta | D)$. Right.

In this concept, why is it necessary to have uniform priors and noiseless data (as mentioned in one of the articles that I went through)?

I do not see any point in having such assumptions in place. For example, even if priors are non-uniform the posterior probability given by $p(\theta|D)$ can easily be obtained. Further, even if the data is noisy, we can always model this noise by some distribution (depending upon the source it comes from) and the parameters of this noise distribution could be appended to form extended $\theta$. Then, we can easily compute the posterior probability by Bayes rule.

Your take on this ?

Upendra01
  • 1,566
  • 4
  • 18
  • 28
  • Could you share the reference where you saw such statement? What exactly did it say (maybe you could give us a whole quote)? As mentioned by @MartijnWeterings, authors of such statement might have in mind some specific scenario and this may be important for understanging the statement. – Tim Nov 06 '17 at 12:17

1 Answers1

2

In this concept, why is it necessary to have uniform priors and noiseless data (as mentioned in one of the articles that I went through)?

No and no.

Maximum a posteriori estimation is estimation procedure aiming at finding the maximum (mode) of the poisterior distribution

$$ \begin{align} \hat{\theta}_{\mathrm{MAP}}(x) &= \underset{\theta}{\operatorname{arg\,max}} \ f(\theta \mid x) \\ &= \underset{\theta}{\operatorname{arg\,max}} \ \frac{f(x \mid \theta) \, g(\theta)} {\displaystyle\int_{\vartheta} f(x \mid \vartheta) \, g(\vartheta) \, d\vartheta} \\ &= \underset{\theta}{\operatorname{arg\,max}} \ f(x \mid \theta) \, g(\theta) \end{align} $$

to find it, you need likelihood function $f(x \mid \theta)$ and a prior $g(\theta)$, any prior. There is no reason whatsoever why the prior should be uniform. With uniform prior $g(\theta) \propto 1$, MAP reduces to maximum likelihood estimation since you are finding the maximum of $f(x \mid \theta) \; g(\theta) = f(x \mid \theta) \times 1$. With non-uniform prior, you simply include such prior information into your model. If prior needed to be uniform, then you wouldn't be able to use non-uniform priors, so the method would be pretty useless for Basyesian estimation.

You answered your second question yourself

Further, even if the data is noisy, we can always model this noise by some distribution (...)

In real world there is no such a ting as a "noiseless" data, at least I never seen it besides the syntetic, simulated data.

Tim
  • 108,699
  • 20
  • 212
  • 390
  • The OP mentions a specific method for MAP estimation. Could it be that for a specific brute force method, noisy data becomes a problem? MAP estimation is an inverse problem. Noise may very well cause quite some troubles due to the creation of an ill-conditioned problem. The case of the uniform prior is a bit strange (because why not call it maximum likelihood estimation instead?), but possibly the case occurs in a wider framework for which the use of the term maximum a posteriori estimation makes sense. I wouldn't be surprised if the authors have reasons (if only the OP would provide a link). – Sextus Empiricus Nov 06 '17 at 11:34
  • @MartijnWeterings grid search as a optimization method wouldn't work well in many cases, especially in high-dimension. You can use it for optimizing *any* function, but in many cases it wouldn't be feasible, it is not a problem of priors or data, but of very primitive optimization algorithm. – Tim Nov 06 '17 at 11:40
  • 1
    I would say that the ill-conditioning is in the first place due to the noise (for instance in an image reconstruction where the noise does not occur as a possible image, and therefore, can not be inverted, or the noise in the measurement being amplified in the inversion and creating highly noisy images). You can argue that brute force methods (which in some people's terminology is a bit wider and extends beyond grid search) are bad for such cases. But that might be exactly the point of the article that is being mentioned in the OP and thus such a comment does not address the OP's question. – Sextus Empiricus Nov 06 '17 at 12:10
  • @MartijnWeterings good, point, I'll ask OP to clarify his question. – Tim Nov 06 '17 at 12:15