5

Problem

Suppose we use a Gaussian PDF to express the likelihood of light intensity prevalent on Clear, Cloudy, and Eclipse weather. The probability of a certain amount of light value (positive or negative) given the weather is given by the Rayleigh probability function.

$$ P(light | w) = \frac{light}{\sigma_w^2}e^{\frac{-light^2}{2\sigma_w^2}}$$

defined for non-negative light values, where $w$ denotes the weather class $ \in \{Clear, Cloudy, Eclipse\}$ and $light$ denotes the light intensity level (integer).

We're given 200 arbitrary light measurements labeled $light_1$, $light_2$, ..., $light_{200}$ while an eclipse is occurring.

  1. Derive the maximum likelihood estimate for $\sigma_{eclipse}$.
  2. Given the prior probability function, $P(\sigma)=2e^{-2\sigma}$, derive the maximum posterior estimate for $\sigma_{eclipse}$.

Assume $ \sigma_w $, the standard deviation of weather, $ = 3 $.

Attempt

  1. Taking the equation $ P(light_1, light_2, ..., light_{200} | \theta) = \prod_i{P(light_i|\sigma_w)}$, I began plugging in for the arbitrary light intensities as follows.

    $ \prod_i{P(light_i|\sigma_w)} = (\frac{light_1}{\sigma_w^2}e^{\frac{-(light_1)^2}{2\sigma_w^2}}) (\frac{light_2}{\sigma_w^2}e^{\frac{-(light_2)^2}{2\sigma_w^2}}) (\frac{light_3}{\sigma_w^2}e^{\frac{-(light_3)^2}{2\sigma_w^2}})...$ $ \prod_i{P(light_i|\sigma_w)} = \frac{1}{\sigma_w^2}(light_1e^{\frac{-(light_1)^2}{2\sigma_w^2}}) (light_2e^{\frac{-(light_2)^2}{2\sigma_w^2}}) (light_3e^{\frac{-(light_3)^2}{2\sigma_w^2}})...$

    Plugging the known value $ \sigma_w = 3 $, $ \prod_i{P(light_i|3)} = \frac{1}{9}(light_1e^{\frac{-(light_1)^2}{18}}) (light_2e^{\frac{-(light_2)^2}{18}}) (light_3e^{\frac{-(light_3)^2}{18}})...$

    $ \prod_i{P(light_i|3)} = \frac{1}{9}\prod_i{light_ie^{\frac{-(light_i)^2}{18}}}$

    I don't know how to progress from here.

  2. After finding the maximum likelihood estimate in (1), I assume finding the maximum posterior estimate requires simply multiplying the MLE value by the prior probabilities, which can easily be obtained from the equation above.

Notes

Here, my question is how to proceed with part (1). Currently taking a grad-level ML course without exposure to undergraduate probability so any help would be greatly appreciated. Thank you!

Anthony Krivonos
  • 163
  • 7

1 Answers1

1

Since this is a self-study question, I will offer only general advice, without showing how to find the MLE of the Rayleigh distribution. Finding MLEs involves essentially three important steps:

  1. Define all your observable variables and parameters clearly, using consistent, unambiguous, and parsimonious notation. Do not use multiple different names/symbols to refer to a single variable or parameter, and do not switch between different forms of notation in different statements;

  2. Once notation is clearly defined, write the sampling density and the corresponding likelihood function (and possibly also the log-likelihood function) in its simplest possible form using your stated notation;

  3. Maximise this using function with respect to its parameters using standard calculus techniques (or discrete optimisation techniques for discrete parameters). For the case of IID data, the likelihood function is a product of sampling densities, so it is easier to maximise the log-likelihood function (which is a sum of log-densities). This usually involves standard methods of finding the first and second derivatives, and checking appropriate critical points/boundary points of the function.

It is important not to underemphasise the importance of Steps 1-2 --- if you do not succeed in defining clear and unambiguous notation, and then simplifying the likelihood function into a compact form, this will make the optimisation step appear much more difficult.

In the present case, you have not succeeded in Steps 1-2, and so naturally you are having problems with Step 3. You need to go back and revise your notation clearly, and then simplify your likelihood function. It is a real mess at the moment, and until you clean up the mess and state the problem much more clearly, the optimisation step will be a long way away. Here is some more specific advice to progress your problem.

Clean up the notation for your variables: You have multiple different ways that you refer to the same variable, sometimes using an entire textual name, sometimes using letter notation, and sometimes switching to different letters, or back to textual description. For example, your initial sampling density is stated to be conditional on a variable called $weather$ that does not appear in the density, but you do refer to an undefined variable $\sigma_w$ in that initial sampling density. Then later you refer to a parameter $\sigma_{eclipse}$, which does not appear in your sampling density at all. You also use entire textual names for variables (but also do this inconsistently) rather than using parsimonious letters to denote them. (This is essentially a regression from modern algebra back to the state of mathematics in the middle-ages!)

If you are not sure how to do this, have a look at some worked solutions of MLE problems in other cases, and pay attention to how the notation for the problem is set up. The normal thing to do is to define a sequence of observable values $x_1,x_2,x_3, ... \sim \text{IID Dist}(\mu, \theta,\sigma,...)$ based on a distribution using some unknown parameters $\mu, \theta,\sigma,...$. You need notation for the observable values and notation for the parameters. And to be clear: you need one single consistent notation for each of these. It should be possible for your reader to tell what each of the symbols mean.

Simplify your likelihood function: Once you have cleaned up your notation, you will need to simplify your likelihood function down to its simplest form. Your data is IID so your likelihood is a product of similar terms. You should be able to collect like terms and use rules for products of exponentials, etc., to simplify this function down to a much simpler expression. Remember that when we are dealing with a likelihood function, the data is treated as fixed and we only care about the parameters, so any multiplicative terms that do not depend on the parameters can be dropped entirely. (To do this, learn how to work with proportionality statements using the $\propto$ relation.)

As with your underlying notation for your variables, you should also make sure you use standard notation for your likelihood function. If you do not know what this looks like, have a look at other worked solutions to MLE problems to see the standard notation. Once you have simplified your likelihood function down to a simple form, using clear notation, you will then be in a position to start the optimisation step.

Ben
  • 91,027
  • 3
  • 150
  • 376