4

That is, say I have a distribution with parameter $\theta$.

If I re-write it with a parameter $a$ such that $a^3=\theta$, is it possible that doing maximum likelihood estimation on a will yield an estimate $\hat a$ such that $\hat a^3 \neq \hat \theta$?

Could it be the case for another function different than $x^3$?

If so, what are some criteria to choose a parametrization?

josinalvo
  • 345
  • 1
  • 7
  • An estimate is a realization of the estimator.Usually, the realization of an estimator does not equal the target, $\hat\theta\neq\theta$. This is because a particular property of a sample usually does not equal the corresponding property of the population. Hence, it would not at all be surprising if $(\hat{a})^3\neq\theta$ even if $(\hat{a})^3$ is an estimator targeting $\theta$. Is your question perhaps whether it is possible that $(\hat{a}^{\text{MLE}})^3\neq\hat\theta^{\text{MLE}}$? – Richard Hardy Aug 17 '20 at 11:18
  • See also https://stats.stackexchange.com/questions/302064/proof-of-invariance-property-of-mle, https://stats.stackexchange.com/questions/438778/question-about-casella-and-bergers-proof-of-mle-invariance, https://stats.stackexchange.com/questions/390741/invariance-property – kjetil b halvorsen Aug 18 '20 at 01:41

1 Answers1

8

The Invariance Property of Maximum Likelihood Estimators (MLEs) says, if $\hat{\theta}$ is the MLE of $\theta$, then for any function $\tau(\theta)$ the MLE of $\tau(\theta)$ is $\tau(\hat{\theta})$.

So, if you define $a^3=\theta$, once you obtained your MLE for $\theta$, $\hat{\theta}$, you can apply the inverse function by taking the cubed root of $\hat{\theta}$ and obtain the MLE of $a$ (i.e. $\hat{a}=\hat{\theta}^{1\over{3}}$)

Update:

I've added the proof mentioned by Thomas Lumley in the comments:

Let $\hat{\eta}$ denote the value that maximizes $L^*(\eta|\textbf{x})$. We must show that $L^*(\hat{\eta}|\textbf{x})$=$L^*(\tau(\hat{\theta})|\textbf{x})$. The maxima of $L$ and $L^*$ coincide, so we have

\begin{eqnarray*} L^{*}(\hat{\eta}|\textbf{x}) & = & \underset{\eta}{\text{sup}}\underset{\{\theta:\tau(\theta)=\eta\}}{\text{sup}}\,L(\theta|\textbf{x})\\ & = & \underset{\theta}{\text{sup}}L(\theta|\textbf{x})\\ & = & L(\hat{\theta}|\textbf{x}), \end{eqnarray*}

The first and third equalities hold by definition of $L^{*}$ and $\hat{\theta}$ respectively, and the second equality holds because the iterated maximization is equal to the unconditional maximization over $\theta$, obtained at $\hat{\theta}$. Further,

\begin{eqnarray*} L(\hat{\theta}|\textbf{x}) & = & \underset{\{\theta:\tau(\theta)=\tau(\hat{\theta})\}}{\text{sup}}L(\theta|\textbf{x})\\ & = & L^{*}\left[\tau(\hat{\theta})|\textbf{x}\right]. \end{eqnarray*}

Hence, the string of equalities shows that $L^{*}(\hat{\eta}|\textbf{x})=L^{*}\left[\tau(\hat{\theta})|\textbf{x}\right]$ and that $\tau(\hat{\theta})$ is the MLE of $\tau(\theta)$. $\blacksquare$

StatsStudent
  • 10,205
  • 4
  • 37
  • 68
  • 2
    Perhaps worth noting that the equivariance property holds even if the transformation isn't invertible. – Thomas Lumley Aug 17 '20 at 00:40
  • Good point, @ThomasLumley. I don't think I made that clear and, in fact, may have obfuscated that point with my example (or the OP's example). – StatsStudent Aug 17 '20 at 00:45
  • Wow. Very cool (and a bit surprising!). No conditions on tau? Do you happen to have a good statement of the theorem – josinalvo Aug 17 '20 at 02:58
  • There are no conditions. The first sentence above is a good statement. It's exactly what you'd find in any textbook definition. – StatsStudent Aug 17 '20 at 03:05
  • 3
    The *proof* is much more straightforward for invertible transformations. This question https://math.stackexchange.com/questions/3246587/does-equivariance-of-the-mle-require-the-function-be-invertible gives a reference to the text by Casella and Berger for the general result. – Thomas Lumley Aug 17 '20 at 05:12
  • 2
    @ThomasLumley, I've added proof under the Update heading in case the OP does not have access to the book. – StatsStudent Aug 17 '20 at 05:32