Bayesian inverse modeling with non-identifiable parameters?

Question

If I have a physical model

\begin{equation} y = \frac{1}{\beta_0} (\beta_1 x_1 + \beta_2 x_2) \end{equation}

and want to estimate coefficients $\beta_0$, $\beta_1$, and $\beta_2$ from given data set $(y, x_1, x_2)$, with regression I would typically have to combine them into two coefficients $\beta_1/\beta_0$ and $\beta_2/\beta_0$ since that would be what I would be able to solve (contributions from $1/\beta_0$ and $\beta_1$ cannot be distinguished from each other individually).

However, if I have separate prior distributions for $\beta_0$, $\beta_1$, and $\beta_2$ - is it possible to obtain sensible posterior distributions for each of them using MCMC or Laplace's method without combining the variables?

Edits Corrected title and variable name error based on comments and answers.

Answer 1

Adding priors does not solve the identifiability problem

This is an case where the parameters are non-identifiable in your model. As you point out, contributions from the individual non-identifiable parameters in these ratios cannot be distinguished using the data.

When using Bayesian analysis with a non-identifiable model, specification of a prior for all the individual non-identifiable parameters will still lead you to a valid posterior, but this is strongly affected by the prior. The posterior for the non-identifiable parameters converges to a fixed asymptotic distribution that also depends heavily on the prior, so it lacks posterior consistency.

The fact that you get a valid posterior, and this converges to a fixed asymptotic distribution, often gives the misleading impression that Bayesian analysis renders the identifiability problem benign. However, it is crucial to note that the posterior in these cases is strongly affected by the prior in ways that do not vanish as we get more and more data. The identifiability problem is not rendered benign merely by using Bayesian analysis with priors.

---

Posterior depends heavily on prior: To see exactly what I mean, define the minimal sufficient parameters $\phi_1 \equiv \beta_1 / \beta_0$ and $\phi_2 \equiv \beta_2 / \beta_0$. These are the parameters that are identified in the present model. Using the rules for density transformation, the posterior distribution for the three non-identifiable parameters of interest can be written as:

$$\begin{equation} \begin{aligned} \pi(\beta_0, \beta_1, \beta_2 | \mathbf{x}, \mathbf{y}) &= \frac{1}{\beta_0^2} \cdot \pi(\beta_0, \phi_1, \phi_2 | \mathbf{x}, \mathbf{y}) \\[6pt] &= \frac{1}{\beta_0^2} \cdot p(\beta_0 | \phi_1, \phi_2) \cdot \pi(\phi_1, \phi_2 | \mathbf{x}, \mathbf{y}). \\[6pt] \end{aligned} \end{equation}$$

Now, the posterior $\pi(\phi_1, \phi_2 | \mathbf{x}, \mathbf{y})$ for the minimal sufficient parameters (which are identifiable) is determined by the prior assumptions and the data as normal. However, the density $p(\beta_0 | \phi_1, \phi_2)$ is determined purely by the prior (i.e., it does not change as you get more data). This latter density is just an aspect of the assumed prior on the three non-identifiable parameters. Hence, the posterior of the non-identifiable parameters will be determined in large measure by a part that is purely a function of the prior.

Posterior converges for indentifiable parameters, not non-identifiable parameters: Bayesian asymptotic theory tells us that, under broad conditions, the posterior distribution of identifiable parameters converges towards a point-mass on the true values. (More specifically, there are a number of convergence results that show asymptotic convergence to a normal distribution with mean that approaches the true identifiable parameter values and variance that approaches zero.) In the context of regression there are some additional convergence conditions on the explanatory variables, but again, that convergence result holds broadly.

Under appropriate conditions, as $n \rightarrow \infty$ the density $\pi(\phi_1, \phi_2 | \mathbf{x}, \mathbf{y})$ will converge closer and closer to a point-mass distribution on the true values $(\phi_1^*, \phi_2^*)$. In the limit the posterior distribution for the non-identifiable parameters converges to a limiting distribution determined by the prior (that is not a point-mass):

$$\begin{equation} \begin{aligned} \pi(\beta_0, \beta_1, \beta_2 | \mathbf{x}, \mathbf{y}) &\rightarrow \pi(\beta_0, \beta_1, \beta_2 | \mathbf{x}_\infty, \mathbf{y}_\infty) \\[6pt] &\propto \frac{1}{\beta_0^2} \cdot p(\beta_0 | \phi_1^* = \beta_1 / \beta_0, \phi_2^* = \beta_2 / \beta_0). \\[6pt] \end{aligned} \end{equation}$$

We can see that this asymptotic density is affected by the data only through the true values of the minimal sufficient parameters. It is still heavily affected by the form of the density $p(\beta_0 | \phi_1, \phi_2)$, which is a function of the prior. Although the posterior for the identifiable parameters has converged to a point-mass on the true values, the posterior density for the non-identifiable parameters $\beta_0, \beta_1, \beta_2$ still retains uncertainty even in this limit. Its distribution is now entirely determined by the prior, conditional on holding the identifiable parameters fixed.

In the other answer by Björn you can see that he gives an excellent example of this phenomenon in the simple case of IID data from a normal distribution with a mean that is a ratio of two non-identifiable parameters. As you can see from his example, with a large amount of data there is posterior convergence for the identifiable mean, but the corresponding posterior for the non-identifiable parameters is still highly variable (and almost entirely dependent on the prior).

Conclusion: In Bayesian analysis you can assign a prior to a set of non-identifiable parameters and you get a valid posterior. However, despite the fact that we get a valid posterior, and asymptotic convergence of the posterior to a limiting distribution, all of those results are heavily affected by the prior, even with an infinite amount of data. In other words, don't let that fool you into thinking that you have "solved" the identifiabiity problem.

Answer 2

Yes, with a Bayesian analysis you can get sensible posteriors and concentrate around sensible values to the point that the combination of information in prior and likelihood allow it. In this sense the Bayesian analysis can deal with this kind of situation a lot better than a frequentist analysis. However, it cannot get around the fundamental non-identifiability of the parameters in a model and the posteriors will still reflect the lack of identifiability of the parameters.

To use an example, let's use a simpler model that is just $\log Z_i \sim N(\theta_1/\theta_2, \sigma^2)$ and let's assume that we have a huge amount of data. We'll have hardly any uncertainty around the ratio $\theta_1/\theta_2$, but lots of values for each parameter are still getting support in the likelihood. Thus, we just get still a wide marginal posterior distribution and the joint distribution does have a very strong correlation between the two parameters.

This is illustrated with example code below (using the re-parameterization $Y_i := \log Z_i$ and $\beta_j := \log \theta_j$).

Obviously, the marginal posterior would be wider, if our prior for the parameters had been wider and the posterior correlation less strong, if we had less data (then the parameter identifiability problem would be less obvious).

I would expect something similar to happen in your example - whether that's a problem or not is a different matter.

library(rstan)
library(bayesplot)
y <- exp(rnorm(10000, 0,1))

stancode <- "
data {
  int n;
  real y[n];
}

parameters{
  real beta0;
  real beta1;
  real<lower=0> sigma;
}

model {
  beta0  ~ normal(0,1);
  beta1 ~ normal(0,1);
  sigma ~ normal(0,1);
  y ~ normal(exp(beta1-beta0), sigma);
}
"

stanfit <- stan(model_code=stancode, data=list(n=length(y), y=y))

posterior <- as.matrix(stanfit)

mcmc_pairs(posterior, pars = c("beta0", "beta1"))