2

Consider the following model:

$${\bf y} = {\bf X}{\bf b} + {\bf e}$$

where ${\bf y}, {\bf n}\in {\cal R}^m$, ${\bf b}\in{\cal R}^n$, and ${\bf X}\in{\cal R}^{m \times n}$ where $m>n = {\rm rank}({\bf X})$. ${\bf e}$ is an error vector (with independent, identically distributed components of zero mean and variance $\sigma^2$).

Upon observation of ${\bf y}$, an estimate of ${\bf b} =[b_0, \cdots, b_{n-1}]^T$ (where $[\cdot]^T$ denotes transpose) is sought subject to bounding constraints on each parameter:

$$b_i \in [l_i,u_i]\quad i\in\{0,1,\cdots, n-1\}$$ There are numerical packages to solve such problems.

In the absence of these constraints, it is well-known that the least squares estimate of $\bf b$ is unbiased with covariance $\sigma^2 ({\bf X}^T {\bf X})^{-1}$.

Do simple results exist for the bias and covariance of the estimation error exist under the constrained parameter scenario?

In addition, assuming, say, normality of the error vector, are there simple expressions for the Cramer Rao Bound (CRB) of $\bf b$ under the parameter constraints?

I think I may have a solution at least for the CRB. As per http://davegiles.blogspot.com/2015/05/maximum-likelihood-estimation.html, I could non-linearly re-parameterize the unknowns such that the constraints are automatically satisfied:

$$b_i = f(\theta_i) = l_i + (u_i-l_i)e^{\theta_i}/(1+e^{\theta_i}).$$
The CRB of $\{\theta_i\}$ is easily calculated for normally distributed errors. Determination of the CRB of the transformation to a "new" set of variables $\{b_i\}$ is also easy. Any thoughts? Does this seem correct?

amoeba
  • 93,463
  • 28
  • 275
  • 317
rhz
  • 205
  • 1
  • 4
  • Not a full answer, but with an appropriate shifting and rescaling of the data, this range constraint might be equivalent to ridge regression or the lasso on the subset of bounded parameters. – jwimberley Dec 26 '16 at 03:53
  • How would you define the bias for $b_i$: in terms of its limitation to $[l_i,u_i]$, or in terms of possible values of $b_i$ over all real numbers? For example, if the "true" $b_i$ is much lower than $l_i$, the bias and variance for $b_i$ values limited to $[l_i,u_i]$ would probably be very small, in the sense of how the estimate would change over multiple replications of the same experiment. Nevertheless, the bias with respect to the "true" value, much lower than $l_i$, would be high. – EdM Dec 27 '16 at 20:28
  • For this problem, assume that the true $b_i$ would never be lower than $l_i$. Rather, the true parameter is constrained to the interval $[l_i, u_i]$ and the estimator is given exact knowledge of the interval. That is: assume that there is no model mis-match in the sense that $b_i$ is guaranteed to reside in $[l_i, u_i]$. – rhz Dec 27 '16 at 20:51
  • Bias is defined in the standard manner: bias = E[\hat{b}_i]-b_i where the expectation is over all realizations of the noise vector ${\bf e}$. – rhz Dec 27 '16 at 20:54
  • As noted by @jwimberley your problem, at least the MAP estimate part, can be transformed to an equivalent LASSO problem. I do not believe there are any simple ["standard error" formulas for LASSO](http://stats.stackexchange.com/q/91462/127790). However Ch. 6 of [Statistical Learning With Sparsity (2015)](http://web.stanford.edu/~hastie/StatLearnSparsity/) summarizes relevant approaches to quantifying parameter uncertainty for LASSO. – GeoMatt22 Dec 28 '16 at 05:40

1 Answers1

1

Your idea could work, but then the problem becomes a more generic likelihood maximization problem: it is no longer an ordinary least-squares regression and the maximization might need to be performed numerically, and edge conditions could be tricky to handle.

A better method is use quadratic programming to solve for the minimum squared residuals given your constraints. This proceeds in a manner analogous to the method of Lagrange multipliers, where an expression is extremized given an equality condition. The basic idea is to take the expression for the squared residuals and the inequalities and form their Wolf Dual. Then this dual object can be much more readily extremized. The original paper by Wolfe solves your problem explicitly and is highly regarded.

Now, regarding the covariance of the estimate: This is much trickier. It's quite likely that if the least-squares estimate of the parameter $\beta$ is outside the constraint range, the quadratic optimization value will be on the boundary. The covariance is likely to be ill-defined. There are two alternate methods you could consider: ridge regression and the lasso. There are popular packages for performing these types of regression. One interpretation of them is that they place an a priori probability density on the parameters. These probabilities are, as it happens, normal distributions and absolute-exponential distributions with means of 0, respectively, but you could redefine and adapt your problem so that either of these is suitable. Since the a priori pdfs are smooth, so are the posterior pdfs, and the covariances are well-defined.

jwimberley
  • 3,679
  • 2
  • 11
  • 20
  • This is interesting. I am currently using spicy’s lsq_linear. It appears to give acceptable results. My concern is more about bias and covariance. You raise an interesting question regarding scenarios where some of the parameters are near their minimum/maximum values. This will likely result in biased estimates. Perhaps the CRB (which assumes unbiasedness or at least known bias) is not the correct tool to describe the inherent estimation accuracy limitations associated with this interval restricted parameter estimation problem. – rhz Dec 28 '16 at 18:26