16

Suppose you fit a linear/logistic regression $g(y) = a_0 + a_1\cdot x_1 + a_2\cdot x_2$, with the aim of an unbiased estimate of $\frac{a_1}{a_2}$. You are very confident that both $a_1$ and $a_2$ are very positive relative to the noise in their estimations.

If you have the joint covariance of $a_1, a_2$, you could calculate, or at least simulate the answer. Are there any better ways, and in real-life problems with a lot of data, how much trouble do you get in for taking the ratio of estimates, or for taking a half-step and assuming the coefficients are independent?

Richard Hardy
  • 54,375
  • 10
  • 95
  • 219
quasi
  • 340
  • 1
  • 7
  • 1
    In logistic regression as described, how do you find an unbiased estimator of $a_0$ or $a_1$? The problem is unrelated with the correlation between the coefficients. – Xi'an Sep 18 '16 at 13:54
  • 6
    Something to ponder: What if one or both of the coefficients were zero? – cardinal Sep 18 '16 at 19:59
  • Yeah, good point. I'm implicitly assuming that both coefficients are sufficiently positive that there's no danger of noise leading to crossed signs (re: http://andrewgelman.com/2011/06/21/inference_for_a/). I'll edit. – quasi Sep 18 '16 at 22:33
  • 2
    How precisely do you estimate $a_1$ and $a_2$ in your regression? Is a consistent estimator with small standard errors sufficient? Is it important that your estimator is unbiased? Would it work for your application to just take $\frac{\hat{a}_1}{\hat{a}_2}$ and calculate the standard-error for that using the [delta method](https://en.wikipedia.org/wiki/Delta_method) and the estimated covariance matrix for $(a_1, a_2)$ from your regression. – Matthew Gunn Sep 18 '16 at 23:07
  • I guess this is an argument for the naive approach: it is biased but at least consistent. That's nice to know, I'm not sure if its enough. To motivate my application, suppose that $x_1$ is a score that reflects an object's characterization in some other application, and $x_2$ represents e.g. an indicator expressing whether some other characteristic is present. In the presence of $x$, I wish to update my score parameter. This is why I need the ratio of estimates: to convert the contribution of $x_2$ into the scale of $x_1$, but you can see why unbiasedness is valuable here. – quasi Sep 19 '16 at 15:06
  • 1
    Have you considered Fieller's theorem? Look here: http://stats.stackexchange.com/questions/16349/how-to-compute-the-confidence-interval-of-the-ratio-of-two-normal-means/16354#16354 – soakley Sep 19 '16 at 21:36
  • Although you might be "very confident" that the estimated denominator is nonzero, nevertheless if there is *any* chance, no matter how small, that it will be zero while the estimated numerator is nonzero, then the expectation of their ratio is undefined and therefore that ratio cannot be unbiased. Since this situation holds for most estimators (except when explicitly constrained to domains that exclude some neighborhood of zero), the directions suggested in the last paragraph do not seem to be fruitful. Some completely different estimator is needed. – whuber Oct 27 '16 at 16:00

1 Answers1

1

I would suggest doing error propagation on the variable type and minimize either the error or relative error of $\frac{a_1}{a_2}$. For example, from Strategies for Variance Estimation or Wikipedia

$f = \frac{A}{B}\,$
$\sigma_f^2 \approx f^2 \left[\left(\frac{\sigma_A}{A}\right)^2 + \left(\frac{\sigma_B}{B}\right)^2 - 2\frac{\sigma_{AB}}{AB} \right]$

$\sigma_f \approx \left| f \right| \sqrt{ \left(\frac{\sigma_A}{A}\right)^2 + \left(\frac{\sigma_B}{B}\right)^2 - 2\frac{\sigma_{AB}}{AB} }$

As a guess, you probably want to minimize $(\frac{\sigma_f}{f})^2$. It is important to understand that when one does regression to find a best parameter target, one has forsaken goodness of fit. The fit process will find a best $\frac{A}{B}$, and this is definitively not related to minimizing residuals. This has been done before by taking logarithms of a non-linear fit equation, for which multiple linear applied with a different parameter target and Tikhonov regularization.

The moral of this story is that unless one asks the data to yield the answer that one desires, one will not obtain that answer. And, regression that does not specify the desired answer as a minimization target will not answer the question.

Carl
  • 11,532
  • 7
  • 45
  • 102