9

I'm trying to predict a response variable in linear regression that should be always positive (cost per click). It's a monetary amount. In adwords, you pay google for clicks on your ads, and a negative number would mean that google pays you when people clicked :P

The predictors are all continuous values. The Rsquared and RMSE are decent when compared to other models, even out-of-sample:

  RMSE        Rsquared 
1.4141477     0.8207303

I cannot rescale the predictions, because it's money, so even a small rescaling factor could change costs significantly.

As far as I understand, for the regression model there's nothing special about zero and negative numbers, so it finds the best regression hyperplane no matter whether the output is partly negative.

This is a very first attempt, using all variables I have. So there's room for refinement.

Is there any way to tell the model that the output cannot be negative?

Glen_b
  • 257,508
  • 32
  • 553
  • 939
usillos
  • 91
  • 1
  • 1
  • 2
  • 6
    You can ensure positive predictions by using a generalized linear model with logarithmic link function. By the way, although your $R^2$ value is quite encouraging, a better check of whether the model follows the main shape of the data is a plot of residual vs predicted. Plots of observed vs predicted may also help illuminate your problem. – Nick Cox Oct 13 '13 at 10:24
  • 1
    @NickCox gave one suggestion. I would plot the data in more ways than just residual vs. predicted. However, you can certainly rescale money variables. One common method is to take log(cost) as the dependent variable. (I think this winds up equivalent to the log link function, but might be easier to comprehend). Log(cost) can, of course, be negative. And logs of money variables are often sensible because, e.g. a difference between 0.01 and 0.02 per click is important, but difference between 1.01 and 10.2 per click is not. – Peter Flom Oct 13 '13 at 13:38
  • 1
    @Peter Flom I think meant 1.02 not 10.2. – Nick Cox Oct 13 '13 at 16:06
  • 1
    A little sample data would help people illustrate potential solutions. – Glen_b Oct 13 '13 at 22:21

1 Answers1

4

I assume that you are using the OLS estimator on this linear regression model. You can use the inequality constrained least-squares estimator, which will be the solution to a minimization problem under inequality constraints. Using standard matrix notation (vectors are column vectors) the minimization problem is stated as

$$\min_{\beta} (\mathbf y-\mathbf X\beta)'(\mathbf y-\mathbf X\beta) \\s.t.-\mathbf Z\beta \le \mathbf 0 $$

...where $\mathbf y$ is $n \times 1$ , $\mathbf X$ is $n\times k$, $\beta$ is $k\times 1$ and $\mathbf Z$ is the $m \times k$ matrix containing the out-of-sample regressor series of length $m$ that are used for prediction. We have $m$ linear inequality constraints (and the objective function is convex, so the first order conditions are sufficient for a minimum).

The Lagrangean of this problem is

$$L = (\mathbf y-\mathbf X\beta)'(\mathbf y-\mathbf X\beta) -\lambda'\mathbf Z\beta = \mathbf y'\mathbf y-\mathbf y'\mathbf X\beta - \beta'\mathbf X'\mathbf y+ \beta'\mathbf X'\mathbf X\beta-\lambda'\mathbf Z\beta$$

$$= \mathbf y'\mathbf y - 2\beta'\mathbf X'\mathbf y+ \beta'\mathbf X'\mathbf X\beta-\lambda'\mathbf Z\beta $$

where $\lambda$ is a $m \times 1$ column vector of non-negative Karush -Kuhn -Tucker multipliers. The first order conditions are (you may want to review rules for matrix and vector differentiation)

$$\frac {\partial L}{\partial \beta}= \mathbb 0\Rightarrow - 2\mathbf X'\mathbf y +2\mathbf X'\mathbf X\beta - \mathbf Z'\lambda $$

$$\Rightarrow \hat \beta_R = \left(\mathbf X'\mathbf X\right)^{-1}\mathbf X'\mathbf y + \frac 12\left(\mathbf X'\mathbf X\right)^{-1}\mathbf Z'\lambda = \hat \beta_{OLS}+ \left(\mathbf X'\mathbf X\right)^{-1}\mathbf Z'\xi \qquad [1]$$

...where $\xi = \frac 12 \lambda$, for convenience, and $\hat \beta_{OLS}$ is the estimator we would obtain from ordinary least squares estimation.

The method is fully elaborated in Liew (1976).

Alecos Papadopoulos
  • 52,923
  • 5
  • 131
  • 241
  • 3
    I have upvoted this because it's a legitimate solution, but it is risky in practice. After all, the solution is arbitrarily sensitive to values of $\mathbf Z$: a single high-leverage value will steer the estimates far from a decent fit merely to enforce the constraint. Thus, at a minimum, this procedure *must* be accompanied by a careful goodness-of-fit test to the data. – whuber Oct 13 '13 at 17:24
  • @whuber You are right. So, OP, tread carefully here. – Alecos Papadopoulos Oct 13 '13 at 17:30