Hiding a Regression Model from Professor (Regression Battleship)

Question

I'm working on a homework assignment where my professor would like us to create a true regression model, simulate a sample of data and he's going to attempt to find our true regression model using some of the techniques we have learned in class. We likewise will have to do the same with a dataset he's given us.

He says that he's been able to produce a pretty accurate model for all past attempts to try and trick him. There have been some students that create some insane model but he arguably was able to produce a simpler model that was just sufficient.

How can I go about developing a tricky model for him to find? I don't want to be super cheap by doing 4 quadratic terms, 3 observations, and massive variance? How can I produce a seemingly innocuous dataset that has a tough little model underneath it?

He simply has 3 Rules to follow:

1. Your dataset must have one "Y" variable and 20 "X" variables labeled as "Y", "X1", ..., "X20".

2. Your response variable $Y$ must come from a linear regression model that satisfies:
   $$ Y_i^\prime = \beta_0 + \beta_1 X_{i1}^\prime + \ldots + \beta_{p-1}X_{i,p-1}^\prime + \epsilon_i $$ where $\epsilon_i \sim N(0,\sigma^2)$ and $p \leq 21$.

3. All $X$-variables that were used to create $Y$ are contained in your dataset.

It should be noted, not all 20 X variables need to be in your real model

I was thinking of using something like the Fama-French 3 Factor Model and having him start with the stock data (SPX and AAPL) and have to transform those variables to the continuously compounded returns in order to obsfucate it a little more. But that leaves me with missing values in the first observation and it's time series (which we haven't discussed in class yet).

Unsure if this is the proper place to post something like this. I felt like it could generate some good discussion.

Edit: I'm also not asking for "pre-built" models in particular. I'm more curious about topics/tools in Statistics that would enable somebody to go about this.

Answer 1

Simply make error term much larger than the explained part. For instance: $y_i=X_{i1}+\epsilon_i$, where $X_{ij}=\sin(i+j)$, $i=1..1000$ and $\sigma=1000000$. Of course, you have to remember what was your seed, so that you can prove to your professor that you were right and he was wrong.

Good luck identifying the phase with this noise/signal ratio.

Answer 2

If his goal is to recover the true data generating process that creates $Y$, fooling your professor is fairly trivial. To give you an example, consider disturbances $\epsilon_i\sim N(0,1)$ and the following structural equations:

$$ X_1 = \epsilon_1 + \epsilon_0\\ X_2 =\epsilon_1 + \epsilon_2\\ y = X_1 + \epsilon_2 $$

Note the true DGP of $Y$, which includes only $X_1$, trivially satisfy condition 2. Condition 3 is also satisfied, since $X_1$ is the only variable to create $Y$ and you are providing $X_1$ and $X_2$.

Yet, there's no way your professor can tell if he should include only $X_1$ only $X_2$ or $X_1$ and $X_2$ to recover the true DGP of $Y$ (if you end up using this example, change the number of the variables). Most likely, he will just give you as an answer the regression with all variables, since they will all show up as significant predictors. You can extend this to 20 variables if you want to, you might want to check this answer here and a Simpson's paradox machine here.

Note all conditional expectations $E[Y|X_1]$, $E[Y|X_2]$ or $E[Y|X_1, X_2]$ are correctly specified conditional expectations, but only $E[Y|X_1]$ reflects the true DGP of $Y$. Thus, after your professor inevitably fails the task, he might argue that his goal was simply to recover any conditional expectation, or to get the best prediction of $Y$ etc. You can argue back that it wasn't what he said, since he states:

variable Y must come from a linear regression model that satisfies (...) variables that were used to create Y (...) your real model (...)

And you might spark a good discussion in class about causality, what true DGP means and identifiability in general.

Answer 3

Use variables with multicollinearity and heteroscedasticity like income versus age: do some painful feature engineering that provides scaling problems: give NAs for some sprinkled in sparseness. The linearity piece really makes it more challenging but it could be made painful. Also, outliers would increase the problem for him upfront.

Answer 4

Are interaction terms allowed? If so, set all the lower order coefficients to 0 and build the entire model out of N-th order interactions (e.g. terms like $X_5X_8X_{12}X_{13}$). For 20 regressors the number of possible interactions is astronomically large and it would be very difficult to find just the ones you included.

Answer 5

Choose any linear model. Give him a data set where most samples are around x=0. Give him few samples around x=1,000,000.

The nice thing here that the samples around x=1,000,000 are not outliers. They are generated from the same source. However, since the scales are so different, errors around 1M won't fit with the errors around 0.

Let's consider an example. Our model is just $$ Y_i^\prime = \beta_0 +\beta_1 X_{i1}^\prime + \epsilon_i $$

We have a data set of n samples, near x=0. We will choose 2 more points in "far enough" values. We assume that these two point have some error.

A "far enough" value is such a value that the error for an estimation the doesn't pass directly in these two points is much larger than the error of the rest of the dataset.

Hence, linear regression will choose coefficients that will pass in these two points and will miss the rest of the dataset and be different from the underlining model.

See the following example. {{1, 782}, {2, 3099}, {3, 110}, {4, 1266}, {5, 1381}, {1000000 ,1002169}, {1000001, 999688}}

This is in WolfarmAlpha series format. In each pair the first item is x and the second was generated in Excel using the formula =A2+NORMINV(RAND(),0,2000).

Hence, $\beta_0=1, \beta_1=1$ and we add normally distributed random noise with mean 0 and standard deviation of 2000. This is a lot of noise near zero but a small one near million.

Using Wolfram Alpha, you get the following linear regression $y= 178433. x - 426805$, which is quite different from the underlining distribution of $y=x$