New revolutionary way of data mining?

Question

The following excerpt is from Schwager's Hedge Fund Market Wizzards (May 2012), an interview with the consistently successful hedge fund manager Jaffray Woodriff:

To the question: "What are some of the worst errors people make in data mining?":

A lot of people think they are okay because they use in-sample data for training and out-of-sample data for testing. Then they sort the models based on how they performed on the in-sample data and choose the best ones to test on the out-of-sample data. The human tendency is to take the models that continue to do well in the out-of-sample data and choose those models for trading. That type of process simply turns the out-of-sample data into part of the training data because it cherry-picks the models that did best in the out-of-sample period. It is one of the most common errors people make and one of the reasons why data mining as it is typically applied yields terrible results.

The interviewer than asks: "What should you be doing instead?":

You can look for patterns where, on average, all the models out-of-sample continue to do well. You know you are doing well if the average for the out-of-sample models is a significant percentage of the in-sample score. Generally speaking, you are really getting somewhere if the out-of-sample results are more than 50 percent of the in-sample. QIM's business model would never have worked if SAS and IBM were building great predictive modeling software.

My questions
Does this make any sense? What does he mean? Do you have a clue - or perhaps even a name for the proposed method and some references? Or did this guy find the holy grail nobody else understands? He even says in this interview that his method could potentially revolutionize science...

Answer 1

Not sure if there'll be any other "ranty" responses, but heres mine.

Cross Validation is in no way "new". Additionally, Cross Validation is not used when analytic solutions are found. For example you don't use cross validation to estimate the betas, you use OLS or IRLS or some other "optimal" solution.

What I see as a glaringly obvious gap in the quote is no reference to any notion of actually checking the "best" models to see if they make sense. Generally, a good model makes sense on some intuitive level. It seems like the claim is that CV is a silver bullet to all prediction problems. There is also no talk off setting up at the higher level of model structure - do we use SVM, Regression Trees, Boosting, Bagging, OLS, GLMS, GLMNS. Do we regularise variables? If so how? Do we group variables together? Do we want robustness to sparsity? Do we have outliers? Should we model the data as a whole or in pieces? There are too many approaches to be decided on the basis of CV.

And another important aspect is what computer systems are available? How is the data stored and processed? Is there missingness - how do we account for this?

And here is the big one: do we have sufficiently good data to make good predictions? Are there known variables that we don't have in our data set? Is our data representative of whatever it is we're trying to predict?

Cross Validation is a useful tool, but hardly revolutionary. I think the main reason people like is that it seems like a "math free" way of doing statistics. But there are many areas of CV which are not theoretically resolved - such as the size of the folds, the numbers of splits (how many times do we divide the data up into $K$ groups?), should the division be random or systematic (eg remove a state or province per fold or just some random 5%)? When does it matter? How do we measure performance? How do we account for the fact that the error rates across different folds are correlated as they are based on the same $K-2$ folds of data.

Additionally, I personally haven't seen a comparison of the trade off between computer intensive CV and less expensive methods such as REML or Variational Bayes. What do we get in exchange for spending the addiional computing time? Also seems like CV is more valuable in the "small $n$" and "big $p$" cases than the "big $n$ small $p$" one as in "big $n$ small $p$" case the out of sample error is very nearly equal to the in sample error.

Answer 2

Does this make any sense? Partly.

What does he mean? Please ask him.

Do you have a clue - or perhaps even a name for the proposed method and some references?

Cross Validation. http://en.wikipedia.org/wiki/Cross-validation_(statistics)

Or did this guy find the holy grail nobody else understands? No.

He even says in this interview that his method could potentially revolutionize science... Perhaps he forgot to include the references for that statement ...

Answer 3

His explanation about a common error in data mining seems sensible. His explanation of what he does does not make any sense. What does he mean when he says "Generally speaking, you are really getting somewhere if the out-of-sample results are more than 50 percent of the in-sample."? Then bad-mouthing SAS and IBM doesn't make him look very smart either. People can have success in the market without understanding statistics and part of success is luck. It is wrong to treat successful businessmen as if they are guru's of forecasting.

Answer 4

You can look for patterns where, on average, all the models out-of-sample continue to do well.

My understanding of the word patterns here, is he means different market conditions. A naive approach will analyse all available data (we all know more data is better), to train the best curve fitting model, then run it on all data, and trade with it all the time.

The more successful hedge fund managers and algorithmic traders use their market knowledge. As a concrete example the first half hour of a trading session can be more volatile. So they'll try the models on all their data but for just that first half hour, and on all their data, but excluding that first half hour. They may discover that two of their models do well on the first half hour, but eight of them lose money. Whereas, when they exclude that first half hour, seven of their models make money, three lose money.

But, rather than taking those two winning models and use them in the first half hour of trading, they say: that is a bad time of day for algorithmic trading, and we're not going to trade at all. The rest of the day they will use their seven models. I.e. it appears that the market is easier to predict with machine learning at those times, so those models have more chance of being reliable going forward. (Time of day isn't the only pattern; others are usually related to news events, e.g. the market is more volatile just before key economic figures are announced.)

That is my interpretation of what he is saying; it may be totally wrong, but I hope it is still useful food for thought for somebody.

Answer 5

As a finance professional I know enough context that the statement does not present any ambiguity. Financial time series are often characterized with regime changes, structural breaks, and concept drift, so cross-validation as practiced in other industries is not as successful in financial applications. In the second part he refers to a financial metric, either return on investment on Sharpe ratio (return in the numerator), not MSE or other loss function. If in-sample strategy produces 10% return, then in real trading it may quite realistically produce only 5%. The "revolutionary" part is most certainly about his proprietary analysis approach, not to the quotes.