t-SNE dimensions as additional predictor variables

Question

This question could also (maybe) relate to PCA. I built a supervised RandomForest on a dataset that I'm currently working on - the actual V Prediction $R^2$ was holding around 80% across many CV folds.

I then decided to look at the data from an unsupervised standpoint by dropping the Response variable and using t-SNE and noticed that we had interesting structure to the data.

I used the t-SNE output (3 columns) and added them my original data set as 3 more predictor variables and rebuilt the Model. The performance of the model jumped to 90%.

I'm looking for a simple explanation of why this would have happened? Is it okay to do this? How would one actually make live real world predictions using this approach? (I mean the t-SNE predictor vars are formed from the whole dataset, which I wouldn't know at "Prediction time").

Answer 1

When you computed the tSNE representation of the data, presumably you used all of the data and so the data in your CV hold-out set was actually considered in your training data. It is built into the tSNE components. I do not believe that there is any way to predict the tSNE coordinates of new data points, so this method does not seem to be useful for building a classifier.

PCA is another story. You could compute PCA on your training folds and then project new data onto the principal components (from training data only). As long as you do not compute the principal components using all of the data before applying other methods, this should be OK

Answer 2

Not worthy of an answer, but too long for a comment. I am also interested in this question. The theory seems quite difficult as you are layering a non-linear algorithm over top of another one. TL;DR I'm surprised it works very well, I would look into the hyperparameters of your random forest. But since the theory is quite complex there may be something happening that I don't understand.

First, an intuitive understanding. t-SNE attempts to map data from a high dimensional space down to a lower dimensional space while keeping the distance between points which are 'close' in high-dimensions but without caring about points which are not close. In effect, it creates clusters of the points which are near one another in predictor space and maps them to two (or three) dimensions. This, along with a classification scheme can provide reasonable accuracy. Consider the example from the original paper of t-SNEs application to the MNIST dataset -- in two dimensions you can easily pick out the digits by eye with ~90% accuracy, a k-means classifier layered on top will do about that or better.

A well-specified random forest ought to pick out that structure in the high-dimensional space without the need to first map it down. As you are not adding any information, and you accept some loss in t-SNE you are (in some sense) just placing your data on a 3-dimensional manifold and appending this lossy new dataset to your existing set. As random forests already have a means of regularizing, it is unclear why your result is improved. It makes me skeptical of how well your hyperparameters are set, i.e. it may be the case that your Random Forest is not regularizing very well so that you have t-SNE do it for you your accuracy improves. Setting the hyperparameters correctly in the first place and removing the t-SNE ought to further improve on this performance.

Answer 3

Do not use $t$-SNE as predictor variables (in general).

As it has been pointed out in a previous answer in a best-case scenario using all the data to compute the $t$-SNE representation is a bad practice even if one holds out the response variable. In addition using $t$-SNE directly also renders the original modelling procedure useless when presented with new data (as the OP correctly comments too) because we cannot reconstruct a new embedding.

As a work-around I think it is worth exploring the option of a deep auto-encoder. This will allow use to readily map new data on our "interesting representation". Deep auto-encoders can learn highly non-linear representations of a dataset. We will train the auto-encoder on our training folds (similar to what we would do with PCA) and then project new data accordingly. amoeba's answer on "Building an autoencoder in Tensorflow to surpass PCA" is an excellent resource to get you started.

In specific cases, one might be able to build a deep neural network to replicate the output of $t$-SNE on a new sample. This option will take care of the above issues (data reuse, no generalisation to unseen data). I do not think it is a substantially different approach from using a deep auto-encoder in the first place though and conceptually I find it more convoluted. Plus the auto-encoder literature has many extensions (eg. denoising auto-encoders) that might come handy.

Regarding the exact phenomenon of 10%+ performance boost reported: Assuming that the response variable is not accidentally used, I suspect that the data might have an non-linear embedding that the RF classifier cannot easily detect. Random forests (or any other model for that matter) do not guarantee that any non-linear association will be discovered. I would suggest examining the $t$-SNE output itself. Maybe some of the patterns are not so hard to explain and a bit of clever/educated features engineering from your part can give you a decent boost.

I hope you see that, somewhat anti-climatically, using $t$-SNE output directly is not very straight-forward. $t$-SNE is a very helpful tool; nevertheless as many great tools, it is easy for us to misuse it as a hammer when we just have screws.

Answer 4

I'll also assume, as did G5W, that you used all your data in generating your coordinates. Given this, I'm aware of perhaps two mechanisms to use t-SNE as a predictor on a go-forward basis:

• For all new "live" data, rebuild the model after running t-SNE on your entire dataset. The order would be something like: 1) Get new data and put it alongside your historical data. 2) Run t-SNE on all data (new & old). 3) Properly orient your train/test/live datasets and rebuild the model using new coordinates in 3-dimensional (or N-dimensional) t-SNE space. 4) Score model on the recently generated coordinates of the new 'live' data. This approach would be valid for batch scoring, but may come at a potentially significant time cost. You may also find issues with prediction stability, as previous scores 'float' around, and coordinates/models are re-calculated.

• Or, using the t-SNE interpretations of your training data, build individual models for your individual coordinate components. This is alluded to by Laurens van der Maaten at the bottom of his t-SNE FAQ. [https://lvdmaaten.github.io/tsne/]. Then, use the component models to estimate the coordinates for the new 'live' data. This, however, will bring new sources of error to be minimized with crossval and tuning of the individual x,y,z model components.