Visually distinct clouds, but stuck together

Question

I have 96 features and the labels are represented by 1 and -1 for inputting to a deep learning model.

1- PCA

Here the 3 axis represent the 3 first principal components. The blue cloud represents the labels 1 and the red cloud represents the labels -1.

Even if we can identify two different clouds visually, they are stuck together. I think we can face a problem during the training phase because of that.

2- t-SNE

For the same features and labels with t-SNE, we can still distinguish two clouds, but again they are stuck together.

Questions :

1- Does the fact that the two clouds of dots are stuck together affect the % accuracy during the training and testing phase?

2- When we remove the red and blue color, we have somehow only one big cloud. Is there a way to work around the problem of the two clouds being stuck together?

Answer 1

As discussed in the thread "Are dimensionality reduction techniques useful in deep learning" employing a dimensionality reduction step is most likely unnecessary and even potentially harmful when this output is subsequently used as input to a classifier. Particularly to the use case presented, having 96 features is fully manageable by a neural network in terms of input dimensionality. It is common to have this many features for regression, let alone a neural network. Finally, as discussed in the tread "Why is t-SNE not used as a dimensionality reduction technique for clustering or classification?" using t-SNE as a preprocessor to a learning algorithm is not a great idea. The applicability of the learning algorithm will suffer as its use with new data will be problematic. One can probably gain more by using an autoencoder as a dimensional reduction step.

These said, it appears that the current application of t-SNE does not provide great results. Sparse, large spherical clouds, suggest that all points are "away from the others" and points have not started to meaningfully cluster yet. Maybe we need to change the perplexity parameter used, maybe we need to change the number of iterations, maybe we need to change the initial exaggeration factor... In any case, the results shown are not compelling currently; if anything, PCA appears to be doing a better job in providing visually distinct clusters.

To touch upon your questions directly:

1. The fact that the clouds are "stuck together", certainly does not immediately help a learning algorithm. The rationale of using dimensional reduction as input to a learning algorithm (aside reducing the number of features we work with) is to make meta-features with stronger discriminatory nature than our original features; this does not appear to be the case here visually. Nevertheless, without seeing actual performance metrics from the learning algorithms using the raw and them the reduced dimensionality data this a bit of a guess. Cross-validate and retry. :)
2. The full work-around is to use the raw data directly as input to the learning algorithm. The two clouds being "stuck together" is not really the problem we need to solve for a learning task. The whole idea of using a very strong learning algorithm like a deep neural network is for the algorithm to do automatic feature construction internally. We can try an autoencoder that might give us more visually appealing results but once more this will address an intermediate step.