I am having trouble getting my head around what we are really trying to do with our unsupervised visualisation algorithms.
I can't help but look at common techniques like t-SNE, PCA, clustering etc.. and think that our 'metric' is, "ohh, that looks pretty"... or simply that it matches our intuitions.
What I am trying to say is that I find the whole thing to be a bit too subjective for my tastes.
Is there a more precise definition of the aim of these algorithms?
An unsupervised learning approach is by definition used on problems for which you don't have any predefined target attribute to learn (from). The objective may be for example to gain new insights from the data you've just collected or to find natural groupings of data instances.
The major challenge here is that one is usually required to possess or to gain some domain knowledge prior to being able to recognise anything. That's what you've called intuition or subjectivity. The more domain knowledge and experience you'll bring into the task, the more you'll be able to carry away.
For example, a simple scatter plot used for data exploration can be cryptic if you don't know anything about the data. You may perfectly understand what a scatter plot does but there's no objective metric than can help you understand what's going on in such a plot, unless you know what you are looking for. Are you looking for a linear or non-linear relationship between two variables? Are you looking for groups of data ? etc.
It is quite similar with a visualisation of principle components because it really depends on why you want to use PCA in the first place:
data exploration of quantitative data? Yes, the interpretation of the PCA visualisations will depend a lot on your knowledge and intuition about the data and its context. There exists of course guidance on which data points to interpret but such guidance can only provide directions;
dimensionality reduction in the context of supervised or unsupervised learning (e.g. transforming a bunch of ordinal binary variables into a single continuous one) ? Here you use the PCA visualisation to confirm the meaningfulness of the variables transformed in this manner: e.g. are they ordered in the expected way along the first PC ? The interpretation is less subjective because you have the original variables to compare with;
outlier or, more precisely, leverage point detection. In this case, a PCA visualisation will help you to see and characterise the outlying observations. Such observations cause a lot of variance and PCA is one of the multivariate tools to detect them.
You could also use clustering for dimensionality reduction prior to supervised learning, and you can even use a cluster bootstrap to identify outlying observations.
Finally, there is the semi-supervised learning approach. You can use e.g. a decision tree to help you with describing the differences between clusters, i.e. you use the cluster output as if it was a well-defined target variable. This makes the clustering exercise a bit less subjective.
Regarding the question on the metrics, your question appears tightly linked with this one.
