Apologies if this is a naïve question, but it's not so naïve to me!
Let's first assume we have 2D data which are perfectly linear but not along the x- or y-axis. PCA will rotate it so that it becomes parallel to x- or y-axis, and the direction of maximum variation will be the 'first' direction.Let's next assume that the data is deviating quite a bit from being perfectly linear. In this case, if we do PCA no matter what, the direction w/ smaller variance is going to be that of noise. My question is: is there a threshold $T$ that we can determine apriori (=before doing anything at all) from the data that'll tell us that the PCA is of use or no use? If yes, how can we determine $T$? For example, one must not apply PCA to sphere or circle-valued data $\{(cos \theta_j, sin \theta_j):1\leq j \leq n\}$, because there's no linearity there.
For two dimensional data (i.e. data with just two features), we can plot them and see if they're close to being linear or not. But for high dimensional data, what can we do? Is there a way we can determine a scalar or a set of scalars that'll tell us whether PCA is any good?
