2

My knowledge of principal component analysis is only conceptual; I know nothing about the nuts and bolts of how it works. I learned about it from its use in sociolinguistics, as in Horvath & D. Sankoff (1987), and what I gather from their description is that PCA is basically an automated method of determining which factor is most strongly associated with a given variable of interest.

This sound to me a lot like making the process of p-hacking a bit easier, although it doesn't seem to suggest that PCA inevitably leads to p-hacking as one could simply describe their study as exploratory to avoid accusations of p-hacking. But, am I correct in thinking that, in cases where a researcher is not describing their study as exploratory, the use of PCA would essentially be an automated method for p-hacking?

Horvath, B., & Sankoff, D. (1987). Delimiting the Sydney speech community. Language in Society, 16(2), 179‑204. https://doi.org/10.1017/s0047404500012252

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
joshisanonymous
  • 163
  • 9
  • 2
    Could you describe more clearly the connection you implicitly assume between PCA and hypothesis testing? What exactly is the mechanism whereby you perceive that PCA "leads to" p-hacking or other abuses of hypothesis testing? – whuber Mar 09 '18 at 15:20
  • Given enough factors, one will eventually find one that is significantly associated with the variable of interest, so if PCA will take any number of factors and tell you which ones are most strongly associated with the variable of interest, does this not just make the process of p-hacking easier? For example, if I want to know what factors are associated with dogs having black fur, it seems like I could just identify a huge number of factors, some which would be ridiculous, like "how much cereal their owner eats", and it would eventually find one that's associated with the color of the fur. – joshisanonymous Mar 09 '18 at 15:32
  • 3
    You don't describe the PCA I know, as explained in many posts at https://stats.stackexchange.com/questions/2691. PCA doesn't involve testing or any notion of "significantly associated." You describe some sort of model development or selection process that involves PCA in some way, but it appears to go far beyond what PCA usually means. – whuber Mar 09 '18 at 15:36
  • @joshisanonymous PCA doesn't determine association but can be used for data reduction. I replicated an analysis where investigators used PCA on a set of related outcomes, then dichotomized the first two PCs and used them as regressors in a set of analyses. The traditional psychometric literature would not advocate this approach since a CFA must be done to confirm the factor structure has any generalizability to a population. It seems to me it doesn't contribute to p-hacking rather than a separate, but related issue of lack of reproducibility. Is this perhaps what you're getting at? – AdamO Mar 09 '18 at 16:23
  • Thanks for the comments. It's quite possible that I'm misunderstanding PCA. Here's the relevant description from the source that introduced me to it: "What the principal components program does is to systematically determine all the relevant axes and point out the axes which are most important in accounting for the variance in the data set" (Horvath & D. Sankoff, 1987, p. 186). To me, the "which are most important in account for the variance" suggests that there's some kind of modeling or significance testing or something being done. They also call it a "program", though, so maybe it's not PCA – joshisanonymous Mar 09 '18 at 16:34

1 Answers1

1

Since you don't specify a specific test or comparison, I am going to talk "loosely" about the general idea that you could present a comparison of a component from PCA with the dataset from which it was generated and test for a statistical relationship. This will necessarily be a bit vague, since you have not specified a test, but I don't think it matters too much to my answer. The short answer here is: yes, your intuition is correct --- PCA is a statistical fitting procedure and so it involves an underlying optimisation taken with respect to the observed data. Standard caveats apply when taking the outputs of such methods and then testing them for "significance".

Because PCA is a fitting procedure, use of any of the components already uses an underlying optimisation step. Moreover, PCA orders the principal components by their explained-variance contribution (largest to smallest), which means that the variance contributions of the components are order statistics. As with any application of hypothesis tests to ordered quantities, a proper test should take account of the "optimisation" implicit in this ordering. If you were to take the first principal component (with the largest explained-variance contribution) but treat it as if it were just a random component in a test (e.g., perform a hypothesis test where you use the distribution of a random component variance as the null hypothesis) then you would get a biased test in a manner that is analogous to p-hacking.

So yes, you are essentially correct. If a researcher applies PCA to a dataset and uses this to identify a "principal" component (especially the one with the highest explained-variance contribution) they will automatically tend to pick a component that is highly predictive of the overall data, since it is the result of a fitting procedure to that data. If this were analysed or presented as if it were just a component selected a priori then that would bias the analysis in favour of confirmation of the existence of a statistical relationship between the component and the overall data it was selected from. Instead of doing this the researcher should adjust the p-value for any testing they do by taking account of the fact that they "maximised" over the explained-variance of the components. (The latter is quite complicated, but it can be done.)

Ben
  • 91,027
  • 3
  • 150
  • 376
  • 2
    (+1) Even something as naive as PCA is a fitting procedure. Sometimes this is forgotton. – Cagdas Ozgenc Oct 06 '20 at 07:11
  • 1
    I am not a fan of PCA before regression, but my understanding is that there are two main styles: PCA on possible predictors to produce fewer composite predictors (sometimes just 1), and PCA on possible outcome variables to produce fewer composite outcomes (often just 1). There are reservations about both of these, compounded if the PCA is on a mix of outcomes and predictors. – Nick Cox Oct 06 '20 at 09:21
  • 1
    But the objection to variables being pre-selected before regression can be stretched indefinitely. All predictors are pre-selected from many more that might be used. although the selection might be a matter of principle or based on past performance. In some fields, "theoretical justification" is emphasised, which can mean no more than that some previous researcher suggested that a variable was influential. – Nick Cox Oct 06 '20 at 09:23
  • @Nick Cox: The difference is that PCA is selecting components directly by an optimisation procedure operating on the underlying variables. – Ben Oct 06 '20 at 22:07
  • 1
    I don't think PCA is selecting components at all, as it produces as many components as variables; it's people who select which they want to use. That there's an optimisation is agreed, and clear as the definition of the machinery. – Nick Cox Oct 06 '20 at 22:10
  • Well, if you select the "principal" component, you are selecting the component that had the highest explained variance. – Ben Oct 06 '20 at 22:32
  • Strictly speaking, PCA is not a fitting procedure (but only a descriptive-summarization one). Because it does not model the data by means of a specific small number m of components. In PCA, you retain m main components, you don't fit them to the data. But in Factor analysis or in CatPCA, for example, you have a prespecified m and fit the m latents to explain the data, so these are fitting techniques. – ttnphns Oct 06 '20 at 23:05
  • Some statistical people seem to think that even looking at the data first should be penalised as warping subsequent inferences. I think that not looking at the data if you can is way more foolish than what they worry about. – Nick Cox Oct 06 '20 at 23:44