Thinking about this question, I came across Bartholemew et al (2011), which lists the following assumptions of the linear factor model, assuming $p$ observed variables:
iii) $e_{1}, e_{2}, ..., e_{p}$ are uncorrelated with each other
v) the $f$s are uncorrelated with the $e$s
They write on p181 that
Assumptions (iii) and (v) imply that the correlations among the $x$s are wholly explained by the factors.
However, as far as I can tell they don't elaborate on this. Why are these conditions sufficient for the factors to wholly explain the observed variables? Are those conditions necessary for the factors to wholly explain the observed variables?
Bartholomew, D. J., Steele, F., Galbraith, J., & Moustaki, I. (2008). Analysis of multivariate social science data. CRC press.