For the $n>1290$ case* (or when not using exact=TRUE
), the test statistic used$^\dagger$ is this one:
$$t=r{\sqrt {{\frac {n-2}{1-r^{2}}}}}$$
which should be asymptotically distributed as $t_{n-2}$. [The same statistic is used when testing whether a Pearson correlation differs from 0, but the justification is different]
This statistic is discussed in the Wikipedia page for the Spearman coefficient.
It says there that
A justification for this result relies on a permutation argument.
and the citation for this claim is Vol 2. of Kendall and Stuart (§ 31.19, 31.21)
* The help is slightly inaccurate; it should say $n\leq 1290$ rather than $n<1290$ for the situation when it uses AS 89.
$\dagger$ You can see for yourself that it's doing this right in the code for the Spearman part of cor.test.default
:
pt(r/sqrt((1 - r^2)/(n - 2)), df = n - 2,
That's the handy thing about actually being able to see the code -- you can check for yourself exactly what it's doing