How are degrees of freedom related to the alpha level criterion?

Question

I think I grasped what degrees of freedom are. In Fox (2008) Applied Regression Analysis and the GLM (2nd. edition) chapter ten, on page 225 the geometrical representation of degrees of freedom is explained in a lot of detail:

Degrees of freedom associated with a sum of squares represent the dimension of the subspace to which the corresponding vector is confined.

I am currently reading Using Multivariate Statistics by Barbara G. Tabachnick and Linda S. Fidell. (6th. edition) In chapter 2.6, page 81 they discuss degrees of freedom in the context of Specific Comparisons.

In analysis, degrees of freedom are best thought of as a nonrenewable ressource. They are analyzed once with conventional alpha levels, but further analysis require very stringent alpha levels.

They then lament that researches often "spend" degrees of freedom on uninteresting ANOVAS while planned comparisons are neglected. I added the specific paragraph in question:

So. I thought that I understand what degrees of freedom are. I also thought that I know that you have to limit the number of statistical tests/comparisons in your analysis (or adjust your alpha level) to avoid a rejection of H0 when it is actually "true". However, I fail to see how degrees of freedom and multiple comparison testings are linked.

• In what way do we "spend" a degree of freedom for each test?
• How are they nonrenewable ressources?
• How are degrees of freedom relevant for your comparisons in the way it is suggested by Tabachnick and Fidell?

Answer 1

My cursory reading of this section gives me the impression that there is some critical conflation of "degrees of freedom" with "multiplicity".

These are only similar insofar as ignoring degrees of freedom corrections as well as ignoring multiple comparisons leads to anti-conservative inference: you are more likely to say that something is significant when it is not. That is a weak connection, at best, because there are virtually infinite settings in which anti-conservative inference can be achieved through poor statistical practice, so each principle should be discussed in its own right.

Degrees of freedom are corrections made to the null sampling distributions imposed on inferential statistics for a single comparison. These corrections result from estimating other parameters, but the actual estimate is inconsequential. For instance, a T-test makes a comparison of mean differences between two groups. The actual mean difference is a useful summary measure, however, without knowledge of the standard error, it is contextually useless to know the mean difference. Thus, we base our inference on the sampling distribution of the Wald statistic: $T = \hat{d} / se(\hat{d})$. Only this standardized difference allows us to use the same methods to compare concentrations of lead in ground water sources (hopefully very small) and the molecular weights of stars (hopefully very large) with similar methods. This Wald statistic requires finite sample correction using the degrees of freedom. I should also add that, to the best of my knowledge, degrees of freedom are never "analyzed" but rather apply to specific analyses, as with the T-test example.

Multiplicity has the concept of spending your alpha level. The authors seem to suggest, incorrectly, that one may apply inference at the usual 0.05 level and that subsequent testing can be performed but at a more stringent level. Technically, to preserve the 0.05 family-wise-error-rate, all subsequent testing must be performed at the 0 level if one inferential comparison has already been made at the 0.05 level. Bonferroni provides a mean of using a similar threshold evenly for any $k$ comparisons ($k > 2$), but you need not use the same threshold for all comparisons. In clinical trials, a "first glance" at data may use a very conservative alpha spending since the optimism is low that sufficient information growth has occurred. Subsequent comparisons may be more "targeted" in using a larger proportion of the total alpha level.

Estimation and inference are fundamentally different things. It's utterly important to be careful and considerate about why and when a p-value should be presented, if at all, in the results. Corrections for degrees of freedom ensure that even one p-value is correct. But if more than one hypothesis is presented, adjustment for multiple comparisons is not achieved by any correction to the degrees of freedom, but rather using different significance thresholds so that the family wise error rate or false discovery rate is conserved.

Answer 2

They're doing what it sounded like -- multiple testing; I had hoped there was something more that would make it clear what they were getting at (which is why I asked for context).

The obvious cost with multiple testing is in overall type I error rate. It's not clear that degrees of freedom would be used up if they're all planned tests -- at least in the direct sense that there's no additional loss of df in the distributions involved in those tests.

Data-driven sequences of tests that lead to different models would use "researcher degrees of freedom", and in that situation there's definitely an issue with d.f. that impacts the tests and standard errors and so on. I don't think that's the issue here.

There's at least one other sense in which we might talk about using up degrees of freedom, in that as you do more comparisons, they're no longer independent of each other -- once you have $n$ "independent" pieces of information, there aren't any more. For example, if you determine that $\hat{\mu}_2-\hat{\mu}_1=3.7$ and you determine that $\hat{\mu}_3-\hat{\mu}_1=5.2$ then $\hat{\mu}_3-\hat{\mu}_2=5.2-3.7$ ... there's no "room" for that to be anything else. In that particular sense, at least, there's fewer degrees of freedom for things to be different from what you might expect given what you already know. It's possible that's what they're getting at, but this is not necessarily a problem for your inference, only for the number of actually different things you can say about your data.