0

I am trying to apply the Dietterich 5x2 cv t-test to test whether one statistical model is better than another on a particular data set.

In order to apply this test, one must assume that under the null hypothesis, the Dietterich statistic follows a t-distribution.

Is there any way to validate this assumption? Is it possible to somehow sample from the null distribution in order to see whether it follows a t-distribution? Can this be done with a permutation test?

rhombidodecahedron
  • 2,322
  • 3
  • 23
  • 37
  • There is no way to validate an assumption, that's why it's called an assumption. At best, you can theoretically substantiate it. Or you can try and seek evidence *against* the assumption and conclude you found none. [But that isn't very compelling evidence *for* the assumption, is it?](https://stats.stackexchange.com/q/2492/176202) – Frans Rodenburg Nov 01 '19 at 00:09
  • Maybe a start toward a test based on some evidence would be to show the test you have in mind and and the kind of data you're using. Then you might speculate how to sample from that distribution and maybe use a goodness-of-fit test to see how well results match the Student t distribution with appropriate degrees of freedom. – BruceET Nov 01 '19 at 00:30
  • 2
    https://twitter.com/tdietterich/status/955280111481208834 – Glen_b Nov 01 '19 at 00:54
  • @Glen_b cool! Thanks for posting this. – NULL Nov 08 '19 at 01:35

1 Answers1

1

You have to distinguish between the basic assumptions of the experiment, and the hypotheses. Typically, you check the basic assumptions first (e.g. checking, if the measurements follow a normal distribution). If these assumptions hold true, then you KNOW, how the distribution of the test statistics will be under the null hypothesis (let us call it "null distribution"), you don't have to check it before performing the test.

In fact, typically you don't want the test statistics to follow the null distribution, but on the contrary, to behave in such an improbable manner, that you conclude, it was not really sampled from the null distribution of the test statistics. This would then be a rejection of the null hypothesis.

ghlavin
  • 457
  • 2
  • 6