I have read that permutation test is preferred to use when sample size is relatively small and when ANOVA fails to capture the difference due to this fact. Is this correct? Could someone please expand this a little bit more. What other approaches I can try when ANOVA fails to find significance due to this reason.
Thank you.
Randomization techniques might be more powerful than ANOVA in some circumstances, but not necessarily. See "Comparative Power Of The Anova, Randomization Anova, And Kruskal-Wallis Test" . Randomization techniques are usually more powerful than non-parametric rank transformation tests like Kruskal Wallis. See Adams and Anthony "Using randomization techniques to analyse behavioural data". I think their main strength though is that they don't invoke assumptions related to normality; non-parametric methods in general (bootstrap, jacknife, randomization) can handle data from any distribution. It has been argued that for many study designs randomization tests offer conceptual advantages over hypothesis testings because available items or subjects are generally randomized to treatments rather than truly randomly selected from a population then randomly assigned to treatment; see Ludbrook & Hugh Dudley "Why Permutation Tests are Superior to t and F Tests in Biomedical Research". There are lower limits to sample size and I;ve seen writers lament that people will take advice that non-parametric methods "work well with small sample sizes" and use them with samples sizes < 5.
You should probably also assess to what extent you are violating ANOVA assumptions; perhaps a generalized linear model with a binomial (logistic regression), poisson or negative binomial distribution would be more appropriate if your response variable is, for example, a count.
I have read that permutation test is preferred to use when sample size is relatively small and when ANOVA fails to capture the difference due to this fact.
I see several difficulties with this statement, of which I'll mention a few:
how can you tell if it's "due to this fact" rather than due to something else?
if sample sizes are very small (particularly with ties), permutation tests only have a small number of possible significance levels, and the smallest possible significance level may be substantially larger than the desired level; this problem isn't shared by ANOVA.
if the ANOVA assumptions are suitable, a permutation test won't necessarily give you any additional power.
Permutation tests may be particularly useful when the actual distribution is non-normal** - if it's heavy-tailed for example - better power might sometimes be had by a rank-based permutation test. This benefit might be quite noticeable when sample sizes are moderately small (but not too small, perhaps).
** but not too heavily non-normal, because then the test statistic is not robust. It's still a valid test (i.e. it's level-robust), but if the distribution is so non-normal that the sample mean is a very inefficient estimate of the population mean, then the test will be low-power. If the distribution is fairly close to normal and the sample sizes aren't tiny, then it's a very good choice.
But there are also robust methods, GLMs, bootstrapping to name some alternative possibilities.
