2

I'm working on a paper involving multiple comparisons that have very small sample sizes (and no chance of increasing them). When I calculate p-values, they usually are not less than the usual .05-level, causing me to not reject the null hypothesis. However, I have very low power due to the small samples, thus can easily be committing Type II errors. Also, the effect size measures (e.g., Cramer's V) that I feel are meaningful have 95% CIs that are so large as to be irrelevant. Does anyone have any idea what I can report? Can I say anything about my comparisons?

stevebyers2000
  • 121
  • 3
  • 4
    "We failed to find evidence that these quantities are different at the $\alpha$ level." – Alexis Jun 09 '20 at 21:30
  • StatQuest has a video on YouTube about what to do when your results are just barely above 0.05. Perhaps check it out. – Dave Jun 09 '20 at 21:31
  • 1
    @Alexis I'd qualify that even further by describing the low power of the study - failing to find a difference can be good evidence that there is no difference if you have enough samples to have a high-powered study. But in a low-powered study, failing to find evidence of a difference tells you very little (even if there was a difference, you wouldn't be likely to see it). Basically, low-power comparisons are doomed from the start unless you have huge effect sizes. – Nuclear Hoagie Jun 09 '20 at 21:40
  • Running experiments, without first exploring whether it is feasible to have an experimental design with decent power, is a waste of time and money (and if human/animal subjects are involved, the possibility of actually doing harm). It may seem harsh, but it may be best all around to have _no report_ of this experiment. If there is, maybe it can center of whatever was learned about the variability of the resulting data. That much at least may be useful for proper planning of future experiments using the same measurement methods. – BruceET Jun 09 '20 at 22:30
  • @NuclearWang I put power directly into the conclusions by performing both a test for difference, and a test for equivalence (i.e. [relevance tests](https://stats.stackexchange.com/questions/95667/)). – Alexis Jun 09 '20 at 23:11
  • @BruceET Do you think publishing the data may be worthwhile for future meta-studies/meta-analyses? (Even if the data are too underpowered to serve any current inferential purpose?) – Alexis Jun 09 '20 at 23:12
  • 2
    @Alexis. Depends on circumstances. OP may want to try again, this time with large enough $n$ to have adequate power, and may have info from current study that would help with that. Also, it may not burnish one's reputation to publish results of an obviously poorly planned study. (If I have the misfortune to make a huge noisy belch at a formal dinner party, I probably won't rush to share that on social media. Not quite an analogy, but I'd think twice about trying to publish nonsignificant results.) – BruceET Jun 09 '20 at 23:48
  • 1
    Thanx for all of your input. Unfortunately,the data I'm analyzing is not an experiment, but a study of a number of skeletons from 4 archaeological sites, hence the small sample sizes and low power. Thus, I had no control over how many individuals were recovered/recoverable. That said, can anyone think of anything I can report on these data? I hate not to publish as these skeletons are from a time period where little is known about the biology of the people, but I don't want to say the people were biologically similar from the 4 sites when I cannot support it with statistics. – stevebyers2000 Jun 10 '20 at 00:22
  • With such small numbers, simply describe each individual's measures and qualities. – Alexis Jun 10 '20 at 16:31

0 Answers0