8

The question is about the 2nd data table from the article Symptom improvement in children with autism spectrum disorder following bumetanide administration is associated with decreased GABA/glutamate ratios, specifically first row — total score.

Authors show the result of $37.27±4.09$ (score in some test) in control group vs $34.51±3.35$ in the group who took the medicine.

In the same row it's stated that p-value for that is $3.46 × 10^{−15}$ even though the difference between groups is only about 1 SD.

So the question is: Are results specifically from the first row statistically significant? I have very basic understanding of statistics, but it seems to me that 1 SD is insignificant. Why do they have such a low p-value? Is it an "artifact" because they just multiplied probabilities?

kirilloid
  • 183
  • 4
  • 2
    You have either mixed up standard deviation and standard error (NOT synonymous) or statistical significance and practical significance. My guess is the former. – Dave Jun 18 '21 at 20:42
  • @Dave Ok, I googled SD vs SEM and I think I'm interested in practical significance rather than statistical one. Which should I look at? – kirilloid Jun 18 '21 at 20:44
  • 1
    The analysis used in the paper was more complex than just a test of the mean and SDs specified, as the p-value in the table is for the 'interaction' time x group. Their analysis captures the difference between the before and after differences of the control and treatment groups. – Michael Lew Jun 18 '21 at 20:51

3 Answers3

12

The p 3.46 × 10−15 is not about the difference between groups but about the time × group interaction. The difference between groups has the p = 0.0012, which is in the text in the section Bumetanide improves ASD symptoms, and this p-value is correct given the numbers they provided. You can check it by plunging them into some t-test calculator.

The time × group interaction p-value is calculated using a random effect model and a permutation test, so it is not possible to just plug the numbers from the paper into a calculator and see if they match. Given the supplementary figure 1, which shows the data this test is testing, I wouldn't be surprised that the p-value is extremely small, you can see that the symptom score for pretty much every control stays the same and the symptom score for every treatment case go down by a constant amount.

Although I think that the p-value is possible, I do not think that clinical data can show such a very regular pattern as shown in the supplementary figure, however, I know nothing about the field and barely even skimmed the paper.

edit the figure in question:enter image description here

rep_ho
  • 6,036
  • 1
  • 22
  • 44
  • 3
    I agree that the data look much more regular than one might usually expect from clinical data. That looks fishy to me. – Michael Lew Jun 18 '21 at 21:05
  • @MichaelLew yeah, that's basically a pediatrician looking at a baby and quantifying how autistic it is. I don't know the scale, but I wouldn't expect it to be nearly as reliable. But who knows, maybe the symptoms are pretty constant. If it's possible should be easy to evaluate by people in the field – rep_ho Jun 18 '21 at 21:12
  • I quickly read about how they got data for "S3" part and this the technique seems pretty empirical and subjective, that's why I was more interested in CARS total score, which seems to be more or less standardized. OTOH even if it's a subjective evaluation, 40 patients in a blind trial seems like a good result. – kirilloid Jun 18 '21 at 21:40
  • After all, most psychiatric disorder evaluation is like "looking at the patiaent and evaluating using expert opinion" rather than objective measure certain protein/hormone in blood. Which might be not very objective because of variation among different people. – kirilloid Jun 18 '21 at 21:44
  • 3
    @kirilloid the problem for me is not objectivity of the measure, but that I don't expect a pediatrician to evalute someone to be 37points autistic, and then evaluate them to be also exactly 37 points in 3 months. I would expect that human evaluation is noisier than that. Also, I wouldn't expect a medication to have exactly the same effect for everybody. It can happen, I don't know these measures, but I wouldn't expect it – rep_ho Jun 19 '21 at 16:05
4

Remember the definition of the p value:

This is the probability, under the null hypothesis, of sampling a test statistic at least as extreme as that which was observed

The null hypothesis is not stated. In such cases it is typically one of "no difference".

If this null hypothesis is true, we can still observe a difference in observed means of 1 SD. This probability is small for small sample sizes... smaller for larger sample sizes... smaller yet for really large sample sizes.

Bottom line: for a given effect size (your difference in means of about 1 SD would correspond to an effect size of $d=1.0$, which is sometimes indeed called "large"), we can make the p value arbitrarily small by increasing the sample size.

Whether a difference of 1 SD is clinically significant is another matter entirely.

EDIT: you add that there are about 40 participants in each group. We can do a rough sanity check of the p value by simulation.

Specifically, we simulate 40 "participants" in each group, with normally distributed outcomes. Per the definition of the p value, we assume that the null hypothesis is true, so we use the same mean $0$ for both groups. For simplicity, we also use the same variance $1$.

We then take means within each simulated group, and record whether the mean of the first group exceeds the mean of the second group by at least $1$ SD (for a one-sided test).

We repeat this simulation many times, and check how often we have a positive result.

Below is R code repeating the simulation $1,000,000$ times.

n_sim <- 1e6
hits <- replicate(n_sim,return(mean(rnorm(40))-mean(rnorm(40))>1))
sum(hits)

When I run this, I get $3$ hits, which corresponds to $p=3\times 10^{-6}$. It's not the exact same number as the p value reported in that paper, but it's in the same ballpark of "exceedingly tiny". (Note that there is little "real" difference between p values this tiny.) In particular, it's not like our simulation yielded $p=0.2$, which would indeed indicate that something is amiss.

So even without digging deeply into potential issues of that paper, the p value is not completely unrealistic.

Stephan Kolassa
  • 95,027
  • 13
  • 197
  • 357
  • well, sample size is stated there: about 80 patients, roughly 50/50 in main and control group. Is that a large sample size? – kirilloid Jun 18 '21 at 20:38
  • you are right, but I think OP is asking about the specific numbers from a specific study. so is it possible authors made a mistake, given the reported effect size and n – rep_ho Jun 18 '21 at 20:41
3

The p-value is determined by the standard error (SEM) rather than the standard deviation (SD), so a large enough sample size can lead to a small p-value even where the effect size is small relative to SD.

Another way (that I initially neglected) for a low p-value when the effect is small relative to the SDs is to have done a within-subject test (e.g. paired t-test). In that case the variance of the effect can be small relative to the overall variance of the data when there is a strong within subject before/after correlation.

You might be correct in your assertion that a difference of about 1 SD is functionally (or biologically, or scientifically) trivial ("insignificant"), but the statistics never tell you directly about that kind of significance as it depends on things that are not in the data.

Michael Lew
  • 10,995
  • 2
  • 29
  • 47
  • So roughly speaking they have 1 SEM = SD / sqrt(40), which is about 6 times less, which makes it like 6 sigma, that's why so low p-value, right? – kirilloid Jun 18 '21 at 20:41