2

A large amount of computing resources is required to compute exact probabilities for the Kruskal–Wallis test. Existing software only provides exact probabilities for sample sizes less than about 30 participants. These software programs rely on asymptotic approximation for larger sample sizes.

Exact probability values for larger sample sizes are available. Spurrier (2003) published exact probability tables for samples as large as 45 participants.[7] Meyer and Seaman (2006) produced exact probability distributions for samples as large as 105 participants.[8]

R949
  • 31
  • 5
  • Please provide more details about your data, the random-effects model on which your forest plot is based, and the function calls that you used. The results of a random-effects model would be expected to differ from those of an independent-samples t-test (which might not even be appropriate, depending on the nature of the data), but it's hard to say just what's going on here when all we can see is the results rather than the inputs. – EdM Apr 18 '20 at 16:04
  • So, as you are using the Cochrane RevMan software, is this is a meta-analysis of multiple studies? Or are you using RevMan on a single data set of your own? Are there other covariates (age, sex, some measure of health status, etc) included in the model besides presence/absence of tinnitus? If so, are any of those covariates themselves associated with tinnitus or with MPV? – EdM Apr 19 '20 at 17:51
  • I've provided an answer. Note that if you have provided information on mean values, standard deviations, and numbers of cases for each of the studies then you have provided the information needed for the software to calculate confidence intervals, so lack of explicit input of confidence intervals is not what's going on here. – EdM Apr 21 '20 at 17:59
  • Your recent edit to this question has made both the comments on it and my answer impossible to understand for a new visitor to this page. Please restore the original question. If your addition was intended to be an additional answer to this question, please provide what you just wrote as an answer. (You may answer your own question on this site if you wish.) If you have a new question, please use the "Ask Question" button near the top of the page to do so. – EdM Jul 10 '20 at 12:58

2 Answers2

2

This discrepancy seems to come from errors in the Ozbay et al paper.

According to the Materials and Methods, the error estimates in Table 1 are standard deviations. To run the independent-samples t-test, however, you also need to take into account the number of samples; the test depends on the standard errors of the mean values, which decrease with the square root of the numbers of observations. The authors report 107 tinnitus patients and 107 matched controls.

If the error terms in Table 1 are standard deviations, as stated in their Materials and Methods, then the p-value reported by Ozbay et al is perhaps consistent with their having used the standard deviations rather than the standard errors of the means in their t-test calculations. If you take into account the 107 cases in each group, however, the standard errors would be only about 1/10 of the standard deviations (0.11 and 0.13, respectively), so that the difference of 1.59 in MPV between the 2 groups would be highly significant.

One might argue that Ozbay et al used instead a Kruskal-Wallis test,* based on the data failing to fit a normal distribution. With 107 in each group, however, the raw-observation normality assumption for the t-test is probably irrelevant. Such tests for differences between mean values depend on normality of the mean-value estimates. Thanks to the central limit theorem, those mean-value estimates with such large sample sizes are likely to have distributions close to normal. Lumley et al go through these issues in some detail, and point out the limitations and hidden assumptions in use of non-parametric tests instead. At least for variables that showed similar standard deviations between tinnitus and control groups, t-tests should give reliable results for data reported by Ozbay et al.

I can't speak to the extra heterogeneity introduced by including that paper in your meta-analysis. A quick look at it suggests that it was not the most carefully controlled study. In addition to the above apparent error in performing statistical tests, some of the underlying data are of questionable quality. For example, although the platelet counts (in units of 10^3 per mm^3) are similar between the 2 groups (225 tinnitus, 260 control) their reported standard deviations differ by more than a factor of 10 (10 and 110, respectively).

Part of doing a thorough meta-analysis is getting as much information as possible about the details of the underlying studies. In this case, I think that if you look into this paper in more detail you might be justified in discounting it based on its deficiencies.

*When comparing 2 groups as in Ozbay et al, this is the Mann-Whitney or Wilcoxon test.

EdM
  • 57,766
  • 7
  • 66
  • 187
  • @R949 now that you have pointed out the specific paper in question, the answer is pretty obvious. I have completely rewritten my answer accordingly. – EdM Apr 22 '20 at 15:12
  • @R949 I can't tell exactly what they did but there seems to be some type of large error in how they did their calculations. Look at the glucose values in Table 1: 46 mmol/l higher in tinnitus than in control, with standard deviations of about 15 corresponding to group mean values over 3 _standard deviations_ apart. So that should have been significant (at about p = 0.02) even if those were _standard errors_ rather than standard deviations. Yet they claim a p value of 0.32! Triglycerides twice as high in tinnitus as control, highly significant if those are SDs with 107 cases in each group. – EdM Apr 22 '20 at 20:21
  • @R949 all you should have to do is apply the [t-test with equal sample sizes](https://en.wikipedia.org/wiki/Student%27s_t-test#Equal_sample_sizes,_not_variance) to the mean and standard deviation values reported in Table 1 of Ozbay et al and show that the results disagree with the reported p values. That means either their standard deviations are incorrect or their statistical testing was incorrect. Either way the study is unreliable. I'll add a bit to the answer to deal with the t-test versus Kruskal-Wallis; t-tests should be fine with sample sizes this large. – EdM Apr 23 '20 at 15:53
  • @R949 I think that your formula for _t_ is missing a value of $\sqrt 2$ in the denominator, but that will still be highly significant. So either the Ozbay values aren't SDs or their statistical calculations are in error. KW and MW tests give [the same p-values with 2 groups](http://www.biostathandbook.com/kruskalwallis.html), so that's just terminology. Many people think you should do a SW test first and then choose _t_ versus MW on that basis, but that's not a good idea. See [this page](https://stats.stackexchange.com/q/2492/28500) for extensive discussion of normality tests like SW. – EdM Apr 23 '20 at 17:58
1

I would like to second EdM's answer. Particularly, I would like to add some thoughts on their last comment about the paper's deficiencies.

As EdM already stated, it is easy to confirm that the provided data and p-values do not make sense in the way that they are reported by Ozbay et al. Maybe the data is actually extremely far from normality, and then a Mann-Whitney-Test, which does not care about the means, could provide the p-values reported. But this is unlikely, to say the least. Or maybe they actually run a more complex model, e.g. a multivariable regression, and all these p-values are adjusted for covariables. But in their Methods, they say otherwise.

Ozbay et al. do not report any power analysis, and therefore any conclusions like "the parameters were similar" based on p-values are invalid. We need at least a confidence interval and enough statistical power to conclude something about similarity.

The description of the statistical analysis is neither insightful nor does it make sense. A Kruskal-Wallis test for two groups is actually a Mann-Whitney test (or Wilcoxon test). Reporting the mean when doing a Mann-Whitney test is not helpful. They did not adjust for any potential confounders.

I am definitely no expert in this field, but I would question whether matching just based on age and sex is appropriate. Patients with Tinnitus have higher stress levels, and the association between stress level and neutrophil to lymphocyte ratio is well known. It is not surprising that Tinnitus patients have higher NLR than healthy controls. (well, actually in their figure, the two groups look quite similar. I would not even trust that p<0.05)

You say it is a highly cited paper, but my naive google scholar request revealed just 17 citations in the past 5 years, and most of them seem to be from Turkey, too. This is not neccesarily a bad sign, of course, but I am always a bit sceptical, if a study is cited mainly by people from the same regions. Sometimes this is an indicator for self-citing circles instead of good quality of the research.

However, the best way to get clarity is to ask the authors for the raw data. Maybe there is a good explanation for all of this. Maybe it were just simple mistakes. And if they do not provide you with the raw data, well, then you might put two and two together and maybe not fully trust the results.

LuckyPal
  • 1,512
  • 7
  • 15