3

Consider a t-test of means. One formula for computing the p-value assumes equal variances. Another formula assumes unequal variances. With small sample sizes the tests can give quite different results and one can examine the variances to see which assumption is more prudent. An alternative would be to compute the p-values using both formulas and then compute a weighted sum of these where the weight is determined by the appropriateness of the assumption.

I have not seen it mentioned in any stats book. Is it an intrinsically bad idea? If not, is there a standard approach to this?

Shayan Shafiq
  • 633
  • 6
  • 17
Tim
  • 3,255
  • 14
  • 24
  • This approach sounds slightly Bayesian to me... My take: this is not really covered in classical Null Hypothesis Significance Testing, where your decisions are binary (accept vs reject the null). There is really no concept of evidence (or as you call it, appropriateness) in classical NHST. For this, we need to use Bayesian methods. – Stephan Kolassa Nov 14 '12 at 07:54
  • 2
    A very closely-related idea is given in the adaptive estimation of [Mukherjee and Chatterjee 2008](http://onlinelibrary.wiley.com/doi/10.1111/j.1541-0420.2007.00953.x/abstract), who give a point estimate that moves between a robust/less-robust version, depending on how the assumptions of the less-robust version are satisfied. They give standard errors too, so you could turn it into a test with the general characteristics you seek. As far as I know, the idea has not yet been generalized. – guest Nov 14 '12 at 17:39

3 Answers3

1

That is a clever idea but any idea that there is a standard approach or even ever will be is misplaced.

First of all if its a randomized study and you follow Fisher's argument (for what is called Fishers exact NULL) all parameters in both groups are exactly equal so one is testing for a possible difference in means given exactly the same variances. Many statisticians and some Bayesians (e.g. Don Rubin) accept this as a sensible answer or approach.

Otherwise you have a composite NULL for which a fully satisfactory test needs to have the same type one error rate uniformly for all parameter values in the NULL (i.e. a similar test) and usually that is not possible. In particular, here it is known as the Fisher-Beren's problem which I believe remains an open question - is there a fully satisfactory test of a difference in means with arbitrary variances? There are lots of approximate methods that in differing ways come close, and I would bet one or more is based on a weighted sum of p_values.

In the Bayesian approach you run into difficult and confusing prior specifications :-(

In general, statistics is "all fully sorted out" only? when there is just one unknown parameter.

And the idea that you can include all uncertainties in say the construction of a test (even if you relax the requirement for similarity) or a prior is also I believe miss-placed. Alternatively, this paper argues that you should not expect or want to insure yourself against all uncertainties S Stigler. The changing history of robustness

Applying statistics means never being certain about your uncertainty assessments ;-)

phaneron
  • 1,252
  • 7
  • 7
1

Several points. There really aren't two t-tests (one for equal variances and one for unequal). If the two distributions are normal with unknown means and equal (but unknown) variance, the distribution of the test statistic has a t distribution under the null hypothesis of equal variances. The key point here is that the unknown variance does not figure in the distribution of the test statistic.

With 2 normals, unknown means and unknown but different variances, the obvious test statistic has a distribution that depends upon the ratio of the variances. This is the so-called Behrens-Fisher problem. The distribution of the test statistic under the null hypothesis of equal means depends on parameter values that are not known; so an exact rejection region cannot be constructed.

Walsh's test is basically a fudge. The test statistic makes intuitive sense; and one adjusts the degrees of freedom so that the result more or less follows a t distribution under the null. Which apparently, it does. Interestingly, R's t.test defaults to Walsh.

In any case, what you suggest would be a fudge applied to a fudge, and I am not sure what rationale in the theory of hypothesis testing would justify it. (Nice try, though).

The failure of Behrens-Fisher illuminates an important truth: when the variances are wildly different, comparing means probably does not make a huge amount of sense. If they are fairly close, Walsh will work.

When two distributions have different variances, you basically need to ask yourself what sort of differences between the underlying processes are of interest to you.

Reasonable approaches include:

  • A non-parametric test.
  • A variance stabilizing transformation
  • Possibly a bootstrap estimate of some kind.
Placidia
  • 13,501
  • 6
  • 33
  • 62
0

If you have multiple P-values (or equivalently, Z-stats), in order to combine them, you can either use Fisher's or Stouffer's formula.

Fisher's test results in a statistic $S = \sum_{i=1}^{N}-\log_{10}(P_i)$, which, under H0 is to $~\chi^2_{N-1}$. Stouffer's test results in a Z-statistic that is $\sum_{i=1}^NZ_i/\sqrt N$.

rezakhorshidi
  • 216
  • 1
  • 3
  • Aren't these formulae for creating a combined test statistics for independent tests? (In the situation I've described the tests are not independent as they involve the same data.) If not, could you please provide a reference, proof or some other form of evidence. – Tim Dec 09 '12 at 23:13