0

Can the Welch t-test be used with the median and MAD instead of mean and variance? I think outliers are causing problems and the median places less weight on extreme outliers.

I'm using this to test if one piece of code is faster than another. I think some of the outliers don't represent a true measurement but some kind of pausing of the process.

user842807
  • 3
  • 1
  • 1
    No. Even in principle the MAD isn't an estimator of the SD. (And on dimensional grounds alone it isn't an estimator of variance for measured variables.) And the reasons don't stop there. There'll be a better answer but it depends on your telling us more about your data. If the data are execution times then working on a logarithmic or even reciprocal scale may make sense. Can you show your data or a plot of them? – Nick Cox Dec 10 '18 at 11:46
  • This is a for a general purpose algorithm to determine if one piece of code runs faster than another. There is no data available. – user842807 Dec 10 '18 at 13:54
  • You presumably looked at data at least loosely to know that outliers are something to worry about. Note that the point about reciprocals of times is that they are in essence speeds, e.g. twice as long means half the speed. – Nick Cox Dec 10 '18 at 14:24
  • If there is no data then there is nothing to test – Peter Flom Dec 10 '18 at 18:01
  • It's in principle possible to construct a test if you make some assumptions (e.g. if you assume that your data are drawn from normal distributions, you should be able to derive an approximate test in this fashion; under specific types of moderate non-normality it might continue to have reasonable properties. A similar question has been asked before, that you might find helpful: [T-test using only summary data in a box plot](https://stats.stackexchange.com/questions/170030/t-test-using-only-summary-data-in-a-box-plot). – Glen_b Dec 11 '18 at 04:17
  • However, if full data are available there may well be better choices, depending on your specific hypotheses. – Glen_b Dec 11 '18 at 04:22

1 Answers1

0

The Welch t-test is a test of the difference in means. If you wish to test the difference in medians, you can use Mood's median test or the Wilcoxon-Mann-Whitney U test.

Another possibility is to use quantile regression and treat "code piece" as a categorical independent variable. This lets you test not only the median but any quantile.

I would also try to figure out exactly what is going on with the outliers. Effort spent figuring that out will be rewarded with better data, which always helps.

Peter Flom
  • 94,055
  • 35
  • 143
  • 276
  • Thanks. This is for a general software algorithm which runs each piece of code in turn until it can to some confidence say one is faster than the other or they are the same. Would you say Wilcoxon-Mann-Whitney U test would be a good fit for this? This iterative stopping criteria is not often seen in statistical testing. – user842807 Dec 10 '18 at 14:27
  • 1
    On the contrary, there is a large statistical literature on monitoring a process under headings like "quality control" or "process control". It's just that I have no detailed knowledge of what's closest to your problem, but those are search terms. – Nick Cox Dec 10 '18 at 16:46
  • Many thanks for this. I've implemented the Mann-Whitney test. The last issue I have is when I was doing the Welch test repeatedly when taking measurements I needed to stop if they where still equal after some point. I did this using the difference and standard error i.e. given confidence they don't differ by 0.5%. In the case of the median I'm not sure if I should do something similar or look at soem sort of convergence test. – user842807 Jan 16 '19 at 16:53