3

I have been working with simulations, among them scheduling simulations. Each simulation chooses the number of steps, the arrival rate of new jobs and the duration of each step for each job. Thus, each simulation is an instance of a problem.

I would like to compare various algorithms for scheduling these instances, so I measure some global metric for each algorithm for each simulation. I end up with data like this

sim      alg1      alg2      alg3     alg4

1         5.4       3.5       6.7      5.6
2        10.5       6.8       9.6      9.8
3         8.9      13.4      11.7     12.3
4         8.0       3.4       2.5      6.5
5         9.4      10.3      12.8      9.5

and so on. I usually have less than 10 different algorithms and hundreds of simulations.

I have been using Tukey HSD to analyse the data but I know I am losing power because Tukey does not know that each entry in the different groups are paired.

Is there a multiple comparison procedure to analyze paired data?

cardinal
  • 24,973
  • 8
  • 94
  • 128
Jacques Wainer
  • 5,032
  • 1
  • 20
  • 32
  • 1
    You may be interested in reading through [this question](http://stats.stackexchange.com/questions/575/post-hocs-for-within-subjects-tests) & the answers there. – gung - Reinstate Monica Jul 28 '12 at 16:26
  • You may also be interested in the following article (which is often cited when comparing classifiers performance on the same dataset): Hothorn T, Leisch F, Zeileis A, Hornik K: [The Design and Analysis of Benchmark Experiments](http://www.stat.uni-muenchen.de/~leisch/papers/Hothorn+Leisch+Zeileis-2005.pdf). J Comput Graphical Stat 2005, 14(3):675–699. – chl Jul 28 '12 at 20:12

0 Answers0