Determine which performance intervention is best?

Question

Suppose I have numerical data describing the total process time for a given software simulation. This data is broken up into 5 groups (Base, AD1, AD2, AD3, AD4) each detailing a different performance intervention with approximately the same number of observations for each group.

My goal is to determine if the performance interventions result in significantly different alive times than the base case and to determine which intervention is "best". "Best" being defined as the least amount of process time.

To clarify, my data is comprised of all the "regression-tests" in our code framework. So at this point I am looking at a high-level what the interventions do to overall process time but eventually will create sub-categories within each intervention to determine inter-group effect on process time.

My data has some extreme outliers as can be seen from this graphic:

My hypothesis is as follows:

$$ H_{0}: \mu_{\text{base}} = \mu_{\text{AD1}} = \mu_{\text{AD2}} = \mu_{\text{AD3}} = \mu_{\text{AD4}} $$

$$ H_{A}: \text{Not all means equal} $$

I am unsure what my hypothesis would be in determining the best "metric". I am also unsure if using the mean is appropriate in this circumstance given the outliers in my data.

My idea is to use some form of ANOVA or Krukall Wallis test and then maybe a Tukey Test to determine which one is best? I am open to Bayesian or Frequentist approaches to this. I might be over thinking this as well.

EDIT: I have made some good progress using Martin's suggestions and have been following along a tutorial found at:

https://www.andrewheiss.com/blog/2019/01/29/diff-means-half-dozen-ways/

and

http://www.flutterbys.com.au/stats/tut/tut7.4b.html

Through this I've built a model that kind of approximates the density of my alive times but unsure if I can make it better or where to go from here:

fit1 <-brm(alive_time ~ intv, data = dat, iter = 10000, warmup = 1000,
    chains = 3, thin = 2, refresh = 0, cores = 12,
    family = student(link = 'log'))

pp_check(fit1, nsamples = 1e3) + xlim(0, 40000)

At the request of @Ben I've went ahead and plotted the densities using library(ggridges) on a log10 scale. Here is code and resulting plot:

dat %>% 
    ggplot(aes(y = intv, x = log10(alive_time), height = ..density..)) +
    geom_density_ridges2(stat = 'density') +
    labs(x = "Alive Time", y = "Interventions", 
         title = "Standard Ridgeline Plot")

I've also provided my data using dput(dat) in R below:

dat <- structure(list(intv = structure(c(4L, 4L, 4L, 4L, 4L, 4L, 4L, 
4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 
4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 
4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 
4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 
4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 
4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 
4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 
4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 
4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 
4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 
4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 
4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 
4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 5L, 5L, 5L, 5L, 5L, 
5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 
5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 
5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 
5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 
5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 
5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 
5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 
5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 
5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 
5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 
5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 
5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 
5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 3L, 3L, 
3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 
3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 
3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 
3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 
3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 
3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 
3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 
3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 
3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 
3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 
3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 
3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 
3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 
3L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 
2L, 2L, 2L, 2L, 2L, 2L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L), .Label = c("Base", "Ad", "Timestep", "Daniel1", "Shane1"
), class = "factor"), alive_time = c(10477.486474353, 15462.898424906, 
7.665776406, 8.414731227, 9.210761743, 4.017424642, 4.493777922, 
4.918436628, 2.246632132, 2.580258666, 2.836145724, 1.540199249, 
1.721755586, 1.898067896, 1.206603031, 1.444920648, 1.586049433, 
1.223530677, 1.374306366, 1.514164025, 2139.804058848, 2268.37153924, 
2178.079627582, 14390.37095987, 3835.342706029, 22252.500414809, 
20465.873024157, 15316.155927163, 23934.43387019, 28684.205118383, 
14395.69150759, 10962.580961668, 12414.105887542, 18584.658076431, 
19239.442516556, 19098.132572659, 13617.944780817, 22924.620553863, 
8750.700240409, 7584.063561984, 13274.532326964, 15149.933944385, 
25756.285614756, 19176.849027582, 12681.561851204, 26455.587340254, 
4233.929780132, 2343.967515272, 3891.569022885, 3524.562912168, 
2295.689004345, 2728.340292475, 3523.230399729, 2898.215419231, 
932.85045787, 791.070993243, 983.207437087, 444.83004738, 444.33692306, 
613.183019205, 851.386419484, 2662.570880028, 23735.121412053, 
14735.013272352, 4752.198156618, 28081.623057182, 28015.177471506, 
27628.062910486, 14792.113743361, 15638.816858656, 11234.280333881, 
24979.333642328, 24946.090644972, 4351.134506585, 4041.531976453, 
831.871612316, 294.260439763, 184.862397718, 9581.000459489, 
4890.532827258, 2062.08722422, 1946.648436285, 2048.737383734, 
1894.671220269, 2049.824549245, 1874.892934304, 1958.000087582, 
1864.139659147, 2145.544504207, 2122.117251084, 2256.390330179, 
2128.925847905, 945.96989839, 833.88139152, 882.556095446, 817.669087374, 
881.799307963, 814.338140236, 911.836684402, 797.269973129, 856.967394152, 
785.393070854, 816.319586729, 741.970131347, 1405.752754121, 
1292.442324213, 882.920916393, 813.523062245, 852.821786426, 
797.822076345, 763.491906962, 694.07273145, 834.63872812, 752.779319785, 
891.583529842, 801.580374674, 1563.664771053, 1503.632590209, 
1337.548753216, 1219.890817284, 1481.882001271, 1383.941468317, 
1450.762971259, 1314.172618048, 782.438301925, 710.769508183, 
1715.480938156, 1621.877874192, 1729.383936174, 1700.499525546, 
920.002176376, 852.479239639, 1410.626143679, 1420.261666573, 
1167.860063204, 1072.899506234, 1041.777046435, 950.473338333, 
683.850159879, 607.631964906, 721.172564459, 651.380077874, 187.309558421, 
171.39616093, 225.945850631, 206.626979269, 1728.408443902, 1570.131804674, 
1204.724222054, 1123.12347573, 1090.879121324, 1018.472985765, 
1023.159600098, 1086.03069888, 845.186872167, 789.796143947, 
960.961935916, 889.133671775, 11620.846130882, 46439.200037553, 
7259.855787583, 2653.115423044, 3199.239825203, 3440.436345477, 
673.920290883, 2982.450748097, 3315.823293488, 4486.025600435, 
167.051735438, 4512.417969168, 3566.115598728, 3241.520018281, 
5439.228304917, 4530.621770649, 2270.911732484, 5633.447199621, 
5112.40159827, 4935.447188634, 2460.384279123, 4964.3893024, 
5066.702498401, 2439.11978179, 3168.333363353, 1349.129484372, 
3500.312713384, 3918.150521527, 2726.80697369, 3400.288018976, 
2992.987326537, 978.367936371, 7703.767023843, 9190.693880286, 
9468.732323055, 11282.88155395, 891.958311166, 987.139602593, 
3513.805206819, 3528.026260471, 4082.680791945, 3089.133899853, 
3561.988491577, 3739.890216091, 4372.556410666, 13107.744535865, 
9714.380801656, 307.136814211, 14897.446630047, 267.651155027, 
11362.784352757, 14984.23798044, 11191.877833735, 15342.983141659, 
7.828128698, 8.623044825, 9.402143513, 4.112654521, 4.618458667, 
5.013303361, 2.275603921, 2.648975134, 2.87794921, 1.572978061, 
1.77455654, 1.933662805, 1.23745918, 1.504298648, 1.612627601, 
1.253439973, 1.407773352, 1.539620222, 2214.819307841, 2398.545663917, 
2299.12900277, 13281.1519391, 3887.276134524, 21394.55865882, 
20692.443567153, 15567.354417516, 23061.46634962, 27790.784615372, 
14427.931217373, 10952.574267423, 12386.612301246, 18428.919087867, 
19610.967793897, 18850.79790296, 13528.941636007, 22930.097302316, 
8716.669216436, 7588.965039386, 13385.207299096, 15201.187886311, 
24922.304407588, 19328.580950517, 12711.375623171, 26416.665976643, 
14541.598161982, 2342.731684273, 3863.917400016, 3278.050308891, 
2176.345707484, 2789.556594967, 3515.14764991, 2849.670591005, 
988.170042563, 801.335176311, 975.49276559, 446.501566006, 447.048951164, 
637.708437447, 846.770587468, 2508.649063334, 23819.336609558, 
13825.465744509, 4733.791002775, 28190.791238796, 27300.683772099, 
27748.596178692, 14721.736257948, 15322.369640528, 11040.193308981, 
39990.377979793, 24404.385388334, 4350.540240463, 4056.009936785, 
931.116539626, 319.765335272, 223.525010321, 2697.183786116, 
7.984119199, 2073.0474675, 9550.203165969, 4972.404636066, 2319.29631474, 
2021.070429394, 2275.218714612, 2124.529387947, 2154.100666089, 
1940.940076053, 2099.558783077, 1922.261148239, 2086.575112868, 
1889.169712931, 2454.566237798, 2192.448736569, 921.950686796, 
868.941175468, 929.183215374, 844.62921356, 1052.221090634, 964.676508192, 
972.585176828, 927.06502814, 986.370431404, 846.000516324, 848.91101562, 
782.230145341, 1654.979174586, 1395.444206727, 922.785503687, 
870.224164995, 1027.069093636, 869.575080871, 800.241833112, 
763.594327035, 909.588004114, 829.168474781, 1009.333139029, 
903.217640648, 1686.586421196, 1553.570950707, 1806.876927074, 
1506.493537691, 1814.172930367, 1657.788556037, 1795.147647595, 
1632.632665616, 1001.236477721, 956.666780932, 1988.905741483, 
1687.30776166, 2006.917264338, 1710.977749095, 944.444070572, 
865.737528435, 1602.364564823, 1375.797141158, 1324.864043672, 
1237.46497257, 1058.142749297, 950.418634537, 705.339442371, 
648.197734393, 856.468516393, 753.644844145, 193.146793088, 176.876608114, 
243.2907308, 226.483472146, 1801.302295661, 1671.775704224, 1123.592254691, 
1018.185336185, 1015.661905041, 920.406374837, 1054.103594745, 
1117.821032257, 1016.319716921, 947.71715952, 1018.900292434, 
953.394801622, 7410.821667027, 3384.885699843, 3472.247453662, 
4176.430670791, 636.668399206, 3097.944262372, 3152.803280179, 
4615.683510109, 158.730013218, 4235.470707974, 3398.953531263, 
3167.87590064, 5186.787018277, 4447.984331733, 2266.629095619, 
7543.009550159, 5347.469612856, 4629.582045525, 1915.129156228, 
4839.399452395, 4910.039662286, 2403.711364161, 3142.290949401, 
1461.608181538, 3462.281670435, 3871.013786235, 2720.766241738, 
3536.153746303, 2980.297139849, 996.217318177, 7723.439677371, 
9205.63009471, 9829.557310343, 11711.878150449, 922.727845076, 
1000.441968311, 3483.646262623, 3451.435345587, 3966.732580686, 
3089.466852748, 3572.152153, 3769.40785802, 4413.80870026, 13077.689399977, 
9904.679566789, 307.974384182, 12868.184606796, 265.701931691, 
11645.575544979, 14909.856861772, 11481.941179964, 16511.094518736, 
6.355724876, 7.029256677, 7.661710719, 3.328646758, 3.76357336, 
4.088905015, 1.875240383, 2.144813038, 2.356469994, 1.276509639, 
1.443256088, 1.577887777, 0.999972174, 1.258132352, 1.32229044, 
1.014554394, 1.144693429, 1.250646139, 1819.331183662, 1973.204714997, 
2241.829591422, 13571.668422278, 3815.354863454, 21478.599747204, 
20337.568289176, 15218.206957769, 22788.977537733, 27496.692554082, 
14759.939253915, 11020.324349908, 12016.256765476, 18433.396609096, 
19679.896331324, 18703.351836913, 13388.375769699, 22842.379256657, 
8661.979948624, 7765.50187174, 11040.604890491, 12324.42653671, 
24960.857193829, 19085.02325561, 12635.964667845, 26374.220175371, 
4177.988625576, 2329.02354921, 4114.925857035, 3470.389881276, 
2155.973841941, 2706.623991752, 3530.035649796, 2849.919815446, 
939.853172928, 836.935501803, 977.527673498, 453.896154977, 440.09072385, 
606.073291705, 840.236903446, 2520.632528438, 23814.890833368, 
14209.168122833, 4833.791239664, 27763.945063929, 26948.81390054, 
27072.016742267, 14507.132775206, 15266.768641656, 11079.145330692, 
24090.4075872, 24411.349698724, 4297.171179291, 4021.557876917, 
829.933891223, 299.054874119, 200.703598451, 2613.501849335, 
8.2063636, 2029.341459918, 9685.923795909, 4873.010096164, 2028.10133965, 
1834.213433725, 2096.966084336, 1827.424802494, 2026.292667234, 
1807.169694145, 2154.672196204, 1850.272501026, 2067.327058762, 
1878.331686777, 2257.275781779, 2013.144377271, 911.402626999, 
818.80063028, 859.364166903, 798.611057489, 890.80310288, 798.983163258, 
865.74126983, 794.560467193, 871.791071736, 775.32144092, 848.841401229, 
767.576850803, 1516.637728836, 1351.12450758, 881.522522404, 
807.124965971, 824.421898511, 748.185282956, 769.732554569, 708.593405995, 
825.390729975, 742.073176756, 875.532132541, 789.678430308, 1540.223702715, 
1365.121657245, 1409.524503707, 1237.051842982, 1465.958294012, 
1333.533910288, 1628.998928184, 1512.026563742, 970.513907986, 
880.828074816, 1628.627129081, 1476.793427861, 1667.83870288, 
1523.150336547, 1005.54297205, 914.605599319, 1374.878915707, 
1225.263090486, 1200.116130938, 1065.922882089, 1074.935257804, 
983.372293322, 713.507423357, 653.060598564, 743.436312856, 675.088117297, 
188.355701823, 168.644767707, 226.568184331, 205.413182984, 1679.764431886, 
1537.103338095, 983.293671476, 905.747310279, 1075.808216583, 
989.106508957, 1017.762151578, 1070.578732749, 1009.057437336, 
900.449059368, 978.223214616, 897.107128479, 6889.536054983, 
2708.03069678, 3031.437100759, 3560.933318343, 656.379732423, 
2939.150029034, 3229.716465672, 4600.28301051, 171.035837069, 
4424.623051697, 3536.763574135, 3147.11260987, 5296.576268739, 
4419.560451017, 2284.932598553, 5393.891198889, 4980.214202219, 
4532.147314084, 2636.166341455, 4925.928053733, 4922.492636589, 
2470.84062464, 3151.305846093, 1339.746317104, 3443.120865724, 
3861.378318655, 2710.572297575, 3421.031231996, 3045.44437603, 
977.70708094, 7768.828592507, 9133.812637472, 9590.997249674, 
11236.764837554, 883.913459746, 990.606527643, 3421.66280523, 
3591.735482467, 4017.343033315, 3081.21936964, 3543.223608265, 
3692.470168507, 4326.174327684, 12747.697743626, 10102.625037673, 
350.837173632, 15024.643932649, 261.14886504, 11584.525414532, 
13924.736299545, 12857.687272456, 17048.110955675, 6.246834566, 
6.881466688, 7.421381818, 3.290735747, 3.673799017, 3.986093252, 
1.836880951, 2.096229927, 2.291800417, 1.255165968, 1.502052612, 
1.53410418, 0.985414042, 1.181204468, 1.28441096, 1.00113897, 
1.121030269, 1.225056385, 2210.130444413, 2377.642025226, 2256.375947833, 
13489.794505582, 3754.142582694, 21359.28489732, 20616.190794592, 
15957.898044331, 23100.773410943, 27495.127084992, 14212.197542456, 
11013.389693721, 11980.292852242, 18348.721758173, 19287.691571951, 
18729.031879416, 13587.106790038, 23543.20943874, 9026.242565173, 
7582.680182644, 13323.018237341, 15146.220617202, 24654.699328121, 
19196.542443184, 12656.878361669, 26101.823403022, 4144.774302329, 
2426.864544501, 4093.087235011, 3433.989631948, 2176.709850973, 
2791.287290145, 3509.363364461, 2887.728488813, 938.685786866, 
808.452451307, 978.125592565, 456.852526414, 443.706321084, 608.390902791, 
853.033446964, 2413.235253012, 23633.218575305, 14122.9963913, 
4762.085998406, 39970.160116301, 27606.297084149, 27212.392648057, 
14768.077456856, 15241.474300417, 11317.084417127, 24567.185970313, 
24337.610854767, 4328.518219795, 4042.39507137, 803.965582122, 
270.282972987, 168.3359665, 2565.451984753, 7.944202868, 2020.416784042, 
9642.870608299, 4846.97771672, 2032.32574534, 1893.23003522, 
1956.06802972, 1822.093055208, 2021.301509325, 1886.582841852, 
2211.147743323, 1966.258590691, 2136.344391969, 1961.278791282, 
2306.280002658, 2172.7889755, 902.546391746, 825.741462499, 887.723997045, 
804.549397232, 882.402084914, 807.16433189, 880.236010728, 807.606717052, 
835.336453687, 762.202091052, 790.726376406, 714.647061412, 1341.657971345, 
1234.358383205, 903.73827845, 823.957750278, 976.280369611, 883.887712592, 
756.452580964, 692.360533558, 837.671345559, 764.448153533, 857.709211118, 
766.281874923, 1513.754878524, 1388.000615287, 1381.850261077, 
1256.669585298, 1466.377393976, 1345.110011771, 1645.592607403, 
1518.797804613, 888.401852326, 825.262540608, 1673.959155636, 
1507.622793317, 1775.982473917, 1611.813054698, 931.445425579, 
866.355945677, 1355.16176587, 1237.616800637, 1197.913452648, 
1097.634224865, 1001.851062876, 908.587257338, 664.559701168, 
592.474707, 697.80596222, 643.054630344, 183.018618446, 168.162314471, 
223.295369939, 212.446657568, 1724.085304572, 1606.955086898, 
952.48937542, 887.638743056, 1068.245320642, 989.875405855, 986.605130777, 
1053.72521362, 975.819999156, 900.386792357, 970.109869966, 898.89271337, 
10781.208046887, 45037.942550129, 7245.167103019, 2712.741092937, 
3164.030695499, 3691.009562725, 656.99925762, 2957.020256788, 
3504.589259205, 4571.053233048, 166.773860709, 4315.364099159, 
3562.893265958, 3220.426987198, 5425.646997566, 4478.730458262, 
2269.039013389, 5736.9136458, 5147.973979647, 4563.214559371, 
2435.627690299, 4949.432417698, 4874.889736798, 2447.370509174, 
3145.076913027, 1389.694649923, 3462.012459963, 4216.064489881, 
2950.829316152, 3407.14359775, 3208.884502676, 971.630172045, 
7833.251446154, 9276.179681786, 10191.588469976, 11739.8725632, 
873.051443516, 1001.923176224, 3454.355846919, 3425.274178718, 
3940.960628103, 3041.257518475, 3521.269460166, 3750.900953434, 
4439.542579025, 12680.947084854, 9733.258527423, 308.751777629, 
15041.183863368, 184.8452911, 9235.422525239, 11603.313679836, 
9513.435374799, 15661.892928366, 5.935499689, 6.483064133, 7.088646266, 
3.128106885, 3.453503382, 3.812565892, 1.743686801, 1.989766839, 
2.162768883, 1.18796692, 1.34776651, 1.460677952, 0.934184326, 
1.110563835, 1.22109392, 0.942966266, 1.059290854, 1.160741022, 
2249.187326262, 2406.920501236, 2372.969988995, 13885.052641784, 
3899.846831725, 21571.318729866, 20442.69884649, 15596.795756857, 
23269.331212058, 28003.045832026, 14391.00981935, 10991.588138466, 
12146.498955404, 18591.153622519, 19715.864962382, 18984.011787921, 
13495.39325162, 22933.268374506, 8932.297666899, 7905.304217295, 
13835.410007437, 15166.433852151, 25039.396226243, 19498.195086677, 
12877.101291995, 26668.263768903, 4233.761748941, 2327.605818745, 
4209.885290145, 3263.021192413, 1848.678983786, 2740.764688357, 
3522.095678614, 2643.799496697, 1027.961905777, 809.863513611, 
981.40024344, 449.54786886, 468.594754441, 611.794937399, 870.832204039, 
2588.091632841, 24184.621409418, 43768.721896141, 38754.006715268, 
13934.565982487, 4778.265090891, 35282.880906662, 25002.217878491, 
22070.171773063, 18451.139765434, 28212.058982346, 28104.822986025, 
27885.255536277, 14436.17849318, 15457.703118251, 11159.483503485, 
24741.803307462, 24782.255355859, 4396.614409775, 4112.328457279, 
1180.872397438, 403.233318452, 311.47541438, 2856.277573194, 
8.58561842, 2185.718471559, 9579.035204336, 4975.550069648, 2568.578937896, 
2089.908295499, 2637.423773557, 2284.730773573, 2393.81462317, 
1968.770922924, 2368.023842406, 2004.187214709, 2387.817824755, 
1937.455075614, 2721.312740164, 2364.053940896, 1144.36029701, 
981.097717693, 1121.308899158, 957.768545721, 1194.467741917, 
979.703759746, 1058.92901845, 955.306783423, 1122.185597996, 
934.796479528, 1087.446368377, 947.182890055, 1720.280668699, 
1396.44131266, 1264.822501925, 1066.467340058, 1129.872654755, 
978.591349084, 1024.847337793, 893.95641702, 1021.667396307, 
897.660025595, 1157.753772199, 987.559410695, 2001.341199852, 
1553.300242875, 1714.026538454, 1369.596672549, 1840.028672943, 
1514.597606925, 2261.869533177, 1742.980544272, 1217.472820174, 
1052.419994718, 2412.091237232, 1840.049556762, 2412.355425093, 
1833.902224002, 1166.150434442, 989.36654541, 1727.399223697, 
1379.823422282, 1521.437037284, 1307.698311662, 1281.978151486, 
1073.978176322, 839.110335868, 745.255285071, 971.298071042, 
822.368065555, 229.866035998, 192.151802573, 269.191494746, 226.679807394, 
1884.205897945, 1570.878395831, 1419.07307384, 1183.35502134, 
1344.128634438, 1115.750833426, 1245.67831164, 1141.48282157, 
1163.157347453, 995.399323402, 1158.509308703, 992.306084298, 
10555.60271354, 44028.286710071, 7319.269197947, 2295.729281756, 
3063.256792958, 3622.027748796, 749.772815644, 3249.538889951, 
3968.357168338, 4565.736226542, 160.991914129, 4385.588327515, 
3459.367581677, 3191.871637791, 5303.571737911, 4439.625936658, 
2397.84635726, 5455.774784214, 5127.936752666, 4607.134537924, 
2541.025856693, 5286.049572237, 4894.706322916, 2508.70503033, 
3181.774190885, 1374.073499192, 3483.986868181, 3980.942928815, 
2773.932650003, 3452.651932961, 2955.494692275, 992.741929243, 
8124.252045849, 10629.581729564, 9522.957665608, 11343.109807218, 
921.364421637, 1041.114150216, 3527.436450492, 3427.798606229, 
3922.490009706, 3102.330209642, 3573.750886853, 3924.736407354, 
4467.245524367, 12816.437197695, 9597.125378863, 304.582440239, 
14914.96392496, 267.127750039, 11706.146702234, 12514.145950955
)), class = c("spec_tbl_df", "tbl_df", "tbl", "data.frame"), row.names = c(NA, 
-1064L))

In $H_A$ do you mean "not all means equal"? Because $\ne$ isn't transitive, your current expression is meaningless. — BruceET, May 22 '19 at 19:40
Yes, I mean "not all means" equal. I didn't realize that. Apologies. — dylanjm, May 22 '19 at 19:41
With distributions this skewed and this close to each other, you will need very large amounts of data. How much to you have? — whuber, May 24 '19 at 18:50
@whuber, so currently I am using proxy data while I wait for the simulations to run over the weekend. I have about 286 tests that I am simulating 10 times for each intervention, so I will have ~2860 observations for each group. The current data is actually just 5 replications of the base case (there is always variance in run time despite being the base case), hence why their distributions are extremely similar. I am expecting there to be a little more variance in the final data but with similar behavior. — dylanjm, May 24 '19 at 18:58
See https://stats.stackexchange.com/questions/69898 for a discussion of data somewhat like yours in a similar setting (but with two groups instead of five). — whuber, May 24 '19 at 19:09
@whuber Dang, do you think if I looked at the subgroups within the data, where I would have less observations but also less skewed data that I could potentially apply ANOVA? — dylanjm, May 24 '19 at 19:25
You can apply ANOVA in any event. I suspect it will be very conservative with such data in the sense of having a very hard time detecting a difference in mean values (which is what you are looking for when you're focusing on total computational time). The KW test doesn't really tell you much about differences in total computational time, but it might more readily detect that there is a difference of some sort among the groups. — whuber, May 24 '19 at 20:31
BTW, a similar situation with a more positive outcome (ANOVA worked well) is discussed at https://stats.stackexchange.com/questions/110418. — whuber, May 24 '19 at 20:34
I notice that your ```alive_time``` goes below zero, but it has a pretty consistent lower bound and shape at the lower end. Was this value derived from some other non-negative time value, with an amount subtracted from that? If so, can you please describe the derivation of ```alive_time``` from the underlying raw measurements? — Ben, May 29 '19 at 00:31
@Ben the density plots show values less than zero but there are actually no values <= 0. My understanding is that it has something to do with the way density is calculated for extremely small values close to zero. Alive time is merely the length of time, in seconds, from the launch of a simulation to the close of said simulation. — dylanjm, May 29 '19 at 00:56

Ben · Answer 1 · 2019-05-29T23:14:00.067

I note from your comment that you have clarified that all the values of alive_time are positive time values, and the density plots in your post go into negative values only because they are kernel densities. In view of this, the first thing I would recommend is to plot your data on a logarithmic scale. This will make it easier to see the shape of your data over the five categories. In particular, it will probably make it easier to see what appears to be a secondary "hump" in the distribution.

Testing equality of means across groups is usually handled with ANOVA, but this generally relies on use of the CLT approximation to the distribution of the sample mean, and this can be sensitive to extremely high positive skew. (In the case where you have an underlying distribution with skewness so extreme as to give infinite variance, the CLT does not even apply.) The Kruskal-Wallis test is a one-way ANOVA on ranks of the data, so it involves a loss of information of the actual times. For this reason, it is probably a poor test if you are interested in equality of means.

Plotting the densities more clearly: By plotting the data on a logarithmic scale it will be easier to see how skewed this data is (e.g., relative to the lognormal distribution), and this will give you a reasonable idea of what you are dealing with. Since you have provided your data, here is some R code to plot the density functions on a logarithmic scale. I have made a plot of the stacked density plots, and another plot of the overlaid density plots. (In my code, the data frame you use is referred to as DATA, so you will need to create this object. I have also extracted the times for the individual interventions, and named these vectors with upper-case intervention names.)

#Load required libraries and set theme
library(ggplot2);
THEME <- theme(plot.title    = element_text(hjust = 0.5, size = 14, face = 'bold'),
           plot.subtitle = element_text(hjust = 0.5, face = 'bold'));

#Plot stacked densities on logarithmic scale
ggplot(data = DATA, aes(x = alive_time)) + 
    geom_density(fill = 'blue') + facet_wrap(~ intv, ncol = 1) + THEME +
    expand_limits(x = c(10^(-1), 10^6)) + 
    scale_x_log10(breaks = scales::trans_breaks("log10", function(x) 10^x),
         labels = scales::trans_format("log10", scales::math_format(10^.x))) +
    ggtitle('Kernel Density Plots') + xlab('Alive Time') + ylab('Density');

#Plot overlayed densities on logarithmic scale
ggplot(data = DATA, aes(x = alive_time, colour = intv)) + 
    geom_density(size = 1) + THEME +
    labs(colour = 'Intervention') +
    expand_limits(x = c(10^(-1), 10^6)) + 
    scale_x_log10(breaks = scales::trans_breaks("log10", function(x) 10^x),
         labels = scales::trans_format("log10", scales::math_format(10^.x))) +
    ggtitle('Kernel Density Plots') + xlab('Alive Time') + ylab('Density');

You can see from these plots that the five intervention groups have very similar densities for the log-time. Visually, this suggests that there are only minor differences in the distributions, if any. We can test this formally using appropriate statistical tests.

Testing for a common distribution: In this case we can formally test for a common distribution using a multivariate Anderson-Darling test. This is a non-parametric test that does not assume any particular distributional form, so it is appropriate for this case, where the density does not conform to any common distributional family.

#Load required libraries and set theme
library(ggplot2);
library(kSamples);

#Apply Anderson-Darling Test
#The vectors DANIELL, ... , BASE were defined by extracting alive times for interventions
TEST <- ad.test(DANIELL, SHANEL, TIMESTEP, AD, BASE);
TEST;

 Anderson-Darling k-sample test.

Number of samples:  5
Sample sizes:  210, 211, 211, 213, 219
Number of ties: 0

Mean of  Anderson-Darling  Criterion: 4
Standard deviation of  Anderson-Darling  Criterion: 1.51789

T.AD = ( Anderson-Darling  Criterion - mean)/sigma

Null Hypothesis: All samples come from a common population.

              AD    T.AD  asympt. P-value
version 1: 3.196 -0.5296           0.6597
version 2: 3.200 -0.5294           0.6595

From this output we see that the test shows no evidence of a difference in the distributions (i.e., we do not reject the null hypothesis of a common distribution). That is, it is plausible that all the data from the five interventions comes from a common underlying distribution.

Hi Ben, thanks for the response. I went ahead and edited my question to include my data and a plot of the `alive_time` on a log10 scale. You were right about a 2nd hump in the data and now it appears to be more skewed left at the smaller values closer to one. I appreciate any insight as to what I could do to quantify differences in these run-times. Keep in mind that I'm currently using proxy-data, which will closely resemble this data's distribution, while I wait for the simulations to finish. — dylanjm, May 29 '19 at 14:57
I have updated the post to do some analysis on the data you provided. — Ben, May 29 '19 at 23:17
Nice work with the plots. Does the Anderson-Darling test give me some some estimate on what is the smallest between-group difference that was ruled out? Or can any kind of "power" (in whatever sense) defined? Not rejecting equality is IMHO not very interesting on its own from a practical standpoint as it may just mean we have too few data points. Obviously we have the plots as some sort of estimate, so this might not be a big deal... Just curious. — Martin Modrák, May 30 '19 at 06:17
Ben, thanks for the in-depth response. I really like this idea. I'm curious, is there something similar to what would be a TukeyHSD test to determine which distributions were most different? Anyway I can quantify further the differences between the cases? I looked into something called KL Divergence which seems close to this idea. I greatly appreciate the time and effort both you and @Martin have spent on this. — dylanjm, May 30 '19 at 16:52
@dylanjm: There are a whole bunch of nonparametric tests that test for equality of distributions. They all involve using some measure of difference between distributions, and then comparing this outcome to its null distribution under the null hypothesis of identical distributions. Some use KL divergence, and some use other measures of difference. If you are interested in this topic, it would be worth reading a book about nonparametric statistics, because it is a pretty big field. — Ben, May 30 '19 at 23:18
The AD test is a classical hypothesis test. It leads to a p-value for the hypotheses being tested. It does not give an estimate of the smallest between-group difference. — Ben, May 30 '19 at 23:19

Martin Modrák · Accepted Answer · 2019-05-29T07:18:18.170

First things first

Maybe a simple solution is enough? Note that provided you have a reasonable number of measurements, you should be quite OK with just measuring the average total time and choosing the intervention that has the fastest one. The only thing a statistical analysis gives you on top of that is a) a new possibility to mess up and b) some assessment of uncertainty, i.e. whether you need to collect more data to be more certain. Given enough data even the tiniest difference will be significant. You are (I presume) making an engineering choice and not publishing a scientific paper.

For the following I assume you want to do statistics.

What do you care about?

It is also IMHO good idea to think about what difference are you actually interested in. Do you really care about the difference in mean performance? Or do you care about some sort of worst-case scenarios?

I would also suggest that for your use case, you might be better served by estimating the pairwise differences with some measure of uncertainty rather than by testing hypothesis - if you see that there is "no significant difference" betweeen treatments A and B it is way less informative than e.g. "95% CI for mean performance of A is (10;30), for B is (15;50) and for the difference A-B is (-45,18)" - in the latter case you are not certain which is better, but you see that B has higher risk of very large absolute times.

Bootstrap - dirty but cheap

A very simple but useful way to estimate uncertainty is the Bootstrap. A simple scheme you could use is to choose $N$ of the runs at random (with replacement) and compute all the quantities of interest (in your case probably all means and pairwise differences in means). Repeat this a lot - say 1000 times. This will get you 1000 "samples" of each quantity. You can than treat the middle 95% of the samples as your 95% uncertainty interval. $N$ could be equal to your actual number of runs or a bit smaller (say 0.95 * number of runs), you may want to test multiple values of $N$ and see if this affects results.

Modelling

The full-fledged (and most difficult) approach is to try to find a good model of your dataset.

My suggestion would be to try brms or other flexible linear model package. I think a student-t response with log link or a gamma response could have reasonably fat tails to not be led astray by the outliers.

Alternatively, you can try to find a transformation that makes your data look more normal and then fit a model with "normal" or "student" families (with identity link) - log(sqrt(x)) might be your friend, see https://en.wikipedia.org/wiki/Variance-stabilizing_transformation for some other ideas.

Using brms you can then use posterior predictive checks (PPC) to test whether your response distribution at least roughly matches your data (e.g. ppc_dens_overlay) and get posterior credible intervals for all the central tendencies and differences of interest, including (with some fiddling) differences in say 95% quantiles of runtimes, should you be interested.

I can elaborate a bit more if you think this is a sensible approach for you.

Thank you for your response. I have minimal experience in `brms` and Bayesian stats in general but very interested in pursuing this. I went ahead and ran `fit1 — dylanjm, May 28 '19 at 17:05
Also, my motivation is to show our new solvers are quicker than our previous solvers and will be reported in a milestone but not an academic paper. The first idea was to just compare means (which I suppose is still possible). I opened my loud mouth to say we should have some statistical measure to be sure they are truly better. I may have bitten off more than I can chew haha. — dylanjm, May 28 '19 at 17:10
@dylanjm The problem with statistics is that it is easy to get *a number* but often quite hard to make sure the number represents anything in the real world :-) A simple trick would be to use Bootstrap (added to the answer). Getting grip of linear models, while interesting and useful might not be the best investment of your time. But anyway: `family=cumulative` is when your *response* (here time taken) is ordered, but distances between neighbouring values may be variable (as in "Strongly agree" - ... - "Strongly disagree" questions). 1/N — Martin Modrák, May 29 '19 at 07:11
2/2 if your `pp_check` shows strong disagreement betwen actual data (y) and what the model produced (yrep), something is wrong :-) I would check whether `intv` is treated as a factor (Pop Level Effects - which is mostly what you care about - have one entry for each but one of the levels of `intv`) and try other response families (`gamma` might also be reasonable). Or transform the data somehow (added to my answer). Note that there is no one-size-fits all solution, you need to understand your data and this may take non-trivial effort to debug. — Martin Modrák, May 29 '19 at 07:16
Seeing above to Ben's plots, do you think it would be reasonable to use a mixture prior of 3 approximately normal distributions on the log of my data and instead of doing differences as my beta parameter, remove the intercept and compute differences afterwards? Would this be overkill? — dylanjm, May 30 '19 at 16:56
Mixture response: this might work (not sure exactly what you mean with the intercept though). I would also plot histograms instead of densities, as sometimes the densities mislead and show spurious peaks. maybe a better way would be if there is some structure to hold on to - do the tests differ in input data/computer they run on etc.? Those could be added as predictors (preferably as varying intercepts I.e. 1|node). Or even varying intercept for each observation (1|id) — Martin Modrák, May 30 '19 at 20:05
Thanks Martin, both answers by you and Ben were great, but your answer lead me to learning an extreme amount of Bayesian modeling and gave me many different options of solving my problem. Both top notch answers. Thank you! — dylanjm, May 31 '19 at 18:55

Determine which performance intervention is best?

2 Answers2