4

I ran a comparison where I estimated the parameters of a model using $S$ different methods several times. More specifically I simulated $M$ datasets $\boldsymbol{y}_1, \dots, \boldsymbol{y}_m$ from a model parametrized by $\boldsymbol{\theta}$, where $\boldsymbol{\theta}$ is $p$-dimensional. Then I have estimated the model parameters by using each dataset and each method. As a results I have $M \times p$ point estimates for each statistical method.

So far I'm presenting the results by using a table where each element is an average squared error (over the $M$ runs):

                Model 1 Average Squared Errors
          theta_1    theta_2    theta_3    theta_4      
Method1     0.1        0.01      0.091      1.11 
Method2     0.3       0.003      0.047      0.79 
Method3     0.24       0.03      0.042      0.08
Method4     0.65       0.01      0.007       0.4 
Best       Method1    Method2   Method4   Method3

In statistical methodology literature this is seems to be the standard way of presenting results, but I find it quite hard to interpret. In particular, it is not clear which method is the winner.

Drawing conclusions is even more difficult when you try the methods on several models, i.e. you have a second table:

                Model 2 Average Squared Errors
          theta_1    theta_2    theta_3    theta_4      
Method1     0.5        0.14      0.01       0.3 
Method2     0.1        0.03      0.007      0.7 
Method3     0.14       0.023     0.1        0.8
Method4     0.05       0.1       0.02       0.4 
Best       Method4    Method2   Method1   Method1

In my case I am using 5 methods and 6 model, so I can't ask the reader to go through page after page of tables. So if you really wanted to come up with a ranking of the methods, how would you pool the results together to come up with it?

Finally this kind of tables are really boring to read, and there must be more entertaining ways of presenting such results!

Matteo Fasiolo
  • 3,134
  • 2
  • 20
  • 29

1 Answers1

4

Use Caterpillar Plots showing Mean +/- squared error. Some examples are shown here:

https://stackoverflow.com/questions/13847936/in-r-plotting-random-effects-from-lmer-lme4-package-using-qqmath-or-dotplot

Community
  • 1
Flask
  • 1,711
  • 1
  • 14
  • 24
  • Thanks, that's certainly better than a table. But wouldn't I need a plot for each parameter? In that case the number of plots grows really fast, especially if I have several tables. – Matteo Fasiolo Nov 24 '13 at 16:50
  • @MatteoFasiolo So you have 30 combinations of method/model times each parameter? I think for each parameter you could have one plot and fit 3 or so per page. I don't think I have seen an approach better than caterpillar plots. How many parameters do you have? – Flask Nov 24 '13 at 16:58
  • Yes I have 30 combinations model-method and 4 parameters per model, so that would make around 24 plots. An idea could be to pull together the results, so to have a single caterpillar plot for each model, representing the squared error averaged over the 4 parameters. I'm not sure if it's possible to do that in a sensitive way. – Matteo Fasiolo Nov 24 '13 at 17:07
  • @MatteoFasiolo For each parameter use one vertical plot. Put the different model results (ie each method used) next to each other in the same plot and colored different. Perhaps leave a blank space in between. This should be 4 plots with 30 lines each, which could possibly fit on one page. Or perhaps I do not understand your data. – Flask Nov 24 '13 at 17:12
  • The problem is that each model has 4 parameters, but the 4 parameters of the first model have nothing to do with the 4 parameters of the second model, etc... So I can't really put them together (they have a different interpretation and are on different scales). – Matteo Fasiolo Nov 24 '13 at 17:18
  • @MatteoFasiolo I think understand now. I think you will just have a few pages of plots. Or maybe you can sort the squared errors by size and do one large barplot. – Flask Nov 24 '13 at 17:33