0

I am trying to compare the predictive ability of various models in predicting survival in patients. I would like to examine the predictive performance of each model using 4 tests: squared Pearson correlation coefficient (R2), root mean squared prediction error (RMSPE), mean absolute prediction error (MAPE), and prediction bias.

I am trying to use SPSS to perform these tests but I cannot figure which would be the dependent (predicted or actual) and which would be the independent variable in the regression analysis.

Could anyone clarify this point?

Thanks, JH

user60247
  • 1
  • 1
  • using actual as an independent variable is the most logical, since it is not affected by the model choice. – katya Nov 24 '14 at 17:50
  • 1
    Thanks for the response. But why would the predicted be an independent variable? Basically, I am trying to do what this study did for a different model: Comparison of Regression Methods for Modeling Intensive Care Length of Stay (http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0109684). Please let me know your thoughts. – user60247 Nov 24 '14 at 19:56
  • I think I was saying actual as independent :), just a convention, though. But what exactly are you trying to do?- they use back-predicted, btw. This paper uses many possible models which might make sense if those were the only ones used in the field (but that isn't stated). Why those particular choices? Why test at the level of jeopardized method performance with already violated assumptions vs. choosing the best method for these particular data/confirming via simulation? – katya Nov 25 '14 at 01:52
  • Well, I want to compare different models that predict survival in patients with brain metastasis and see which comes closest to the actual survival (I have the actual survival for these patients). Since this paper compared 8 different models with the actual data using the aforementioned 4 tests, I thought it would be best to use the same methodology to my project. I am not sure if I understand what you mean by "Why test at the level of jeopardized method performance with already violated assumptions vs. choosing the best method for these particular data/confirming via simulation?" – user60247 Nov 25 '14 at 18:20
  • I actually played around on SPSS and figured out that for R squared test, it doesn't matter which is the independent or dependent variable. However, I later realized that it does matter by a significant amount for the other tests: – user60247 Nov 25 '14 at 18:25

1 Answers1

2

Too long for a comment: OLS regression of x on y vs. y on x minimizes horizontal vs. vertical differences, respectively. A very good treatment of this and additional references can be found here What is the difference between linear regression on y with x and x with y?

What I meant there is that a choice between untransformed vs. transformed data and reg type should be made prior to fitting the model. E.g. if the data are skewed (they were indeed) and do not satisfy assumptions of OLS, this model should not be used, unless a certain transformation improves data distribution. So a legit approach would be to understand the structure of your data, select the most appropriate transformation, then choose and justify the appropriate model(s), not try all models and see which give a better, or less worse in their case:) result.

Although I don't think it is stated directly, I believe they used actual survival as x and back-predicted survival as y.

Community
  • 1
katya
  • 2,084
  • 8
  • 11
  • Katya,Thanks for your comment. Very insightful and helpful. Perhaps I am not clarifying the purpose of my study. I would to know which of the 4 models comes closest to predicting the actual survival in a certain population. In other words, this is a head-head comparison of various models and see which is most predictive in order to determine their clinical utility and applicability. In the end, I will submit my results for each model and not ones that only come closest. – user60247 Nov 26 '14 at 02:08
  • If you have a large dataset, I would recommend developing the models on a training set and then applying them to a reserved subset of test data. This would be a true test of the models. But comparing models is fine, as long as their initial assumptions were not violated, which is my pt above, and separate pt from having a test set. This paper is somewhat insightful, even in its current state, but I would not have accepted it with this unjustified meddley of methods. Not everyone thinks this way, and it was. – katya Nov 26 '14 at 05:37
  • Ahh I see. The models I am comparing have been published in the litereature already by folks who developed them. I would be simply comparing what each model predicted for my cohort of 100 patients to their actual survival and analyzing them against each other for their utility, or futility for that matter. Hence the need for using the aforementioned tests. I am not attempting to develop my own model. – user60247 Nov 26 '14 at 18:20