1

I am using about 256 predictors and target is sales. I am using a software called Alteryx which is R based. I have tried to run Random Forest, Spline model and Neural nets on same data.

I used partitioning to create test and training data sets.

I am fairly new to field, hence I am not sure how to compare performance of these models.

I was advised to compare Pearson correlation coefficient between predicted and actual value for all models and select the best one.

Although I am new, from what I have learned, I don't think its a good way to compare models.Pearson for Forest model is about 0.98 and for neural nets it's about 0.91. both figures are for testing set.

Is there any better way to compare and cross validate models?

Should I use R squared value/ AIC to compare?Is there any other manual way to compare models?

I have predicted and actual values for training data sets for all models, is there any way to compare using those?

Sycorax
  • 76,417
  • 20
  • 189
  • 313
Neeraj Patki
  • 21
  • 4
  • 1
    Is this cross-validated accuracy? I ask because your model will obviously perform very well when you test it on the same data that you trained it on... – Sycorax Apr 07 '15 at 15:09
  • 2
    See my answer here: http://stats.stackexchange.com/a/112052/1569 – Hong Ooi Apr 07 '15 at 15:12
  • Please provide more info on your data and experimental setup. – Vladislavs Dovgalecs Apr 07 '15 at 15:58
  • 1
    possible duplicate of [Random Forest - How to handle overfitting](http://stats.stackexchange.com/questions/111968/random-forest-how-to-handle-overfitting) – Sycorax Apr 07 '15 at 16:29
  • In such cases, when predictive accuracy is implausibly high, it's worth checking whether any predictors are actually part and parcel of the outcome. E.g., if Y = Sales and one predictor is Sales in Region A. – rolando2 Jul 13 '18 at 15:42

0 Answers0