2

I currently have a quite ok model fitting my data with Regression Tree and a Random Forest. However, while trying to move on to a Neural Network (to evaluate other models), I have been struggling to put above average R2_score results. Even though my RF model could put good results.

So far I:

  • Tested several combinations of activation (relu, tanh etc.)
  • Tested different layers-size combinations
  • Normalizing/Standardizing my Data
  • Removed a couple features (which actually performed OK in the Random Forest)
  • Tried different random samplings by changing the stratification and the RandomSeed values

As I'm pretty much getting a bit confused by those approaches, my question might be a bit elementary, but: Does a model that was able to be fit by a regression tree, should also be able to be fit by a NN? Is there any restriction I'm unaware of?

Nicolas Vieira
  • 21
  • 4
  • 1
    I do not think you are forgetting some particular restriction but I hope you can appreciate that there is not reason a particular (well-tuned) neural network *has to* outperform a random forest. That being said, you do not mention the `hidden_layer_sizes` and the optimisers you have tried which are rather significant influences to the networks performance. – usεr11852 Mar 23 '18 at 10:13
  • Absolutely! My goal is not, by any means, outperform the RF using NN, but more like being able to tune in both and see if I can get at least closer results. For the hidden_layer_sizes, I've been iterating in a range fashion and checking if any combination performs best (hidden_layer_sizes=(j, ), with j iterating from 10 to 100 in steps of 10). As for the optimizer I've been trying mostly with the stochastic gradient-based 'ADAM'. Will have a more look on some others. It is already a heads up! thanks! – Nicolas Vieira Mar 23 '18 at 10:21
  • Cool. Best of luck and if you do answer your question yourself do not hesitate to post the answer for it! :) (+1 by the way) – usεr11852 Mar 23 '18 at 10:24
  • What are your convergence criteria for the NN training? Are you using the default weight initializations? These two things can have a dramatic effect on neural network performance. I agree with @usεr11852 that there's no reason a NN has to outperform (or even perform equally to) a RF on any particular problem, but equivalently there's no reason it should systematically underperform either. – Josh Mar 23 '18 at 10:59
  • Closely related: https://stats.stackexchange.com/questions/352036/what-should-i-do-when-my-neural-network-doesnt-learn – Sycorax Jul 03 '18 at 02:05

0 Answers0