There is no "gradient" in the standard Random Forest formulation, but can I combine random Forests with an optimisation method like Gradient Descent or SGD?
Can I use Adam (Adaptive moment estimation) for a Random Forest?
Asked
Active
Viewed 1,291 times
1
-
1Can you please describe in more detail what the scenario you are exploring? There is no "gradient" in the standard Random Forest formulation so it is unclear what is being asked. – usεr11852 May 07 '19 at 09:41
-
Thanks. Hopefully it is formulated better now. – Ferdi May 07 '19 at 10:19
-
2Erm.. Now the Q literally reads "*it is unclear what is being asked*" so I think it is even more confusing. But anyway I think I understand it enough to "attempt" an answer. Yes, we can. If we use GD we are simply describing Gradient Boosted Regression Trees (GBRT), i.e. Gradient Boosting. If we want SGD, we are getting closer to something like DART boosters (Drop-outs). Momentum methods have not been created yet mostly because we get great results with Newton's method (e.g. XGBoost, LightGBM). – usεr11852 May 07 '19 at 10:57
-
2I answered based on the text of the question. In my opinion, the title does not make much sense, perhaps it could be improved. – Firebug May 07 '19 at 12:29
1 Answers
2
In a sense, yes, you can.
As I alluded in my question here What *is* an Artificial Neural Network?, a decision tree is a neural network, lato sensu. So you can train these neural networks in parallel and combine then, with end-to-end gradient based optimization.
Another way you could combine random forests is through gradient boosting. Usually, for gradient boosting, "weak" learners are combined into a "strong" learner. In random forests and bagging techniques in general, "strong", biased, learners, are combined into a, hopefully, more parsimonious model. But nothing precludes you from combining random forests in gradient boosting fashion, which answers "can I combine random Forests with an optimisation method like Gradient Descent or SGD?".
Firebug
- 15,262
- 5
- 60
- 127