Statistical Analysis Stack Exchange
Statistical Analysis Stack Exchange

Questions tagged [ensemble-learning]

In machine learning, ensemble methods combine multiple algorithms to make a prediction. Bagging, boosting and stacking are some examples.

423 questions
287
votes
8 answers

Bagging, boosting and stacking in machine learning

What's the similarities and differences between these 3 methods: Bagging, Boosting, Stacking? Which is the best one? And why? Can you give me an example for each?
Bucsa Lucian
  • 2,979
  • 3
  • 13
  • 3
162
votes
3 answers

Gradient Boosting Tree vs Random Forest

Gradient tree boosting as proposed by Friedman uses decision trees as base learners. I'm wondering if we should make the base decision tree as complex as possible (fully grown) or simpler? Is there any explanation for the choice? Random Forest is…
FihopZz
  • 1,923
  • 4
  • 11
  • 9
107
votes
15 answers

US Election results 2016: What went wrong with prediction models?

First it was Brexit, now the US election. Many model predictions were off by a wide margin, and are there lessons to be learned here? As late as 4 pm PST yesterday, the betting markets were still favoring Hillary 4 to 1. I take it that the betting…
horaceT
  • 3,162
  • 3
  • 15
  • 19
57
votes
3 answers

Won't highly-correlated variables in random forest distort accuracy and feature-selection?

In my understanding, highly correlated variables won't cause multi-collinearity issues in random forest model (Please correct me if I'm wrong). However, on the other way, if I have too many variables containing similar information, will the model…
Yoki
  • 739
  • 1
  • 7
  • 10
44
votes
3 answers

Gradient Boosting for Linear Regression - why does it not work?

While learning about Gradient Boosting, I haven't heard about any constraints regarding the properties of a "weak classifier" that the method uses to build and ensemble model. However, I could not imagine an application of a GB that uses linear…
Matek
  • 749
  • 1
  • 6
  • 14
35
votes
4 answers

Ensemble of different kinds of regressors using scikit-learn (or any other python framework)

I am trying to solve the regression task. I found out that 3 models are working nicely for different subsets of data: LassoLARS, SVR and Gradient Tree Boosting. I noticed that when I make predictions using all these 3 models and then make a table of…
Maksim Khaitovich
  • 658
  • 1
  • 7
  • 12
30
votes
2 answers

Boosting neural networks

Well recently I was working on learning boosting algorithms, such as adaboost, gradient boost, and I have known the fact that the most common used weak-learner is trees. I really want to know are there some recent successful examples (I mean some…
user4380802
  • 401
  • 1
  • 4
  • 5
24
votes
1 answer

k-fold Cross validation of ensemble learning

I am confused about how to partition the data for k-fold cross validation of ensemble learning. Assuming I have an ensemble learning framework for classification. My first layer contains the classification models, e.g. svm, decision trees. My second…
Michael
  • 561
  • 2
  • 5
  • 11
23
votes
2 answers

On the "strength" of weak learners

I have several closely-related questions regarding weak learners in ensemble learning (e.g. boosting). This may sound dumb, but what are the benefits of using weak as opposed to strong learners? (e.g. why not boost with "strong" learning…
Amelio Vazquez-Reina
  • 17,546
  • 26
  • 74
  • 110
22
votes
5 answers

How do ensemble methods outperform all their constituents?

I am a bit confused about ensemble learning. In a nutshell, it runs k models and gets the average of these k models. How can it be guaranteed that the average of the k models would be better than any of the models by themselves? I do understand that…
user1691278
  • 541
  • 3
  • 8
21
votes
3 answers

Stacking/ensembling models with caret

I often find myself training several different predictive models using caret in R. I'll train them all on the same cross validation folds, using caret::: createFolds, then choose the best model based on cross-validated error. However, the median…
Zach
  • 22,308
  • 18
  • 114
  • 158
21
votes
3 answers

When should I not use an ensemble classifier?

In general, in a classification problem where the goal is to accurately predict out-of-sample class membership, when should I not to use an ensemble classifier? This question is closely related to Why not always use ensemble learning?. That question…
shadowtalker
  • 11,395
  • 3
  • 49
  • 109
20
votes
4 answers

Combining machine learning models

I'm kind of new to datamining/machine learning/etc. and have been reading about a couple ways to combine multiple models and runs of the same model to improve predictions. My impression from reading a couple papers (which are often interesting and…
screechOwl
  • 1,677
  • 3
  • 21
  • 32
19
votes
3 answers

What is the difference between bagging and random forest if only one explanatory variable is used?

" The fundamental difference between bagging and random forest is that in Random forests, only a subset of features are selected at random out of the total and the best split feature from the subset is used to split each node in a tree, unlike in…
Muhammed A. Zidan
  • 411
  • 2
  • 4
  • 6
17
votes
1 answer

How does gradient boosting calculate probability estimates?

I have been trying to understand gradient boosting reading various blogs, websites and trying to find my answer by looking through for example the XGBoost source code. However, I cannot seem to find an understandable explanation of how gradient…
Icyeval
  • 383
  • 3
  • 7
1
2 3
…
28 29