Ensembles

References

• Ensemble Learning Wikipedia Article

Notes

In statistics and machine learning, ensemble methods use multiple learning algorithms to obtain better predictive performance than could be obtained from any of the constituent learning algorithms alone. Unlike a statistical ensemble in statistical mechanics, which is usually infinite, a machine learning ensemble consists of only a concrete set of alternative models, but typically allows for much more flexible structure to exist among those alternatives.

Ensembles combine multiple hypotheses to form one which should be theoretically better. Ensemble learning trains two or more machine learning algorithms on a specific classification or regression task. The algorithms within the ensemble model are generally referred to as base models, base learners, or weak learners in literature. These base models can be constructed using a single modeling algorithm, or several different algorithms. The idea is to train a diverse set of weak models on the same modeling task, such that the outputs of each weak learner have poor predictive ability (high bias) and among all weak learners, the outcome and error values exhibit high variance.

Ensemble learning typically refers to bagging, boosting, or stacking/blending techniques to induce high variance among the base models. Bagging creates diversity by generating random samples from the training observation and fitting the same model to each different sample. Boosting follows an iterative process by sequentially training each base model on the up-weighted errors of the previous base model, producing an additive model to reduce the final model errors. Stacking or blending consists of different base models, each trained independently to be combined into the ensemble model. Common application of ensemble learning include random forests, Boosted Tree models, and Gradient Boosted Tree Models.