171 lines
7.3 KiB
ReStructuredText
171 lines
7.3 KiB
ReStructuredText
.. _ensemble:
|
|
|
|
================
|
|
Ensemble methods
|
|
================
|
|
|
|
.. currentmodule:: sklearn.ensemble
|
|
|
|
The goal of **ensemble methods** is to combine the predictions of several
|
|
models built with a given learning algorithm in order to improve
|
|
generalizability / robustness over a single model.
|
|
|
|
Two families of ensemble methods are usually distinguished:
|
|
|
|
- In **averaging methods**, the driving principle is to build several
|
|
models independently and then to average their predictions. On average,
|
|
the combined model is usually better than any of the single model
|
|
because its variance is reduced.
|
|
|
|
**Examples:** Bagging methods, :ref:`Forests of randomized trees <forest>`...
|
|
|
|
- By contrast, in **boosting methods**, models are built sequentially and one
|
|
tries to reduce the bias of the combined model. The motivation is to combine
|
|
several weak models to produce a powerful ensemble.
|
|
|
|
**Examples:** AdaBoost, Least Squares Boosting, Gradient Tree Boosting...
|
|
|
|
|
|
.. _forest:
|
|
|
|
Forests of randomized trees
|
|
===========================
|
|
|
|
The :mod:`sklearn.ensemble` module includes two averaging algorithms based
|
|
on randomized :ref:`decision trees <tree>`: the RandomForest algorithm
|
|
and the Extra-Trees method. Both algorithms are perturb-and-combine
|
|
techniques [B1998]_ specifically designed for trees. This means a diverse
|
|
set of classifiers is created by introducing randomness in the classifier
|
|
construction. The prediction of the ensemble is given as the averaged
|
|
prediction of the individual classifiers.
|
|
|
|
As other classifiers, forest classifiers have to be fitted with two
|
|
arrays: an array X of size ``[n_samples, n_features]`` holding the
|
|
training samples, and an array Y of size ``[n_samples]`` holding the
|
|
target values (class labels) for the training samples::
|
|
|
|
>>> from sklearn.ensemble import RandomForestClassifier
|
|
>>> X = [[0, 0], [1, 1]]
|
|
>>> Y = [0, 1]
|
|
>>> clf = RandomForestClassifier(n_estimators=10)
|
|
>>> clf = clf.fit(X, Y)
|
|
|
|
Random Forests
|
|
--------------
|
|
In random forests (see :class:`RandomForestClassifier` and
|
|
:class:`RandomForestRegressor` classes), each tree in the ensemble is
|
|
built from a sample drawn with replacement (i.e., a bootstrap sample)
|
|
from the training set. In addition, when splitting a node during the
|
|
construction of the tree, the split that is chosen is no longer the
|
|
best split among all features. Instead, the split that is picked is the
|
|
best split among a random subset of the features. As a result of this
|
|
randomness, the bias of the forest usually slightly increases (with
|
|
respect to the bias of a single non-random tree) but, due to averaging,
|
|
its variance also decreases, usually more than compensating for the
|
|
increase in bias, hence yielding an overall better model.
|
|
|
|
In contrast to the original publication [B2001]_, the scikit-learn
|
|
implementation combines classifiers by averaging their probabilistic
|
|
prediction, instead of letting each classifier vote for a single class.
|
|
|
|
|
|
Extremely Randomized Trees
|
|
--------------------------
|
|
|
|
In extremely randomized trees (see :class:`ExtraTreesClassifier`
|
|
and :class:`ExtraTreesRegressor` classes), randomness goes one step
|
|
further in the way splits are computed. As in random forests, a random
|
|
subset of candidate features is used, but instead of looking for the
|
|
most discriminative thresholds, thresholds are drawn at random for each
|
|
candidate feature and the best of these randomly-generated thresholds is
|
|
picked as the splitting rule. This usually allows to reduce the variance
|
|
of the model a bit more, at the expense of a slightly greater increase
|
|
in bias::
|
|
|
|
>>> from sklearn.cross_validation import cross_val_score
|
|
>>> from sklearn.datasets import make_blobs
|
|
>>> from sklearn.ensemble import RandomForestClassifier
|
|
>>> from sklearn.ensemble import ExtraTreesClassifier
|
|
>>> from sklearn.tree import DecisionTreeClassifier
|
|
|
|
>>> X, y = make_blobs(n_samples=10000, n_features=10, centers=100,
|
|
... random_state=0)
|
|
|
|
>>> clf = DecisionTreeClassifier(max_depth=None, min_samples_split=1,
|
|
... random_state=0)
|
|
>>> scores = cross_val_score(clf, X, y)
|
|
>>> scores.mean() # doctest: +ELLIPSIS
|
|
0.978...
|
|
|
|
>>> clf = RandomForestClassifier(n_estimators=10, max_depth=None,
|
|
... min_samples_split=1, random_state=0)
|
|
>>> scores = cross_val_score(clf, X, y)
|
|
>>> scores.mean() # doctest: +ELLIPSIS
|
|
0.999...
|
|
|
|
>>> clf = ExtraTreesClassifier(n_estimators=10, max_depth=None,
|
|
... min_samples_split=1, random_state=0)
|
|
>>> scores = cross_val_score(clf, X, y)
|
|
>>> scores.mean() > 0.999
|
|
True
|
|
|
|
.. figure:: ../auto_examples/ensemble/images/plot_forest_iris_1.png
|
|
:target: ../auto_examples/ensemble/plot_forest_iris.html
|
|
:align: center
|
|
:scale: 75%
|
|
|
|
Parameters
|
|
----------
|
|
|
|
The main parameters to adjust when using these methods is ``n_estimators``
|
|
and ``max_features``. The former is the number of trees in the forest. The
|
|
larger the better, but also the longer it will take to compute. In
|
|
addition, note that results will stop getting significantly better
|
|
beyond a critical number of trees. The latter is the size of the random
|
|
subsets of features to consider when splitting a node. The lower the
|
|
greater the reduction of variance, but also the greater the increase in
|
|
bias. Empiricial good default values are ``max_features=n_features``
|
|
for regression problems, and ``max_features=sqrt(n_features)`` for
|
|
classification tasks (where ``n_features`` is the number of features
|
|
in the data). The best results are also usually reached when setting
|
|
``max_depth=None`` in combination with ``min_samples_split=1`` (i.e.,
|
|
when fully developping the trees). Bear in mind though that these values
|
|
are usually not optimal. The best parameter values should always be cross-
|
|
validated. In addition, note that bootstrap samples are used by default
|
|
in random forests (``bootstrap=True``) while the default strategy is to
|
|
use the original dataset for building extra-trees (``bootstrap=False``).
|
|
|
|
When training on large datasets, where runtime and memory requirements
|
|
are important, it might also be beneficial to adjust the ``min_density``
|
|
parameter, that controls a heuristic for speeding up computations in
|
|
each tree. See :ref:`Complexity of trees<tree_complexity>` for details.
|
|
|
|
|
|
Parallelization
|
|
---------------
|
|
|
|
Finally, this module also features the parallel construction of the trees
|
|
and the parallel computation of the predictions through the ``n_jobs``
|
|
parameter. If ``n_jobs=k`` then computations are partitioned into
|
|
``k`` jobs, and run on ``k`` cores of the machine. If ``n_jobs=-1``
|
|
then all cores available on the machine are used. Note that because of
|
|
inter-process communication overhead, the speedup might not be linear
|
|
(i.e., using ``k`` jobs will unfortunately not be ``k`` times as
|
|
fast). Significant speedup can still be achieved though when building
|
|
a large number of trees, or when building a single tree requires a fair
|
|
amount of time (e.g., on large datasets).
|
|
|
|
.. topic:: Examples:
|
|
|
|
* :ref:`example_ensemble_plot_forest_iris.py`
|
|
* :ref:`example_ensemble_plot_forest_importances_faces.py`
|
|
|
|
.. topic:: References
|
|
|
|
.. [B2001] Leo Breiman, "Random Forests", Machine Learning, 45(1), 5-32, 2001.
|
|
|
|
.. [B1998] Leo Breiman, "Arcing Classifiers", Annals of Statistics 1998.
|
|
|
|
.. [GEW2006] Pierre Geurts, Damien Ernst., and Louis Wehenkel, "Extremely randomized
|
|
trees", Machine Learning, 63(1), 3-42, 2006.
|