203 lines
11 KiB
ReStructuredText
203 lines
11 KiB
ReStructuredText
.. _calibration:
|
|
|
|
=======================
|
|
Probability calibration
|
|
=======================
|
|
|
|
.. currentmodule:: sklearn.calibration
|
|
|
|
|
|
When performing classification you often want not only to predict the class
|
|
label, but also obtain a probability of the respective label. This probability
|
|
gives you some kind of confidence on the prediction. Some models can give you
|
|
poor estimates of the class probabilities and some even do not support
|
|
probability prediction. The calibration module allows you to better calibrate
|
|
the probabilities of a given model, or to add support for probability
|
|
prediction.
|
|
|
|
Well calibrated classifiers are probabilistic classifiers for which the output
|
|
of the predict_proba method can be directly interpreted as a confidence level.
|
|
For instance, a well calibrated (binary) classifier should classify the samples
|
|
such that among the samples to which it gave a predict_proba value close to 0.8,
|
|
approximately 80% actually belong to the positive class. The following plot compares
|
|
how well the probabilistic predictions of different classifiers are calibrated:
|
|
|
|
.. figure:: ../auto_examples/calibration/images/plot_compare_calibration_001.png
|
|
:target: ../auto_examples/calibration/plot_compare_calibration.html
|
|
:align: center
|
|
|
|
.. currentmodule:: sklearn.linear_model
|
|
|
|
:class:`LogisticRegression` returns well calibrated predictions by default as it directly
|
|
optimizes log-loss. In contrast, the other methods return biased probabilities;
|
|
with different biases per method:
|
|
|
|
.. currentmodule:: sklearn.naive_bayes
|
|
|
|
* :class:`GaussianNB` tends to push probabilties to 0 or 1 (note the
|
|
counts in the histograms). This is mainly because it makes the assumption
|
|
that features are conditionally independent given the class, which is not
|
|
the case in this dataset which contains 2 redundant features.
|
|
|
|
.. currentmodule:: sklearn.ensemble
|
|
|
|
* :class:`RandomForestClassifier` shows the opposite behavior: the histograms
|
|
show peaks at approximately 0.2 and 0.9 probability, while probabilities close to
|
|
0 or 1 are very rare. An explanation for this is given by Niculescu-Mizil
|
|
and Caruana [4]: "Methods such as bagging and random forests that average
|
|
predictions from a base set of models can have difficulty making predictions
|
|
near 0 and 1 because variance in the underlying base models will bias
|
|
predictions that should be near zero or one away from these values. Because
|
|
predictions are restricted to the interval [0,1], errors caused by variance
|
|
tend to be one-sided near zero and one. For example, if a model should
|
|
predict p = 0 for a case, the only way bagging can achieve this is if all
|
|
bagged trees predict zero. If we add noise to the trees that bagging is
|
|
averaging over, this noise will cause some trees to predict values larger
|
|
than 0 for this case, thus moving the average prediction of the bagged
|
|
ensemble away from 0. We observe this effect most strongly with random
|
|
forests because the base-level trees trained with random forests have
|
|
relatively high variance due to feature subseting." As a result, the
|
|
calibration curve shows a characteristic sigmoid shape, indicating that the
|
|
classifier could trust its "intuition" more and return probabilties closer
|
|
to 0 or 1 typically.
|
|
|
|
.. currentmodule:: sklearn.svm
|
|
|
|
* Linear Support Vector Classification (:class:`LinearSVC`) shows an even more sigmoid curve
|
|
as the RandomForestClassifier, which is typical for maximum-margin methods
|
|
(compare Niculescu-Mizil and Caruana [4]), which focus on hard samples
|
|
that are close to the decision boundary (the support vectors).
|
|
|
|
.. currentmodule:: sklearn.calibration
|
|
|
|
Two approaches for performing calibration of probabilistic predictions are
|
|
provided: a parametric approach based on Platt's sigmoid model and a
|
|
non-parametric approach based on isotonic regression (:mod:`sklearn.isotonic`).
|
|
Probability calibration should be done on new data not used for model fitting.
|
|
The class :class:`CalibratedClassifierCV` uses a cross-validation generator and
|
|
estimates for each split the model parameter on the train samples and the
|
|
calibration of the test samples. The probabilities predicted for the
|
|
folds are then averaged. Already fitted classifiers can be calibrated by
|
|
:class:`CalibratedClassifierCV` via the paramter cv="prefit". In this case,
|
|
the user has to take care manually that data for model fitting and calibration
|
|
are disjoint.
|
|
|
|
The following images demonstrate the benefit of probability calibration.
|
|
The first image present a dataset with 2 classes and 3 blobs of
|
|
data. The blob in the middle contains random samples of each class.
|
|
The probability for the samples in this blob should be 0.5.
|
|
|
|
.. figure:: ../auto_examples/calibration/images/plot_calibration_001.png
|
|
:target: ../auto_examples/calibration/plot_calibration.html
|
|
:align: center
|
|
|
|
The following image shows on the data above the estimated probability
|
|
using a Gaussian naive Bayes classifier without calibration,
|
|
with a sigmoid calibration and with a non-parametric isotonic
|
|
calibration. One can observe that the non-parametric model
|
|
provides the most accurate probability estimates for samples
|
|
in the middle, i.e., 0.5.
|
|
|
|
.. figure:: ../auto_examples/calibration/images/plot_calibration_002.png
|
|
:target: ../auto_examples/calibration/plot_calibration.html
|
|
:align: center
|
|
|
|
.. currentmodule:: sklearn.metrics
|
|
|
|
The following experiment is performed on an artificial dataset for binary
|
|
classification with 100.000 samples (1.000 of them are used for model fitting)
|
|
with 20 features. Of the 20 features, only 2 are informative and 10 are
|
|
redundant. The figure shows the estimated probabilities obtained with
|
|
logistic regression, a linear support-vector classifier (SVC), and linear SVC with
|
|
both isotonic calibration and sigmoid calibration. The calibration performance
|
|
is evaluated with Brier score :func:`brier_score_loss`, reported in the legend
|
|
(the smaller the better).
|
|
|
|
.. figure:: ../auto_examples/calibration/images/plot_calibration_curve_002.png
|
|
:target: ../auto_examples/calibration/plot_calibration_curve.html
|
|
:align: center
|
|
|
|
One can observe here that logistic regression is well calibrated as its curve is
|
|
nearly diagonal. Linear SVC's calibration curve has a sigmoid curve, which is
|
|
typical for an under-confident classifier. In the case of LinearSVC, this is
|
|
caused by the margin property of the hinge loss, which lets the model focus on
|
|
hard samples that are close to the decision boundary (the support vectors). Both
|
|
kinds of calibration can fix this issue and yield nearly identical results.
|
|
The next figure shows the calibration curve of Gaussian naive Bayes on
|
|
the same data, with both kinds of calibration and also without calibration.
|
|
|
|
.. figure:: ../auto_examples/calibration/images/plot_calibration_curve_001.png
|
|
:target: ../auto_examples/calibration/plot_calibration_curve.html
|
|
:align: center
|
|
|
|
One can see that Gaussian naive Bayes performs very badly but does so in an
|
|
other way than linear SVC: While linear SVC exhibited a sigmoid calibration
|
|
curve, Gaussian naive Bayes' calibration curve has a transposed-sigmoid shape.
|
|
This is typical for an over-confident classifier. In this case, the classifier's
|
|
overconfidence is caused by the redundant features which violate the naive Bayes
|
|
assumption of feature-independence.
|
|
|
|
Calibration of the probabilities of Gaussian naive Bayes with isotonic
|
|
regression can fix this issue as can be seen from the nearly diagonal
|
|
calibration curve. Sigmoid calibration also improves the brier score slightly,
|
|
albeit not as strongly as the non-parametric isotonic calibration. This is an
|
|
intrinsic limitation of sigmoid calibration, whose parametric form assumes a
|
|
sigmoid rather than a transposed-sigmoid curve. The non-parametric isotonic
|
|
calibration model, however, makes no such strong assumptions and can deal with
|
|
either shape, provided that there is sufficient calibration data. In general,
|
|
sigmoid calibration is preferable in cases where the calibration curve is sigmoid
|
|
and where there is limited calibration data, while isotonic calibration is
|
|
preferable for non-sigmoid calibration curves and in situations where large
|
|
amounts of data are available for calibration.
|
|
|
|
.. currentmodule:: sklearn.calibration
|
|
|
|
:class:`CalibratedClassifierCV` can also deal with classification tasks that
|
|
involve more than two classes if the base estimator can do so. In this case,
|
|
the classifier is calibrated first for each class separately in an one-vs-rest
|
|
fashion. When predicting probabilities for unseen data, the calibrated
|
|
probabilities for each class are predicted separately. As those probabilities
|
|
do not necessarily sum to one, a postprocessing is performed to normalize them.
|
|
|
|
The next image illustrates how sigmoid calibration changes predicted
|
|
probabilities for a 3-class classification problem. Illustrated is the standard
|
|
2-simplex, where the three corners correspond to the three classes. Arrows point
|
|
from the probability vectors predicted by an uncalibrated classifier to the
|
|
probability vectors predicted by the same classifier after sigmoid calibration
|
|
on a hold-out validation set. Colors indicate the true class of an instance
|
|
(red: class 1, green: class 2, blue: class 3).
|
|
|
|
.. figure:: ../auto_examples/calibration/images/plot_calibration_multiclass_000.png
|
|
:target: ../auto_examples/calibration/plot_calibration_multiclass.html
|
|
:align: center
|
|
|
|
The base classifier is a random forest classifier with 25 base estimators
|
|
(trees). If this classifier is trained on all 800 training datapoints, it is
|
|
overly confident in its predictions and thus incurs a large log-loss.
|
|
Calibrating an identical classifier, which was trained on 600 datapoints, with
|
|
method='sigmoid' on the remaining 200 datapoints reduces the confidence of the
|
|
predictions, i.e., moves the probability vectors from the edges of the simplex
|
|
towards the center:
|
|
|
|
.. figure:: ../auto_examples/calibration/images/plot_calibration_multiclass_001.png
|
|
:target: ../auto_examples/calibration/plot_calibration_multiclass.html
|
|
:align: center
|
|
|
|
This calibration results in a lower log-loss. Note that an alternative would
|
|
have been to increase the number of base estimators which would have resulted in
|
|
a similar decrease in log-loss.
|
|
|
|
.. topic:: References:
|
|
|
|
.. [1] Obtaining calibrated probability estimates from decision trees
|
|
and naive Bayesian classifiers, B. Zadrozny & C. Elkan, ICML 2001
|
|
|
|
.. [2] Transforming Classifier Scores into Accurate Multiclass
|
|
Probability Estimates, B. Zadrozny & C. Elkan, (KDD 2002)
|
|
|
|
.. [3] Probabilistic Outputs for Support Vector Machines and Comparisons to
|
|
Regularized Likelihood Methods, J. Platt, (1999)
|
|
|
|
.. [4] Predicting Good Probabilities with Supervised Learning,
|
|
A. Niculescu-Mizil & R. Caruana, ICML 2005
|