280 lines
13 KiB
ReStructuredText
280 lines
13 KiB
ReStructuredText
.. _calibration:
|
||
|
||
=======================
|
||
Probability calibration
|
||
=======================
|
||
|
||
.. currentmodule:: sklearn.calibration
|
||
|
||
|
||
When performing classification you often want not only to predict the class
|
||
label, but also obtain a probability of the respective label. This probability
|
||
gives you some kind of confidence on the prediction. Some models can give you
|
||
poor estimates of the class probabilities and some even do not support
|
||
probability prediction (e.g., some instances of
|
||
:class:`~sklearn.linear_model.SGDClassifier`).
|
||
The calibration module allows you to better calibrate
|
||
the probabilities of a given model, or to add support for probability
|
||
prediction.
|
||
|
||
Well calibrated classifiers are probabilistic classifiers for which the output
|
||
of the :term:`predict_proba` method can be directly interpreted as a confidence
|
||
level.
|
||
For instance, a well calibrated (binary) classifier should classify the samples
|
||
such that among the samples to which it gave a :term:`predict_proba` value
|
||
close to 0.8,
|
||
approximately 80% actually belong to the positive class.
|
||
|
||
.. _calibration_curve:
|
||
|
||
Calibration curves
|
||
------------------
|
||
|
||
Calibration curves (also known as reliability diagrams) compare how well the
|
||
probabilistic predictions of a binary classifier are calibrated. It plots
|
||
the true frequency of the positive label against its predicted probability,
|
||
for binned predictions.
|
||
The x axis represents the average predicted probability in each bin. The
|
||
y axis is the *fraction of positives*, i.e. the proportion of samples whose
|
||
class is the positive class (in each bin). The top calibration curve plot
|
||
is created with :func:`CalibrationDisplay.from_estimators`, which uses
|
||
:func:`calibration_curve` to calculate the per bin average predicted
|
||
probabilities and fraction of positives.
|
||
:func:`CalibrationDisplay.from_estimator`
|
||
takes as input a fitted classifier, which is used to calculate the predicted
|
||
probabilities. The classifier thus must have :term:`predict_proba` method. For
|
||
the few classifiers that do not have a :term:`predict_proba` method, it is
|
||
possible to use :class:`CalibratedClassifierCV` to calibrate the classifier
|
||
outputs to probabilities.
|
||
|
||
The bottom histogram gives some insight into the behavior of each classifier
|
||
by showing the number of samples in each predicted probability bin.
|
||
|
||
.. figure:: ../auto_examples/calibration/images/sphx_glr_plot_compare_calibration_001.png
|
||
:target: ../auto_examples/calibration/plot_compare_calibration.html
|
||
:align: center
|
||
|
||
.. currentmodule:: sklearn.linear_model
|
||
|
||
:class:`LogisticRegression` returns well calibrated predictions by default as it directly
|
||
optimizes :ref:`log_loss`. In contrast, the other methods return biased probabilities;
|
||
with different biases per method:
|
||
|
||
.. currentmodule:: sklearn.naive_bayes
|
||
|
||
:class:`GaussianNB` tends to push probabilities to 0 or 1 (note the counts
|
||
in the histograms). This is mainly because it makes the assumption that
|
||
features are conditionally independent given the class, which is not the
|
||
case in this dataset which contains 2 redundant features.
|
||
|
||
.. currentmodule:: sklearn.ensemble
|
||
|
||
:class:`RandomForestClassifier` shows the opposite behavior: the histograms
|
||
show peaks at approximately 0.2 and 0.9 probability, while probabilities
|
||
close to 0 or 1 are very rare. An explanation for this is given by
|
||
Niculescu-Mizil and Caruana [1]_: "Methods such as bagging and random
|
||
forests that average predictions from a base set of models can have
|
||
difficulty making predictions near 0 and 1 because variance in the
|
||
underlying base models will bias predictions that should be near zero or one
|
||
away from these values. Because predictions are restricted to the interval
|
||
[0,1], errors caused by variance tend to be one-sided near zero and one. For
|
||
example, if a model should predict p = 0 for a case, the only way bagging
|
||
can achieve this is if all bagged trees predict zero. If we add noise to the
|
||
trees that bagging is averaging over, this noise will cause some trees to
|
||
predict values larger than 0 for this case, thus moving the average
|
||
prediction of the bagged ensemble away from 0. We observe this effect most
|
||
strongly with random forests because the base-level trees trained with
|
||
random forests have relatively high variance due to feature subsetting." As
|
||
a result, the calibration curve also referred to as the reliability diagram
|
||
(Wilks 1995 [2]_) shows a characteristic sigmoid shape, indicating that the
|
||
classifier could trust its "intuition" more and return probabilities closer
|
||
to 0 or 1 typically.
|
||
|
||
.. currentmodule:: sklearn.svm
|
||
|
||
Linear Support Vector Classification (:class:`LinearSVC`) shows an even more
|
||
sigmoid curve than :class:`~sklearn.ensemble.RandomForestClassifier`, which is
|
||
typical for maximum-margin methods (compare Niculescu-Mizil and Caruana [1]_),
|
||
which focus on difficult to classify samples that are close to the decision
|
||
boundary (the support vectors).
|
||
|
||
Calibrating a classifier
|
||
------------------------
|
||
|
||
.. currentmodule:: sklearn.calibration
|
||
|
||
Calibrating a classifier consists of fitting a regressor (called a
|
||
*calibrator*) that maps the output of the classifier (as given by
|
||
:term:`decision_function` or :term:`predict_proba`) to a calibrated probability
|
||
in [0, 1]. Denoting the output of the classifier for a given sample by :math:`f_i`,
|
||
the calibrator tries to predict :math:`p(y_i = 1 | f_i)`.
|
||
|
||
The samples that are used to fit the calibrator should not be the same
|
||
samples used to fit the classifier, as this would introduce bias.
|
||
This is because performance of the classifier on its training data would be
|
||
better than for novel data. Using the classifier output of training data
|
||
to fit the calibrator would thus result in a biased calibrator that maps to
|
||
probabilities closer to 0 and 1 than it should.
|
||
|
||
Usage
|
||
-----
|
||
|
||
The :class:`CalibratedClassifierCV` class is used to calibrate a classifier.
|
||
|
||
:class:`CalibratedClassifierCV` uses a cross-validation approach to ensure
|
||
unbiased data is always used to fit the calibrator. The data is split into k
|
||
`(train_set, test_set)` couples (as determined by `cv`). When `ensemble=True`
|
||
(default), the following procedure is repeated independently for each
|
||
cross-validation split: a clone of `base_estimator` is first trained on the
|
||
train subset. Then its predictions on the test subset are used to fit a
|
||
calibrator (either a sigmoid or isotonic regressor). This results in an
|
||
ensemble of k `(classifier, calibrator)` couples where each calibrator maps
|
||
the output of its corresponding classifier into [0, 1]. Each couple is exposed
|
||
in the `calibrated_classifiers_` attribute, where each entry is a calibrated
|
||
classifier with a :term:`predict_proba` method that outputs calibrated
|
||
probabilities. The output of :term:`predict_proba` for the main
|
||
:class:`CalibratedClassifierCV` instance corresponds to the average of the
|
||
predicted probabilities of the `k` estimators in the `calibrated_classifiers_`
|
||
list. The output of :term:`predict` is the class that has the highest
|
||
probability.
|
||
|
||
When `ensemble=False`, cross-validation is used to obtain 'unbiased'
|
||
predictions for all the data, via
|
||
:func:`~sklearn.model_selection.cross_val_predict`.
|
||
These unbiased predictions are then used to train the calibrator. The attribute
|
||
`calibrated_classifiers_` consists of only one `(classifier, calibrator)`
|
||
couple where the classifier is the `base_estimator` trained on all the data.
|
||
In this case the output of :term:`predict_proba` for
|
||
:class:`CalibratedClassifierCV` is the predicted probabilities obtained
|
||
from the single `(classifier, calibrator)` couple.
|
||
|
||
The main advantage of `ensemble=True` is to benefit from the traditional
|
||
ensembling effect (similar to :ref:`bagging`). The resulting ensemble should
|
||
both be well calibrated and slightly more accurate than with `ensemble=False`.
|
||
The main advantage of using `ensemble=False` is computational: it reduces the
|
||
overall fit time by training only a single base classifier and calibrator
|
||
pair, decreases the final model size and increases prediction speed.
|
||
|
||
Alternatively an already fitted classifier can be calibrated by setting
|
||
`cv="prefit"`. In this case, the data is not split and all of it is used to
|
||
fit the regressor. It is up to the user to
|
||
make sure that the data used for fitting the classifier is disjoint from the
|
||
data used for fitting the regressor.
|
||
|
||
:func:`sklearn.metrics.brier_score_loss` may be used to assess how
|
||
well a classifier is calibrated. However, this metric should be used with care
|
||
because a lower Brier score does not always mean a better calibrated model.
|
||
This is because the Brier score metric is a combination of calibration loss
|
||
and refinement loss. Calibration loss is defined as the mean squared deviation
|
||
from empirical probabilities derived from the slope of ROC segments.
|
||
Refinement loss can be defined as the expected optimal loss as measured by the
|
||
area under the optimal cost curve. As refinement loss can change
|
||
independently from calibration loss, a lower Brier score does not necessarily
|
||
mean a better calibrated model.
|
||
|
||
:class:`CalibratedClassifierCV` supports the use of two 'calibration'
|
||
regressors: 'sigmoid' and 'isotonic'.
|
||
|
||
.. _sigmoid_regressor:
|
||
|
||
Sigmoid
|
||
^^^^^^^
|
||
|
||
The sigmoid regressor is based on Platt's logistic model [3]_:
|
||
|
||
.. math::
|
||
p(y_i = 1 | f_i) = \frac{1}{1 + \exp(A f_i + B)}
|
||
|
||
where :math:`y_i` is the true label of sample :math:`i` and :math:`f_i`
|
||
is the output of the un-calibrated classifier for sample :math:`i`. :math:`A`
|
||
and :math:`B` are real numbers to be determined when fitting the regressor via
|
||
maximum likelihood.
|
||
|
||
The sigmoid method assumes the :ref:`calibration curve <calibration_curve>`
|
||
can be corrected by applying a sigmoid function to the raw predictions. This
|
||
assumption has been empirically justified in the case of :ref:`svm` with
|
||
common kernel functions on various benchmark datasets in section 2.1 of Platt
|
||
1999 [3]_ but does not necessarily hold in general. Additionally, the
|
||
logistic model works best if the calibration error is symmetrical, meaning
|
||
the classifier output for each binary class is normally distributed with
|
||
the same variance [6]_. This can be a problem for highly imbalanced
|
||
classification problems, where outputs do not have equal variance.
|
||
|
||
In general this method is most effective when the un-calibrated model is
|
||
under-confident and has similar calibration errors for both high and low
|
||
outputs.
|
||
|
||
Isotonic
|
||
^^^^^^^^
|
||
|
||
The 'isotonic' method fits a non-parametric isotonic regressor, which outputs
|
||
a step-wise non-decreasing function (see :mod:`sklearn.isotonic`). It
|
||
minimizes:
|
||
|
||
.. math::
|
||
\sum_{i=1}^{n} (y_i - \hat{f}_i)^2
|
||
|
||
subject to :math:`\hat{f}_i >= \hat{f}_j` whenever
|
||
:math:`f_i >= f_j`. :math:`y_i` is the true
|
||
label of sample :math:`i` and :math:`\hat{f}_i` is the output of the
|
||
calibrated classifier for sample :math:`i` (i.e., the calibrated probability).
|
||
This method is more general when compared to 'sigmoid' as the only restriction
|
||
is that the mapping function is monotonically increasing. It is thus more
|
||
powerful as it can correct any monotonic distortion of the un-calibrated model.
|
||
However, it is more prone to overfitting, especially on small datasets [5]_.
|
||
|
||
Overall, 'isotonic' will perform as well as or better than 'sigmoid' when
|
||
there is enough data (greater than ~ 1000 samples) to avoid overfitting [1]_.
|
||
|
||
Multiclass support
|
||
^^^^^^^^^^^^^^^^^^
|
||
|
||
Both isotonic and sigmoid regressors only
|
||
support 1-dimensional data (e.g., binary classification output) but are
|
||
extended for multiclass classification if the `base_estimator` supports
|
||
multiclass predictions. For multiclass predictions,
|
||
:class:`CalibratedClassifierCV` calibrates for
|
||
each class separately in a :ref:`ovr_classification` fashion [4]_. When
|
||
predicting
|
||
probabilities, the calibrated probabilities for each class
|
||
are predicted separately. As those probabilities do not necessarily sum to
|
||
one, a postprocessing is performed to normalize them.
|
||
|
||
.. topic:: Examples:
|
||
|
||
* :ref:`sphx_glr_auto_examples_calibration_plot_calibration_curve.py`
|
||
* :ref:`sphx_glr_auto_examples_calibration_plot_calibration_multiclass.py`
|
||
* :ref:`sphx_glr_auto_examples_calibration_plot_calibration.py`
|
||
* :ref:`sphx_glr_auto_examples_calibration_plot_compare_calibration.py`
|
||
|
||
.. topic:: References:
|
||
|
||
.. [1] `Predicting Good Probabilities with Supervised Learning
|
||
<https://www.cs.cornell.edu/~alexn/papers/calibration.icml05.crc.rev3.pdf>`_,
|
||
A. Niculescu-Mizil & R. Caruana, ICML 2005
|
||
|
||
.. [2] `On the combination of forecast probabilities for
|
||
consecutive precipitation periods.
|
||
<https://journals.ametsoc.org/waf/article/5/4/640/40179>`_
|
||
Wea. Forecasting, 5, 640–650., Wilks, D. S., 1990a
|
||
|
||
.. [3] `Probabilistic Outputs for Support Vector Machines and Comparisons
|
||
to Regularized Likelihood Methods.
|
||
<https://www.cs.colorado.edu/~mozer/Teaching/syllabi/6622/papers/Platt1999.pdf>`_
|
||
J. Platt, (1999)
|
||
|
||
.. [4] `Transforming Classifier Scores into Accurate Multiclass
|
||
Probability Estimates.
|
||
<https://dl.acm.org/doi/pdf/10.1145/775047.775151>`_
|
||
B. Zadrozny & C. Elkan, (KDD 2002)
|
||
|
||
.. [5] `Predicting accurate probabilities with a ranking loss.
|
||
<https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4180410/>`_
|
||
Menon AK, Jiang XJ, Vembu S, Elkan C, Ohno-Machado L.
|
||
Proc Int Conf Mach Learn. 2012;2012:703-710
|
||
|
||
.. [6] `Beyond sigmoids: How to obtain well-calibrated probabilities from
|
||
binary classifiers with beta calibration
|
||
<https://projecteuclid.org/euclid.ejs/1513306867>`_
|
||
Kull, M., Silva Filho, T. M., & Flach, P. (2017).
|