404 lines
16 KiB
ReStructuredText
404 lines
16 KiB
ReStructuredText
.. _introduction:
|
|
|
|
An introduction to machine learning with scikit-learn
|
|
=====================================================
|
|
|
|
.. topic:: Section contents
|
|
|
|
In this section, we introduce the `machine learning
|
|
<https://en.wikipedia.org/wiki/Machine_learning>`_
|
|
vocabulary that we use throughout scikit-learn and give a
|
|
simple learning example.
|
|
|
|
|
|
Machine learning: the problem setting
|
|
-------------------------------------
|
|
|
|
In general, a learning problem considers a set of n
|
|
`samples <https://en.wikipedia.org/wiki/Sample_(statistics)>`_ of
|
|
data and then tries to predict properties of unknown data. If each sample is
|
|
more than a single number and, for instance, a multi-dimensional entry
|
|
(aka `multivariate <https://en.wikipedia.org/wiki/Multivariate_random_variable>`_
|
|
data), it is said to have several attributes or **features**.
|
|
|
|
We can separate learning problems in a few large categories:
|
|
|
|
* `supervised learning <https://en.wikipedia.org/wiki/Supervised_learning>`_,
|
|
in which the data comes with additional attributes that we want to predict
|
|
(:ref:`Click here <supervised-learning>`
|
|
to go to the scikit-learn supervised learning page).This problem
|
|
can be either:
|
|
|
|
* `classification
|
|
<https://en.wikipedia.org/wiki/Classification_in_machine_learning>`_:
|
|
samples belong to two or more classes and we
|
|
want to learn from already labeled data how to predict the class
|
|
of unlabeled data. An example of classification problem would
|
|
be the handwritten digit recognition example, in which the aim is
|
|
to assign each input vector to one of a finite number of discrete
|
|
categories. Another way to think of classification is as a discrete
|
|
(as opposed to continuous) form of supervised learning where one has a
|
|
limited number of categories and for each of the n samples provided,
|
|
one is to try to label them with the correct category or class.
|
|
|
|
* `regression <https://en.wikipedia.org/wiki/Regression_analysis>`_:
|
|
if the desired output consists of one or more
|
|
continuous variables, then the task is called *regression*. An
|
|
example of a regression problem would be the prediction of the
|
|
length of a salmon as a function of its age and weight.
|
|
|
|
* `unsupervised learning <https://en.wikipedia.org/wiki/Unsupervised_learning>`_,
|
|
in which the training data consists of a set of input vectors x
|
|
without any corresponding target values. The goal in such problems
|
|
may be to discover groups of similar examples within the data, where
|
|
it is called `clustering <https://en.wikipedia.org/wiki/Cluster_analysis>`_,
|
|
or to determine the distribution of data within the input space, known as
|
|
`density estimation <https://en.wikipedia.org/wiki/Density_estimation>`_, or
|
|
to project the data from a high-dimensional space down to two or three
|
|
dimensions for the purpose of *visualization*
|
|
(:ref:`Click here <unsupervised-learning>`
|
|
to go to the Scikit-Learn unsupervised learning page).
|
|
|
|
.. topic:: Training set and testing set
|
|
|
|
Machine learning is about learning some properties of a data set
|
|
and applying them to new data. This is why a common practice in
|
|
machine learning to evaluate an algorithm is to split the data
|
|
at hand into two sets, one that we call the **training set** on which
|
|
we learn data properties and one that we call the **testing set**
|
|
on which we test these properties.
|
|
|
|
.. _loading_example_dataset:
|
|
|
|
Loading an example dataset
|
|
--------------------------
|
|
|
|
`scikit-learn` comes with a few standard datasets, for instance the
|
|
`iris <https://en.wikipedia.org/wiki/Iris_flower_data_set>`_ and `digits
|
|
<http://archive.ics.uci.edu/ml/datasets/Pen-Based+Recognition+of+Handwritten+Digits>`_
|
|
datasets for classification and the `boston house prices dataset
|
|
<http://archive.ics.uci.edu/ml/datasets/Housing>`_ for regression.
|
|
|
|
In the following, we start a Python interpreter from our shell and then
|
|
load the ``iris`` and ``digits`` datasets. Our notational convention is that
|
|
``$`` denotes the shell prompt while ``>>>`` denotes the Python
|
|
interpreter prompt::
|
|
|
|
$ python
|
|
>>> from sklearn import datasets
|
|
>>> iris = datasets.load_iris()
|
|
>>> digits = datasets.load_digits()
|
|
|
|
A dataset is a dictionary-like object that holds all the data and some
|
|
metadata about the data. This data is stored in the ``.data`` member,
|
|
which is a ``n_samples, n_features`` array. In the case of supervised
|
|
problem, one or more response variables are stored in the ``.target`` member. More
|
|
details on the different datasets can be found in the :ref:`dedicated
|
|
section <datasets>`.
|
|
|
|
For instance, in the case of the digits dataset, ``digits.data`` gives
|
|
access to the features that can be used to classify the digits samples::
|
|
|
|
>>> print(digits.data) # doctest: +NORMALIZE_WHITESPACE
|
|
[[ 0. 0. 5. ..., 0. 0. 0.]
|
|
[ 0. 0. 0. ..., 10. 0. 0.]
|
|
[ 0. 0. 0. ..., 16. 9. 0.]
|
|
...,
|
|
[ 0. 0. 1. ..., 6. 0. 0.]
|
|
[ 0. 0. 2. ..., 12. 0. 0.]
|
|
[ 0. 0. 10. ..., 12. 1. 0.]]
|
|
|
|
and ``digits.target`` gives the ground truth for the digit dataset, that
|
|
is the number corresponding to each digit image that we are trying to
|
|
learn::
|
|
|
|
>>> digits.target
|
|
array([0, 1, 2, ..., 8, 9, 8])
|
|
|
|
.. topic:: Shape of the data arrays
|
|
|
|
The data is always a 2D array, shape ``(n_samples, n_features)``, although
|
|
the original data may have had a different shape. In the case of the
|
|
digits, each original sample is an image of shape ``(8, 8)`` and can be
|
|
accessed using::
|
|
|
|
>>> digits.images[0]
|
|
array([[ 0., 0., 5., 13., 9., 1., 0., 0.],
|
|
[ 0., 0., 13., 15., 10., 15., 5., 0.],
|
|
[ 0., 3., 15., 2., 0., 11., 8., 0.],
|
|
[ 0., 4., 12., 0., 0., 8., 8., 0.],
|
|
[ 0., 5., 8., 0., 0., 9., 8., 0.],
|
|
[ 0., 4., 11., 0., 1., 12., 7., 0.],
|
|
[ 0., 2., 14., 5., 10., 12., 0., 0.],
|
|
[ 0., 0., 6., 13., 10., 0., 0., 0.]])
|
|
|
|
The :ref:`simple example on this dataset
|
|
<sphx_glr_auto_examples_classification_plot_digits_classification.py>` illustrates how starting
|
|
from the original problem one can shape the data for consumption in
|
|
scikit-learn.
|
|
|
|
.. topic:: Loading from external datasets
|
|
|
|
To load from an external dataset, please refer to :ref:`loading external datasets <external_datasets>`.
|
|
|
|
Learning and predicting
|
|
------------------------
|
|
|
|
In the case of the digits dataset, the task is to predict, given an image,
|
|
which digit it represents. We are given samples of each of the 10
|
|
possible classes (the digits zero through nine) on which we *fit* an
|
|
`estimator <https://en.wikipedia.org/wiki/Estimator>`_ to be able to *predict*
|
|
the classes to which unseen samples belong.
|
|
|
|
In scikit-learn, an estimator for classification is a Python object that
|
|
implements the methods ``fit(X, y)`` and ``predict(T)``.
|
|
|
|
An example of an estimator is the class ``sklearn.svm.SVC`` that
|
|
implements `support vector classification
|
|
<https://en.wikipedia.org/wiki/Support_vector_machine>`_. The
|
|
constructor of an estimator takes as arguments the parameters of the
|
|
model, but for the time being, we will consider the estimator as a black
|
|
box::
|
|
|
|
>>> from sklearn import svm
|
|
>>> clf = svm.SVC(gamma=0.001, C=100.)
|
|
|
|
.. topic:: Choosing the parameters of the model
|
|
|
|
In this example we set the value of ``gamma`` manually. It is possible
|
|
to automatically find good values for the parameters by using tools
|
|
such as :ref:`grid search <grid_search>` and :ref:`cross validation
|
|
<cross_validation>`.
|
|
|
|
We call our estimator instance ``clf``, as it is a classifier. It now must
|
|
be fitted to the model, that is, it must *learn* from the model. This is
|
|
done by passing our training set to the ``fit`` method. As a training
|
|
set, let us use all the images of our dataset apart from the last
|
|
one. We select this training set with the ``[:-1]`` Python syntax,
|
|
which produces a new array that contains all but
|
|
the last entry of ``digits.data``::
|
|
|
|
>>> clf.fit(digits.data[:-1], digits.target[:-1]) # doctest: +NORMALIZE_WHITESPACE
|
|
SVC(C=100.0, cache_size=200, class_weight=None, coef0=0.0,
|
|
decision_function_shape='ovr', degree=3, gamma=0.001, kernel='rbf',
|
|
max_iter=-1, probability=False, random_state=None, shrinking=True,
|
|
tol=0.001, verbose=False)
|
|
|
|
Now you can predict new values, in particular, we can ask to the
|
|
classifier what is the digit of our last image in the ``digits`` dataset,
|
|
which we have not used to train the classifier::
|
|
|
|
>>> clf.predict(digits.data[-1:])
|
|
array([8])
|
|
|
|
The corresponding image is the following:
|
|
|
|
.. image:: /auto_examples/datasets/images/sphx_glr_plot_digits_last_image_001.png
|
|
:target: ../../auto_examples/datasets/plot_digits_last_image.html
|
|
:align: center
|
|
:scale: 50
|
|
|
|
As you can see, it is a challenging task: the images are of poor
|
|
resolution. Do you agree with the classifier?
|
|
|
|
A complete example of this classification problem is available as an
|
|
example that you can run and study:
|
|
:ref:`sphx_glr_auto_examples_classification_plot_digits_classification.py`.
|
|
|
|
|
|
Model persistence
|
|
-----------------
|
|
|
|
It is possible to save a model in the scikit by using Python's built-in
|
|
persistence model, namely `pickle <https://docs.python.org/2/library/pickle.html>`_::
|
|
|
|
>>> from sklearn import svm
|
|
>>> from sklearn import datasets
|
|
>>> clf = svm.SVC()
|
|
>>> iris = datasets.load_iris()
|
|
>>> X, y = iris.data, iris.target
|
|
>>> clf.fit(X, y) # doctest: +NORMALIZE_WHITESPACE
|
|
SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,
|
|
decision_function_shape='ovr', degree=3, gamma='auto', kernel='rbf',
|
|
max_iter=-1, probability=False, random_state=None, shrinking=True,
|
|
tol=0.001, verbose=False)
|
|
|
|
>>> import pickle
|
|
>>> s = pickle.dumps(clf)
|
|
>>> clf2 = pickle.loads(s)
|
|
>>> clf2.predict(X[0:1])
|
|
array([0])
|
|
>>> y[0]
|
|
0
|
|
|
|
In the specific case of the scikit, it may be more interesting to use
|
|
joblib's replacement of pickle (``joblib.dump`` & ``joblib.load``),
|
|
which is more efficient on big data, but can only pickle to the disk
|
|
and not to a string::
|
|
|
|
>>> from sklearn.externals import joblib
|
|
>>> joblib.dump(clf, 'filename.pkl') # doctest: +SKIP
|
|
|
|
Later you can load back the pickled model (possibly in another Python process)
|
|
with::
|
|
|
|
>>> clf = joblib.load('filename.pkl') # doctest:+SKIP
|
|
|
|
.. note::
|
|
|
|
``joblib.dump`` and ``joblib.load`` functions also accept file-like object
|
|
instead of filenames. More information on data persistence with Joblib is
|
|
available `here <https://pythonhosted.org/joblib/persistence.html>`_.
|
|
|
|
Note that pickle has some security and maintainability issues. Please refer to
|
|
section :ref:`model_persistence` for more detailed information about model
|
|
persistence with scikit-learn.
|
|
|
|
|
|
Conventions
|
|
-----------
|
|
|
|
scikit-learn estimators follow certain rules to make their behavior more
|
|
predictive.
|
|
|
|
|
|
Type casting
|
|
~~~~~~~~~~~~
|
|
|
|
Unless otherwise specified, input will be cast to ``float64``::
|
|
|
|
>>> import numpy as np
|
|
>>> from sklearn import random_projection
|
|
|
|
>>> rng = np.random.RandomState(0)
|
|
>>> X = rng.rand(10, 2000)
|
|
>>> X = np.array(X, dtype='float32')
|
|
>>> X.dtype
|
|
dtype('float32')
|
|
|
|
>>> transformer = random_projection.GaussianRandomProjection()
|
|
>>> X_new = transformer.fit_transform(X)
|
|
>>> X_new.dtype
|
|
dtype('float64')
|
|
|
|
In this example, ``X`` is ``float32``, which is cast to ``float64`` by
|
|
``fit_transform(X)``.
|
|
|
|
Regression targets are cast to ``float64``, classification targets are
|
|
maintained::
|
|
|
|
>>> from sklearn import datasets
|
|
>>> from sklearn.svm import SVC
|
|
>>> iris = datasets.load_iris()
|
|
>>> clf = SVC()
|
|
>>> clf.fit(iris.data, iris.target) # doctest: +NORMALIZE_WHITESPACE
|
|
SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,
|
|
decision_function_shape='ovr', degree=3, gamma='auto', kernel='rbf',
|
|
max_iter=-1, probability=False, random_state=None, shrinking=True,
|
|
tol=0.001, verbose=False)
|
|
|
|
>>> list(clf.predict(iris.data[:3]))
|
|
[0, 0, 0]
|
|
|
|
>>> clf.fit(iris.data, iris.target_names[iris.target]) # doctest: +NORMALIZE_WHITESPACE
|
|
SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,
|
|
decision_function_shape='ovr', degree=3, gamma='auto', kernel='rbf',
|
|
max_iter=-1, probability=False, random_state=None, shrinking=True,
|
|
tol=0.001, verbose=False)
|
|
|
|
>>> list(clf.predict(iris.data[:3])) # doctest: +NORMALIZE_WHITESPACE
|
|
['setosa', 'setosa', 'setosa']
|
|
|
|
Here, the first ``predict()`` returns an integer array, since ``iris.target``
|
|
(an integer array) was used in ``fit``. The second ``predict()`` returns a string
|
|
array, since ``iris.target_names`` was for fitting.
|
|
|
|
Refitting and updating parameters
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Hyper-parameters of an estimator can be updated after it has been constructed
|
|
via the :func:`sklearn.pipeline.Pipeline.set_params` method. Calling ``fit()``
|
|
more than once will overwrite what was learned by any previous ``fit()``::
|
|
|
|
>>> import numpy as np
|
|
>>> from sklearn.svm import SVC
|
|
|
|
>>> rng = np.random.RandomState(0)
|
|
>>> X = rng.rand(100, 10)
|
|
>>> y = rng.binomial(1, 0.5, 100)
|
|
>>> X_test = rng.rand(5, 10)
|
|
|
|
>>> clf = SVC()
|
|
>>> clf.set_params(kernel='linear').fit(X, y) # doctest: +NORMALIZE_WHITESPACE
|
|
SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,
|
|
decision_function_shape='ovr', degree=3, gamma='auto', kernel='linear',
|
|
max_iter=-1, probability=False, random_state=None, shrinking=True,
|
|
tol=0.001, verbose=False)
|
|
>>> clf.predict(X_test)
|
|
array([1, 0, 1, 1, 0])
|
|
|
|
>>> clf.set_params(kernel='rbf').fit(X, y) # doctest: +NORMALIZE_WHITESPACE
|
|
SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,
|
|
decision_function_shape='ovr', degree=3, gamma='auto', kernel='rbf',
|
|
max_iter=-1, probability=False, random_state=None, shrinking=True,
|
|
tol=0.001, verbose=False)
|
|
>>> clf.predict(X_test)
|
|
array([0, 0, 0, 1, 0])
|
|
|
|
Here, the default kernel ``rbf`` is first changed to ``linear`` after the
|
|
estimator has been constructed via ``SVC()``, and changed back to ``rbf`` to
|
|
refit the estimator and to make a second prediction.
|
|
|
|
Multiclass vs. multilabel fitting
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
When using :class:`multiclass classifiers <sklearn.multiclass>`,
|
|
the learning and prediction task that is performed is dependent on the format of
|
|
the target data fit upon::
|
|
|
|
>>> from sklearn.svm import SVC
|
|
>>> from sklearn.multiclass import OneVsRestClassifier
|
|
>>> from sklearn.preprocessing import LabelBinarizer
|
|
|
|
>>> X = [[1, 2], [2, 4], [4, 5], [3, 2], [3, 1]]
|
|
>>> y = [0, 0, 1, 1, 2]
|
|
|
|
>>> classif = OneVsRestClassifier(estimator=SVC(random_state=0))
|
|
>>> classif.fit(X, y).predict(X)
|
|
array([0, 0, 1, 1, 2])
|
|
|
|
In the above case, the classifier is fit on a 1d array of multiclass labels and
|
|
the ``predict()`` method therefore provides corresponding multiclass predictions.
|
|
It is also possible to fit upon a 2d array of binary label indicators::
|
|
|
|
>>> y = LabelBinarizer().fit_transform(y)
|
|
>>> classif.fit(X, y).predict(X)
|
|
array([[1, 0, 0],
|
|
[1, 0, 0],
|
|
[0, 1, 0],
|
|
[0, 0, 0],
|
|
[0, 0, 0]])
|
|
|
|
Here, the classifier is ``fit()`` on a 2d binary label representation of ``y``,
|
|
using the :class:`LabelBinarizer <sklearn.preprocessing.LabelBinarizer>`.
|
|
In this case ``predict()`` returns a 2d array representing the corresponding
|
|
multilabel predictions.
|
|
|
|
Note that the fourth and fifth instances returned all zeroes, indicating that
|
|
they matched none of the three labels ``fit`` upon. With multilabel outputs, it
|
|
is similarly possible for an instance to be assigned multiple labels::
|
|
|
|
>> from sklearn.preprocessing import MultiLabelBinarizer
|
|
>> y = [[0, 1], [0, 2], [1, 3], [0, 2, 3], [2, 4]]
|
|
>> y = MultiLabelBinarizer().fit_transform(y)
|
|
>> classif.fit(X, y).predict(X)
|
|
array([[1, 1, 0, 0, 0],
|
|
[1, 0, 1, 0, 0],
|
|
[0, 1, 0, 1, 0],
|
|
[1, 0, 1, 1, 0],
|
|
[0, 0, 1, 0, 1]])
|
|
|
|
In this case, the classifier is fit upon instances each assigned multiple labels.
|
|
The :class:`MultiLabelBinarizer <sklearn.preprocessing.MultiLabelBinarizer>` is
|
|
used to binarize the 2d array of multilabels to ``fit`` upon. As a result,
|
|
``predict()`` returns a 2d array with multiple predicted labels for each instance. |