134 lines
4.3 KiB
ReStructuredText
134 lines
4.3 KiB
ReStructuredText
|
|
.. _feature_selection:
|
|
|
|
=================
|
|
Feature selection
|
|
=================
|
|
|
|
.. currentmodule:: sklearn.feature_selection
|
|
|
|
The classes in the :mod:`sklearn.feature_selection` module can be used
|
|
for feature selection/dimensionality reduction on sample sets, either to
|
|
improve estimators' accuracy scores or to boost their performance on very
|
|
high-dimensional datasets.
|
|
|
|
Univariate feature selection
|
|
============================
|
|
|
|
Univariate feature selection works by selecting the best features based on
|
|
univariate statistical tests. It can seen as a preprocessing step
|
|
to an estimator. Scikit-Learn exposes feature selection routines
|
|
a objects that implement the `transform` method:
|
|
|
|
* selecting the k-best features :class:`SelectKBest`
|
|
|
|
* setting a percentile of features to keep :class:`SelectPercentile`
|
|
|
|
* using common univariate statistical tests for each feature:
|
|
false positive rate :class:`SelectFpr`, false discovery rate
|
|
:class:`SelectFdr`, or family wise error :class:`SelectFwe`.
|
|
|
|
These objects take as input a scoring function that returns
|
|
univariate p-values:
|
|
|
|
* For regression: :func:`f_regression`
|
|
|
|
* For classification: :func:`chi2` or :func:`f_classif`
|
|
|
|
.. topic:: Feature selection with sparse data
|
|
|
|
If you use sparse data (i.e. data represented as sparse matrices),
|
|
only :func:`chi2` will deal with the data without making it dense.
|
|
|
|
.. warning::
|
|
|
|
Beware not to use a regression scoring function with a classification
|
|
problem, you will get useless results.
|
|
|
|
.. topic:: Examples:
|
|
|
|
:ref:`example_plot_feature_selection.py`
|
|
|
|
|
|
Recursive feature elimination
|
|
=============================
|
|
|
|
.. currentmodule:: sklearn.feature_selection.rfe
|
|
|
|
Given an external estimator that assigns weights to features (e.g., the
|
|
coefficients of a linear model), the goal of recursive feature elimination (RFE)
|
|
is to select features by recursively considering smaller and smaller sets of
|
|
features. First, the estimator is trained on the initial set of features and
|
|
weights are assigned to each one of them. Then, features whose absolute weights
|
|
are the smallest are pruned from the current set features. That procedure is
|
|
recursively repeated on the pruned set until the desired number of features to
|
|
select is eventually reached.
|
|
|
|
.. topic:: Examples:
|
|
|
|
* :ref:`example_plot_rfe_digits.py`: A recursive feature elimination example
|
|
showing the relevance of pixels in a digit classification task.
|
|
|
|
* :ref:`example_plot_rfe_with_cross_validation.py`: A recursive feature
|
|
elimination example with automatic tuning of the number of features
|
|
selected with cross-validation.
|
|
|
|
|
|
L1-based feature selection
|
|
==========================
|
|
|
|
.. currentmodule:: sklearn.feature_selection
|
|
|
|
Linear models penalized with the L1 norm have sparse solutions. When the goal
|
|
is to reduce the dimensionality of the data to use with another classifier, the
|
|
`transform` method of `LogisticRegression` and `LinearSVC` can be used::
|
|
|
|
>>> from sklearn.svm import LinearSVC
|
|
>>> from sklearn.datasets import load_iris
|
|
>>> iris = load_iris()
|
|
>>> X, y = iris.data, iris.target
|
|
>>> X.shape
|
|
(150, 4)
|
|
>>> X_new = LinearSVC(C=0.01, penalty="l1", dual=False).fit_transform(X, y)
|
|
>>> X_new.shape
|
|
(150, 3)
|
|
|
|
The parameter C controls the sparsity: the smaller the fewer features.
|
|
|
|
.. topic:: Examples:
|
|
|
|
* :ref:`example_document_classification_20newsgroups.py`: Comparison
|
|
of different algorithms for document classification including L1-based
|
|
feature selection.
|
|
|
|
|
|
Tree-based feature selection
|
|
============================
|
|
|
|
Tree-based estimators (see the :mod:`sklearn.tree` module and forest
|
|
of trees in the :mod:`sklearn.ensemble` module) can be used to compute
|
|
feature importances, which in turn can be used to discard irrelevant
|
|
features::
|
|
|
|
>>> from sklearn.ensemble import ExtraTreesClassifier
|
|
>>> from sklearn.datasets import load_iris
|
|
>>> iris = load_iris()
|
|
>>> X, y = iris.data, iris.target
|
|
>>> X.shape
|
|
(150, 4)
|
|
>>> clf = ExtraTreesClassifier(compute_importances=True, random_state=0)
|
|
>>> X_new = clf.fit(X, y).transform(X)
|
|
>>> X_new.shape
|
|
(150, 2)
|
|
|
|
.. topic:: Examples:
|
|
|
|
* :ref:`example_ensemble_plot_forest_importances.py`: example on
|
|
synthetic data showing the recovery of the actually meaningful
|
|
features.
|
|
|
|
* :ref:`example_ensemble_plot_forest_importances_faces.py`: example
|
|
on face recognition data.
|
|
|
|
|