* Initial implementation
* Forgot to add to second __add__ list
* Update split method parameter doc
* Added example; changed default test_size to 0.1; added to author list
* StratifiedGroupKFold impl and other improvements
* Add class to __all__ spec
* Remove random_state when no shuffle
* Tighter formatting
* Update the implementation of StratifiedGroupKFold
* Add StratifiedGroupKFold to __init__
* Add y checks to StartifiedGroupKFold
* Raise error if n_splits > max num samples in class
* Warn if n_splits > mn num samples in class
* Add SGKfold to general repr test
* Add SGKFold to 2d_y test case
* Add SGKfold to value erros test case
Parameters are the same as for StratifiedKFold
to ensure similar behavior given n_groups == n_samples
* Add SGKFold to StratifiedKFold test cases
The idea is to ensure similar behavior when groups are trivial
(n_groups == n_samples)
* Add SGKFold to reproducibility test case
* Add SGKFold to GroupKFold test case
* Add SGKFold to nested cv test case
* Add SGKFold to random_state with shuffle=False test case
* Add SGKFold to constant splits test case
* Fix repr test case
* Fix formatting issues
* Add samples to a fold with least num samples
Required to produce balanced size folds when the distribution of y is
more or less the same
* Remove GroupShuffleSplit impl
* Add notes to StratifiedGroupKFold
* Fix doctest
* Added stratified group kfold tests
* Better variable naming
* Add section to documentation
* Remove leftover StratifiedGroupShuffleSplit import
* Add changelist and reference to original kernel
* Better naming for least populated class check
* Better expression for number of labels
* Remove use of Counter
We already have this data in output of np.unique
* Add tests for homogeneous groups
* Add StratifiedGroupKFold test against GroupKFold
* Add changes to changelist in docstring
* Add StratifiedGroupKFold to classes.rst
* Fix description of StratifiedGroupKFold
* Move license notice out of docstring
* Disambiguate labels to classes in doc
* Add changelog entry
* Fix changelog author entry
* Fix StratifiedGroupKFold docstring
* Better variable names
* Remove defaultdict in favor of numpy indexing
* Extracted best_fold search into a separate method
* Make use of numpy broadcasting instead of for loop
* Encode groups and use arrays instead of dicts
* Use numpy sort instead of python
* Clarify shuffling behavior of StratifiedGroupKF in docs
* Switch name from label_idx to class_idx
* Remove accidentally leftover comment
* Fix np.sort keyword to support numpy < 1.15
* Fix typo in docstring
* Add StratifiedGroupKFold to visualization doc
* Add visualization for uneven group as an example
* Fix image numbers to match updated example
* Add author
* Add SGKF visualization to docs
* Add comments for groups in stratified CV tests
Co-authored-by: Leandro Hermida <hermidal@cs.umd.edu>
Co-authored-by: marrodion <rodion_martynov@epam.com>
* Updated cross validation accuracy metric
* Added deterministic results
* Update std value
Co-authored-by: Juan Carlos Alfaro Jiménez <JuanCarlos.Alfaro@uclm.es>
* Revert output
Co-authored-by: Thomas J. Fan <thomasjpfan@gmail.com>
Co-authored-by: Juan Carlos Alfaro Jiménez <JuanCarlos.Alfaro@uclm.es>
Co-authored-by: Thomas J. Fan <thomasjpfan@gmail.com>
#### Reference Issues/PRs
<!--
Example: Fixes#1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
This PR addresses issue #12466.
#### What does this implement/fix? Explain your changes.
This PR does the 3 following things:
- Rewrite the `cv` parameter description in `GridSearchCV`
- Link the new `CV splitter` description to an existing example
- Add an example with a custom iterable
Thanks for reviewing this!
Close#12466
* add FutureWarning for methods with defaults=3
* add explicit cv values to fix assertion errors
* add tests for catching the FutureWarning
* Write current deprecation version
* Add deprecation in docstring
* change default cv value to None
* change cv from 3 to 5 in the examples
* upgrade doctests
* update doctest in tutorial
* update doctest in cross-validation doc
* fix tests
* add entry to whats new
* address Gael comments
* address Gael comments 2
* fix wrong indentation
* update doc
* add docstring deprecation warning in CV subclasses
* address Andy's comments
* fix PR number
* fix flake8
* add filterwarnings in tests
* fix doctests
* cv=None mendatory in Ridge
* fix warning related errors
* skip some doctests warnings
* make travis happy
* change from deprecated to versionchanged
* fix doctests and remove skipping
* address comments
* DOC Add NOTE that unless random_state is set, split will not be identical
* TST use np.testing.assert_equal for nested lists/arrays
* TST Make sure cv param can be a generator
* DOC rank_ becomes a link when rendered
* Use test_...
* Remove blank line; Add if shuffle is True
* Fix tests
* Explicitly test for GeneratorType
* TST Add the else clause
* TST Add comment on usage of np.testing.assert_array_equal
* TYPO
* MNT Remove if ;
* Address Joel's comments
* merge the identical points in doc
* DOC address Andy's comments
* Move comment to before the check for generator type
* ENH cross_val_score now supports multiple metrics
* DOCFIX permutation_test_score
* ENH validate multiple metric scorers
* ENH Move validation of multimetric scoring param out
* ENH GridSearchCV and RandomizedSearchCV now support multiple metrics
* EXA Add an example demonstrating the multiple metric in GridSearchCV
* ENH Let check_multimetric_scoring tell if its multimetric or not
* FIX For single metric name of scorer should remain 'score'
* ENH validation_curve and learning_curve now support multiple metrics
* MNT move _aggregate_score_dicts helper into _validation.py
* TST More testing/ Fixing scores to the correct values
* EXA Add cross_val_score to multimetric example
* Rename to multiple_metric_evaluation.py
* MNT Remove scaffolding
* FIX doctest imports
* FIX wrap the scorer and unwrap the score when using _score() in rfe
* TST Cleanup the tests. Test for is_multimetric too
* TST Make sure it registers as single metric when scoring is of that type
* PEP8
* Don't use dict comprehension to make it work in python2.6
* ENH/FIX/TST grid_scores_ should not be available for multimetric evaluation
* FIX+TST delegated methods NA when multimetric is enabled...
TST Add general tests to GridSearchCV and RandomizedSearchCV
* ENH add option to disable delegation on multimetric scoring
* Remove old function from __all__
* flake8
* FIX revert disable_on_multimetric
* stash
* Fix incorrect rebase
* [ci skip]
* Make sure refit works as expected and remove irrelevant tests
* Allow passing standard scorers by name in multimetric scorers
* Fix example
* flake8
* Address reviews
* Fix indentation
* Ensure {'acc': 'accuracy'} and ['precision'] are valid inputs
* Test that for single metric, 'score' is a key
* Typos
* Fix incorrect rebase
* Compare multimetric grid search with multiple single metric searches
* Test X, y list and pandas input; Test multimetric for unsupervised grid search
* Fix tests; Unsupervised multimetric gs will not pass until #8117 is merged
* Make a plot of Precision vs ROC AUC for RandomForest varying the n_estimators
* Add example to grid_search.rst
* Use the classic tuning of C param in SVM instead of estimators in RF
* FIX Remove scoring arg in deafult scorer test
* flake8
* Search for min_samples_split in DTC; Also show f-score
* REVIEW Make check_multimetric_scoring private
* FIX Add more samples to see if 3% mismatch on 32 bit systems gets fixed
* REVIEW Plot best score; Shorten legends
* REVIEW/COSMIT multimetric --> multi-metric
* REVIEW Mark the best scores of P/R scores too
* Revert "FIX Add more samples to see if 3% mismatch on 32 bit systems gets fixed"
This reverts commit ba766d98353380a186fbc3dade211670ee72726d.
* ENH Use looping for iid testing
* FIX use param grid as scipy's stats dist in 0.12 do not accept seed
* ENH more looping less code; Use small non-noisy dataset
* FIX Use named arg after expanded args
* TST More testing of the refit parameter
* Test that in multimetric search refit to single metric, the delegated methods
work as expected.
* Test that setting probability=False works with multimetric too
* Test refit=False gives sensible error
* COSMIT multimetric --> multi-metric
* REV Correct example doc
* COSMIT
* REVIEW Make tests stronger; Fix bugs in _check_multimetric_scorer
* REVIEW refit param: Raise for empty strings
* TST Invalid refit params
* REVIEW Use <scorer_name> alone; recall --> Recall
* REV specify when we expect scorers to not be None
* FLAKE8
* REVERT multimetrics in learning_curve and validation_curve
* REVIEW Simpler coding style
* COSMIT
* COSMIT
* REV Compress example a bit. Move comment to top
* FIX fit_grid_point's previous API must be preserved
* Flake8
* TST Use loop; Compare with single-metric
* REVIEW Use dict-comprehension instead of helper
* REVIEW Remove redundant test
* Fix tests incorrect braces
* COSMIT
* REVIEW Use regexp
* REV Simplify aggregation of score dicts
* FIX precision and accuracy test
* FIX doctest and flake8
* TST the best_* attributes multimetric with single metric
* Address @jnothman's review
* Address more comments \o/
* DOCFIXES
* Fix use the validated fit_param from fit's arguments
* Revert alpha to a lower value as before
* Using def instead of lambda
* Address @jnothman's review batch 1: Fix tests / Doc fixes
* Remove superfluous tests
* Remove more superfluous testing
* TST/FIX loop over refit and check found n_clusters
* Cosmetic touches
* Use zip instead of manually listing the keys
* Fix inverse_transform
* FIX bug in fit_grid_point; Allow only single score
TST if fit_grid_point works as intended
* ENH Use only ROC-AUC and F1-score
* Fix typos and flake8; Address Andy's reviews
MNT Add a comment on why we do such a transpose + some fixes
* ENH Better error messages for incorrect multimetric scoring values +...
ENH Avoid exception traceback while using incorrect scoring string
* Dict keys must be of string type only
* 1. Better error message for invalid scoring 2...
Internal functions return single score for single metric scoring
* Fix test failures and shuffle tests
* Avoid wrapping scorer as dict in learning_curve
* Remove doc example as asked for
* Some leftover ones
* Don't wrap scorer in validation_curve either
* Add a doc example and skip it as dict order fails doctest
* Import zip from six for python2.7 compat
* Make cross_val_score return a cv_results-like dict
* Add relevant sections to userguide
* Flake8 fixes
* Add whatsnew and fix broken links
* Use AUC and accuracy instead of f1
* Fix failing doctests cross_validation.rst
* DOC add the wrapper example for metrics that return multiple return values
* Address andy's comments
* Be less weird
* Address more of andy's comments
* Make a separate cross_validate function to return dict and a cross_val_score
* Update the docs to reflect the new cross_validate function
* Add cross_validate to toc-tree
* Add more tests on type of cross_validate return and time limits
* FIX failing doctests
* FIX ensure keys are not plural
* DOC fix
* Address some pending comments
* Remove the comment as it is irrelevant now
* Remove excess blank line
* Fix flake8 inconsistencies
* Allow fit_times to be 0 to conform with windows precision
* DOC specify how refit param is to be set in multiple metric case
* TST ensure cross_validate works for string single metrics + address @jnothman's reviews
* Doc fixes
* Remove the shape and transform parameter of _aggregate_score_dicts
* Address Joel's doc comments
* Fix broken doctest
* Fix the spurious file
* Address Andy's comments
* MNT Remove erroneous entry
* Address Andy's comments
* FIX broken links
* Update whats_new.rst
missing newline
* Add _RepeatedSplits and RepeatedKFold class
* Add RepeatedStratifiedKFold and doc for repeated cvs
* Change default value of n_repeats
* Change input parameters of repeated cv constructor to n_splits, n_repeats, random_state
* Generate random states in split function rather than store it beforehand
* Doc changes, inheriting RepeatedKFold, RepeatedStratifiedKFold from _RepeatedSplits and other review changes
* Remove blank line, put testcases for deterministic split in loop and add StopIteration check in testcase
* Using rng directly as random_state param to create cv instance and added a check for cvargs
* Fix pep8 warnings
* Changing default values for n_splits and n_repeats and add entry in changelog
* Adding name to the feature
* Missing space
siavrez