Reasoning: when eg OneHotEncoder is used as part of ColumnTransformer it would cause the final result to be a sparse matrix. As this is a typical case when we have mixed dtype data, it means that many pipeline will have to deal with sparse data implicitly, even if you have only some categorical features with low cardinality.
Idea was first to change default of `OneHotEncoder` sparse to False, but based on gitter discussion (https://gitter.im/scikit-learn/dev?at=5b4e5a69a94c5255523bc9fc) we decided to let ColumnTransformer switch between both based on a threshold. The user still has full control if he/she wants always or never sparse.
* add FutureWarning for methods with defaults=3
* add explicit cv values to fix assertion errors
* add tests for catching the FutureWarning
* Write current deprecation version
* Add deprecation in docstring
* change default cv value to None
* change cv from 3 to 5 in the examples
* upgrade doctests
* update doctest in tutorial
* update doctest in cross-validation doc
* fix tests
* add entry to whats new
* address Gael comments
* address Gael comments 2
* fix wrong indentation
* update doc
* add docstring deprecation warning in CV subclasses
* address Andy's comments
* fix PR number
* fix flake8
* add filterwarnings in tests
* fix doctests
* cv=None mendatory in Ridge
* fix warning related errors
* skip some doctests warnings
* make travis happy
* change from deprecated to versionchanged
* fix doctests and remove skipping
* address comments
In order to fix#11408, this swaps `joblib` and `_joblib`. It however, allows users to access joblib's `Memory` or `Parallel` functionality without accessing `sklearn.externals._joblib` by importing `Memory`, `Parallel`, etc. into `sklearn.utils`.
* Add averaging option to AMI and NMI
Leave current behavior unchanged
* Flake8 fixes
* Incorporate tests of means for AMI and NMI
* Add note about `average_method` in NMI
* Update docs from AMI, NMI changes (#1)
* Correct the NMI and AMI descriptions in docs
* Update docstrings due to averaging changes
- V-measure
- Homogeneity
- Completeness
- NMI
- AMI
* Update documentation and remove nose tests (#2)
* Update v0.20.rst
* Update test_supervised.py
* Update clustering.rst
* Fix multiple spaces after operator
* Rename all arguments
* No more arbitrary values!
* Improve handling of floating-point imprecision
* Clearly state when the change occurs
* Update AMI/NMI docs
* Update v0.20.rst
* Catch FutureWarnings in AMI and NMI
MissingIndicator transformer for the missing values indicator mask.
see #6556
#### What does this implement/fix? Explain your changes.
The current implementation returns a indicator mask for the missing values.
#### Any other comments?
It is a very initial attempt and currently no tests are present. Please do have a look and give suggestions on the design. Thanks !
- [X] Implementation
- [x] Documentation
- [x] Tests
* OPTICS clustering algorithm
Equivalent results to DBSCAN, but allows execution on arbitrarily large datasets. After initial construction, allows multiple 'scans' to quickly extract DBSCAN clusters at variable epsilon distances
* Create plot_optics
Shows example usage of OPTICS to extract clustering structure
* pep8 fixes
Mainly issues in long lines, long comments
* fixed conditional to be pep8
* updated to match sklearn API
OPTICS object, with fit method. Documentation updates.
extract method
* removed extra files
* plotting example updated, small changes
new plot example that matches the updated API
updated n_cluster attribute with reruns of extract
removed scaling factor on first ‘fit’ run
* updated OPTICS.labels to OPTICS.labels_
should pass unit test now?
* additional labels_ changes
* added stability warning
Scales eps to give stable results from first input distance. Extraction
above scaled eps is not allowed; extraction below scaled eps but
greater than input distance prints stability warning but still will run
clustering extraction. All distances below initial input distance are
stable; no warning printed
* Noise fix; updated example plot
Fixed noise points from being initialized as type ‘core point’
Fixed initialization for first ‘fit’ call
Decoupled eps and eps_prime (deep copy)
Matched plot example to same random state as dbscan
Added second plot to show ‘extract’ usage
* Changed to match Sklearn API
eps is not modified in init; kwargs fix
* Forcing 2 parameters
Why do I have to do this?
* Conforming to API
Fit now returns self; test fix for unit test fail in ball tree (asarray
problem)
* Fixing plot example
labels to labels_
* Fixed issue with sparse matrices
* Another attempt at fixing the sparse matrix error
Temporary fix until balltree can be updated to deal with sparse
matrices.
* Better checking of sparse arrays
Using ‘check_array’
* General cleanup
Removed extraneous lines and comments (old commented out code has been
removed)
* Added unit tests for extract function
Added the unit tests in test_optics for fit, extract, and declaration.
Should bring coverage to ~100%. Additionally, fixed a small bug that
cropped up in the extract function during testing.
* Attempting for near 100% coverage
Removed unused imports, added to get warning.
* Fixed error in unit tests
* Trimmed extraneous 'if-else' check
see title
* forcing to check for a warning.
Result should be 100% coverage
* Updates to doc strings
All public methods now have doc strings; forcing a rebuild of OPTICS so
that build tests pass (last round failed due to external module)
* Style / pep8 changes
99% pep8 now… line 138 isn’t, but reads better with the long variable
names
* Added Narrative Documentation
Includes general description, discussion of algorithm output, and
comparison with DBSCAN. References and implementation notes are included
* Vectorized nneighbors lookups
Following suggestion from jnothman for doing nneighbors queries enmass.
Added OPTICS to cluster init
* fixing init build error
* reverting init
* All code now vectorized
…at least all code that can be ;)
Some general pruning and cleanup as well
* Style changes
Now 100% pep8
* Changing parameter style
matching DBSCAN
* Extraction change; Authors update
—initialize all points as ‘not core’ (fixes bug when plotting at
epsPrime larger than eps)
—added Sean Freeman to authors list
* Changed eps scaling to 5x instead of 10x
10x scaling is too conservative…eps scaling at 5x is perfectly stable,
and much faster as well.
* Fixing unit test
Should be ‘None’ for this; initialization previously at 1 for ‘is_core’
was incorrect
* Actually fixing unit test
Null comparison doesn’t work, using size
* Making ordering_ and other attributes public
renamed core_samples to core_sample_indices_ (as in DBSCAN). Used
attribute naming conventions (trailing _ character), and made
ordering_, reachability_, and core_dists_ public.
* Updates for Documentation
includes attribute strings for now public attributes
* Pep8 cleanup
Minor pep8 and pyflakes fixes
* updating plot example to match new attribute name
* CamelCase fixes
conforming to sklearn API on CamelCase
* adding hierarchical extraction function
Added hierarchical extraction from #2043#2043 is BSD licensed and hasn’t had any activity for 10 months, so
this seems pretty kosher; authors are cited in the code
* added hierarchical switch to extract
Additional style and documentation changes as well
* cluster order bug fix
ensured that ordered list is always same length as input data
* removed hierarchical cluster extraction
Code from FredrikAppelros is totally unable to handle noise— as
currently written every point is assigned to a cluster, except the
first point of that cluster. May include later as a third method
* initial import of automatic cluster extraction code
Adding the excellent (and working) automatic cluster extraction code
from Amy X. Zhang. Some minor formatting changes on import to conform
to pep8
* wrapper for 'auto' extraction
additional fixes to style (camelCase, etc.), comments; made all helper
functions private
* test and example updates
Much better example data to showcase ‘auto’ extraction. Unit tests now
test both extraction methods. Set ‘auto’ as default, as it doesn’t
require any parameters and gives a better result. Pruned references to
hierarchical clustering
* fixing unit coverage; pruning unused functions
probably could still get a better unit test for ‘auto’ clustering…
* Added 'filter' fuction
Allows density-based filtering. Useful for cases where only a
‘noise’/‘no noise’ classification is desired. Function does not require
fitting prior to running, although it can be run after a fit if desired
* Vectorizing auto_cluster
generalizes input to multiple dimensions as well…
* updated filter function
* fixing test error
setting minPts to < data size returns None (with error message)
* removing exception handling in favor of conditional check
* Updated unit tests
Coverage to 90%. PEP8 fixes
* Additional unit test
Now at 94%; auto extract method is very hard to test… this new unit
test adds a more robust dataset to trigger more branches for testing.
Some of the remaining conditionals are pretty rare … :-/
* Fix unit test bug / python 3 compat
None type comparison problem…
* Fixed annoying deprecation warning for 1d array in BallTree
PEP8 Fixes
* More PEP8 and remove print statements fromt est
* 70 to 80% faster, fixed distance metrics
Modified to remove extraneous sorts, nneighbors query, and reduced
pairwise distance calculations to the upper triangle of a distance
matrix (instead of the full matrix). It appears that in ‘full’ scans
(i.e., when epsilon is set to inf, or the width of the data set),
OPTICS now actually outperforms DBSCAN… also, should be easy to run
distance calculations in parallel now for large datasets, with proper
heuristics.
* Fixing unit test failure
* Exposed auto_cluster parameters as public
documentation and API update so that users can tweek the auto method
* Fixed def with missing ':'
* Fix bugs from api change...
make sure that arguments are being called correctly with the new
extract_auto() method
* pep8 / pyflakes changes
* Updated example / plot
Correctly generates figure with subplots for DBSCAN / OPTICS comparison
* Tuning plot example / pep8 change
* Bug fix for commit 0b4cbdd (enforce stable sort)
the returned index from “sp.argmin(setofobjects.reachability_[n_pr])”
assumes that entires are stably sorted by distance from query point
(i.e., that ties in the argmin return the closest point to the input
point).
Still overall faster compared to the pre 0b4cbdd version, since we’re
filtering processed points before sorting by distance… and also only
calculating distances for non-processed points, instead of all within
the epsilon query.
* Code review fixes (style)
Fixes coding style (test comments, camel case, author list, relative
imports, etc). Included a new unit test and .npy file to explicitly
test reachability distances (test coverage at 99%).
* fixing new unit test
can’t import reach_values.npy in current directory….just placed testing
values in script directly (~200 lines for 1500 testing values)
* refactored min_heap to c extension
reduced optics file by about 100 lines of code… new c extension for
speedup (needs further optimization…6-10X speedup still possible)
* minor fixes..
* small cython optimizations
* cython fixes
* Add foo.txt
* Remove foo.txt
* fix compilation error
* last optimizations cython/numpy
MinHeap is only called once…so it’s faster to do a simple linear scan
here. The np.argmin() function does *almost* the exact same thing, but
the custom quick scan function is needed for cases where reachability
distances are tied (and the next point is selected based on which of
the points tied in reachability are closest to the querying point).
OPTICS is now faster than DBSCAN for medium-to-large number of input
points, and has better worst case run time (eps=inf).
* fix pyflakes errors; change default eps value
* _
* API changes from agramfort
Removed ‘filter method’, changed print statements to exceptions and
warnings as needed, remove ‘processed’ flag and replaced with fit_check
method, changed array copies to parameters, updated unit tests, changed
inline documentation to match proper doc string formatting, made
private class private. Probably some other changes too…
* fixed core samples bug
* added fit_predict
conforming to scipy api
* updates to variable names; update plot
* refactor to remove balltree specific code
* major refactor
finished decoupling balltree; lots of changes
* fixed bugs; test all pass again
:-)
* fixed weird cython bug
…not at all sure why pairwise_distances doesn’t automatically return
np.float arrays. Makes no sense to me. I could understand cases in
which the metric call returns int’s (i.e., city block)… but why it
would return float.32 instead of float.64 seems super odd.
* major refactor
deleted extract and auto_extract methods; added optics function; added
extract_dbscan method; added extract_dbscan and extract_optics
functions; updated unit tests; renamed `eps` to `max_bounds`; flake8
corrections; added types: int —> labels, bool —> is_core, enforced X to
be ‘float’ (fixes cython type errors)
* Updated Documentation!
Updated plot with reachability plot, as well as documentation :)
* fix flake8 error
* added optics to cluster comparison
Don't like the figure since it doesn't use black for noise, but added OPTICS for consistency
* Updated comparison plot to transpose
...kinda kludgy fix for the transpose
* small fix
* flake8 error
* reverting transpose of cluster comparison (seperate PR #9739)
* fixes from agramfort's review
public/private changes, numpydocstring fixes, a few pep8 fixes that flake8 didn't flag for some reason...
* fix for error message
unit test should pass now
* force cluster_id's to start at 0
* Fix sp. error and flake8 warning(s)
* Updated documentation
responses to reviews
* Removed extraneous files
also fixed small typo
* fixing lgtm alert
* changes from jnothman
small fixes for docs, plots, and tests
* Fixes from jnothman's review
Reorded parameters, updated documentation, removed unneeded else statement, changes to varible names.
* Fixing flake8 error
* Removed neighbors / balltree inheritance
Also decouples n_jobs -- can set n_jobs for just the kneighbors lookup, while keeping pairwise lookups to single job
* Made nbrs private and moved initiation to fit()
also renamed core_dists to core_distances.
* fixed non-standard characters
* Response to TomDLT review
narrative changes to tests. Normalized reachability distances for significant_min parameter. Cleaned up plots; small changes to documentation with optics_.py
* Fixed labeling bug
also minor documentation updates
* update unit test
since labels are 0 indexed, max of labels is (1 - total number of clusters). We can't take len(set(clusters)) because of noise (will be 4, not 3).
* Simple fixes per jnothman
Fixed float division, condensed variable names to be shorter, renamed bools to True and False, removed un-needed code block. Still need to add tests for cluster tree extraction :-(
* Auto-cluster tests
coverage should be complete now; fixed minor bug; removed un-needed check.
* fixing test error
* removed python loop
also fixed documentation link
* Fixing test error
* Fix typo in unit test
entry was supposed to be '1.0' not '10'; the test is supposed to posit 3 clusters, 2 of which are too small and are merged. Old version posited 4 clusters, two of which were merged as intended, and two of which were discarded (cluster merging requires one of the clusters to be large enough to be an independent cluster; with 4 instead of three, this case did not happen for either of the first two clusters).
* documentation updates
* Post-merge doctest fix
merge conflict in clustering.rst in previous commit; this push updates the doctest values to current correct values, and resolves the conflict
* DBSCAN / OPTICS invariant test
Restructured documentation. Small unit test fixes. Added test to ensure clustering metrics between OPTICS dbscan_extract and DBSCAN are within 2% of each other.
* Update _auto_cluster docstring
renamed reachability_ordering --> ordering for consistency
* changes fro jnothman
changes unit tests to check for specific error message (instead of 'a' error); minor updates to documentation. This also fixes a bug in the extract_dbscan function whereby some core points were erroneously marked as periphery... this is fixed by reverting to a previous extraction code block that initalizes all points as core and then demotes noise and periphery points during the extract scan. Parameterized unit test.
* fix spelling error in tests
* contingency_matrix test
New invarient test between optics and dbscan
* small unit test updates per jnothman
* unit test typo fix
* extract dbscan updates
Vectorized extract dbscan function to see if would improve performance of periphery point labeling; it did not, but the function is vectorized. Changed unit test with min_samples=1 to min_samples=3, as at min_samples=1 the test isn't meaningful (no noise is possible, all points are marked core). Parameterized parity test. Changed parity test to assert ~5% or better mismatch, instead of 5 points (this is needed for larger clusters, as the starting point mismatch effect scales with cluster size).
* updated documentation comparing OPTICS/DBSCAN
* DOC: phrasing and whats_new
* MISC: small mem footprint in OPTICS
The subscript of the Huber function (H_m) was not relevant:
it was written H_m but the index m was nowhere used later; it should have been epsilon instead.
* ENH: Add positivity option for code and dictionary
Provides an option for dictionary learning to positively constrain the
dictionary and the sparse code. This is useful in applications of
dictionary learning where the data is know to be positive (e.g. images),
but the sparsity constraint that dictionary learning has is better
suited for factorizing the data in contrast to other positively
constrained factorization techniques like NMF, which may not be
similarly sparse.
* TST: Test positivity with code and dictionary
Ensure that when the positivity constraint is applied that the
dictionary and code end up having only positive values in the respective
results depending on whether dictionary and/or code are positively
constrained.
* DOC: Positivity constraints dictionary learning
Shows the various positivity constraints on dictionary learning and what
the results of these look like using a Red to Blue color map. These are
included in the examples and also in the docs below dictionary learning.
All of these use the Olivetti faces as a training set.
Markus Müller
Hanmin Qin