scikit-learn/examples/cluster/plot_optics.py

102 lines
3.1 KiB
Python
Raw Normal View History

OPTICS (#11547) * OPTICS clustering algorithm Equivalent results to DBSCAN, but allows execution on arbitrarily large datasets. After initial construction, allows multiple 'scans' to quickly extract DBSCAN clusters at variable epsilon distances * Create plot_optics Shows example usage of OPTICS to extract clustering structure * pep8 fixes Mainly issues in long lines, long comments * fixed conditional to be pep8 * updated to match sklearn API OPTICS object, with fit method. Documentation updates. extract method * removed extra files * plotting example updated, small changes new plot example that matches the updated API updated n_cluster attribute with reruns of extract removed scaling factor on first ‘fit’ run * updated OPTICS.labels to OPTICS.labels_ should pass unit test now? * additional labels_ changes * added stability warning Scales eps to give stable results from first input distance. Extraction above scaled eps is not allowed; extraction below scaled eps but greater than input distance prints stability warning but still will run clustering extraction. All distances below initial input distance are stable; no warning printed * Noise fix; updated example plot Fixed noise points from being initialized as type ‘core point’ Fixed initialization for first ‘fit’ call Decoupled eps and eps_prime (deep copy) Matched plot example to same random state as dbscan Added second plot to show ‘extract’ usage * Changed to match Sklearn API eps is not modified in init; kwargs fix * Forcing 2 parameters Why do I have to do this? * Conforming to API Fit now returns self; test fix for unit test fail in ball tree (asarray problem) * Fixing plot example labels to labels_ * Fixed issue with sparse matrices * Another attempt at fixing the sparse matrix error Temporary fix until balltree can be updated to deal with sparse matrices. * Better checking of sparse arrays Using ‘check_array’ * General cleanup Removed extraneous lines and comments (old commented out code has been removed) * Added unit tests for extract function Added the unit tests in test_optics for fit, extract, and declaration. Should bring coverage to ~100%. Additionally, fixed a small bug that cropped up in the extract function during testing. * Attempting for near 100% coverage Removed unused imports, added to get warning. * Fixed error in unit tests * Trimmed extraneous 'if-else' check see title * forcing to check for a warning. Result should be 100% coverage * Updates to doc strings All public methods now have doc strings; forcing a rebuild of OPTICS so that build tests pass (last round failed due to external module) * Style / pep8 changes 99% pep8 now… line 138 isn’t, but reads better with the long variable names * Added Narrative Documentation Includes general description, discussion of algorithm output, and comparison with DBSCAN. References and implementation notes are included * Vectorized nneighbors lookups Following suggestion from jnothman for doing nneighbors queries enmass. Added OPTICS to cluster init * fixing init build error * reverting init * All code now vectorized …at least all code that can be ;) Some general pruning and cleanup as well * Style changes Now 100% pep8 * Changing parameter style matching DBSCAN * Extraction change; Authors update —initialize all points as ‘not core’ (fixes bug when plotting at epsPrime larger than eps) —added Sean Freeman to authors list * Changed eps scaling to 5x instead of 10x 10x scaling is too conservative…eps scaling at 5x is perfectly stable, and much faster as well. * Fixing unit test Should be ‘None’ for this; initialization previously at 1 for ‘is_core’ was incorrect * Actually fixing unit test Null comparison doesn’t work, using size * Making ordering_ and other attributes public renamed core_samples to core_sample_indices_ (as in DBSCAN). Used attribute naming conventions (trailing _ character), and made ordering_, reachability_, and core_dists_ public. * Updates for Documentation includes attribute strings for now public attributes * Pep8 cleanup Minor pep8 and pyflakes fixes * updating plot example to match new attribute name * CamelCase fixes conforming to sklearn API on CamelCase * adding hierarchical extraction function Added hierarchical extraction from #2043 #2043 is BSD licensed and hasn’t had any activity for 10 months, so this seems pretty kosher; authors are cited in the code * added hierarchical switch to extract Additional style and documentation changes as well * cluster order bug fix ensured that ordered list is always same length as input data * removed hierarchical cluster extraction Code from FredrikAppelros is totally unable to handle noise— as currently written every point is assigned to a cluster, except the first point of that cluster. May include later as a third method * initial import of automatic cluster extraction code Adding the excellent (and working) automatic cluster extraction code from Amy X. Zhang. Some minor formatting changes on import to conform to pep8 * wrapper for 'auto' extraction additional fixes to style (camelCase, etc.), comments; made all helper functions private * test and example updates Much better example data to showcase ‘auto’ extraction. Unit tests now test both extraction methods. Set ‘auto’ as default, as it doesn’t require any parameters and gives a better result. Pruned references to hierarchical clustering * fixing unit coverage; pruning unused functions probably could still get a better unit test for ‘auto’ clustering… * Added 'filter' fuction Allows density-based filtering. Useful for cases where only a ‘noise’/‘no noise’ classification is desired. Function does not require fitting prior to running, although it can be run after a fit if desired * Vectorizing auto_cluster generalizes input to multiple dimensions as well… * updated filter function * fixing test error setting minPts to < data size returns None (with error message) * removing exception handling in favor of conditional check * Updated unit tests Coverage to 90%. PEP8 fixes * Additional unit test Now at 94%; auto extract method is very hard to test… this new unit test adds a more robust dataset to trigger more branches for testing. Some of the remaining conditionals are pretty rare … :-/ * Fix unit test bug / python 3 compat None type comparison problem… * Fixed annoying deprecation warning for 1d array in BallTree PEP8 Fixes * More PEP8 and remove print statements fromt est * 70 to 80% faster, fixed distance metrics Modified to remove extraneous sorts, nneighbors query, and reduced pairwise distance calculations to the upper triangle of a distance matrix (instead of the full matrix). It appears that in ‘full’ scans (i.e., when epsilon is set to inf, or the width of the data set), OPTICS now actually outperforms DBSCAN… also, should be easy to run distance calculations in parallel now for large datasets, with proper heuristics. * Fixing unit test failure * Exposed auto_cluster parameters as public documentation and API update so that users can tweek the auto method * Fixed def with missing ':' * Fix bugs from api change... make sure that arguments are being called correctly with the new extract_auto() method * pep8 / pyflakes changes * Updated example / plot Correctly generates figure with subplots for DBSCAN / OPTICS comparison * Tuning plot example / pep8 change * Bug fix for commit 0b4cbdd (enforce stable sort) the returned index from “sp.argmin(setofobjects.reachability_[n_pr])” assumes that entires are stably sorted by distance from query point (i.e., that ties in the argmin return the closest point to the input point). Still overall faster compared to the pre 0b4cbdd version, since we’re filtering processed points before sorting by distance… and also only calculating distances for non-processed points, instead of all within the epsilon query. * Code review fixes (style) Fixes coding style (test comments, camel case, author list, relative imports, etc). Included a new unit test and .npy file to explicitly test reachability distances (test coverage at 99%). * fixing new unit test can’t import reach_values.npy in current directory….just placed testing values in script directly (~200 lines for 1500 testing values) * refactored min_heap to c extension reduced optics file by about 100 lines of code… new c extension for speedup (needs further optimization…6-10X speedup still possible) * minor fixes.. * small cython optimizations * cython fixes * Add foo.txt * Remove foo.txt * fix compilation error * last optimizations cython/numpy MinHeap is only called once…so it’s faster to do a simple linear scan here. The np.argmin() function does *almost* the exact same thing, but the custom quick scan function is needed for cases where reachability distances are tied (and the next point is selected based on which of the points tied in reachability are closest to the querying point). OPTICS is now faster than DBSCAN for medium-to-large number of input points, and has better worst case run time (eps=inf). * fix pyflakes errors; change default eps value * _ * API changes from agramfort Removed ‘filter method’, changed print statements to exceptions and warnings as needed, remove ‘processed’ flag and replaced with fit_check method, changed array copies to parameters, updated unit tests, changed inline documentation to match proper doc string formatting, made private class private. Probably some other changes too… * fixed core samples bug * added fit_predict conforming to scipy api * updates to variable names; update plot * refactor to remove balltree specific code * major refactor finished decoupling balltree; lots of changes * fixed bugs; test all pass again :-) * fixed weird cython bug …not at all sure why pairwise_distances doesn’t automatically return np.float arrays. Makes no sense to me. I could understand cases in which the metric call returns int’s (i.e., city block)… but why it would return float.32 instead of float.64 seems super odd. * major refactor deleted extract and auto_extract methods; added optics function; added extract_dbscan method; added extract_dbscan and extract_optics functions; updated unit tests; renamed `eps` to `max_bounds`; flake8 corrections; added types: int —> labels, bool —> is_core, enforced X to be ‘float’ (fixes cython type errors) * Updated Documentation! Updated plot with reachability plot, as well as documentation :) * fix flake8 error * added optics to cluster comparison Don't like the figure since it doesn't use black for noise, but added OPTICS for consistency * Updated comparison plot to transpose ...kinda kludgy fix for the transpose * small fix * flake8 error * reverting transpose of cluster comparison (seperate PR #9739) * fixes from agramfort's review public/private changes, numpydocstring fixes, a few pep8 fixes that flake8 didn't flag for some reason... * fix for error message unit test should pass now * force cluster_id's to start at 0 * Fix sp. error and flake8 warning(s) * Updated documentation responses to reviews * Removed extraneous files also fixed small typo * fixing lgtm alert * changes from jnothman small fixes for docs, plots, and tests * Fixes from jnothman's review Reorded parameters, updated documentation, removed unneeded else statement, changes to varible names. * Fixing flake8 error * Removed neighbors / balltree inheritance Also decouples n_jobs -- can set n_jobs for just the kneighbors lookup, while keeping pairwise lookups to single job * Made nbrs private and moved initiation to fit() also renamed core_dists to core_distances. * fixed non-standard characters * Response to TomDLT review narrative changes to tests. Normalized reachability distances for significant_min parameter. Cleaned up plots; small changes to documentation with optics_.py * Fixed labeling bug also minor documentation updates * update unit test since labels are 0 indexed, max of labels is (1 - total number of clusters). We can't take len(set(clusters)) because of noise (will be 4, not 3). * Simple fixes per jnothman Fixed float division, condensed variable names to be shorter, renamed bools to True and False, removed un-needed code block. Still need to add tests for cluster tree extraction :-( * Auto-cluster tests coverage should be complete now; fixed minor bug; removed un-needed check. * fixing test error * removed python loop also fixed documentation link * Fixing test error * Fix typo in unit test entry was supposed to be '1.0' not '10'; the test is supposed to posit 3 clusters, 2 of which are too small and are merged. Old version posited 4 clusters, two of which were merged as intended, and two of which were discarded (cluster merging requires one of the clusters to be large enough to be an independent cluster; with 4 instead of three, this case did not happen for either of the first two clusters). * documentation updates * Post-merge doctest fix merge conflict in clustering.rst in previous commit; this push updates the doctest values to current correct values, and resolves the conflict * DBSCAN / OPTICS invariant test Restructured documentation. Small unit test fixes. Added test to ensure clustering metrics between OPTICS dbscan_extract and DBSCAN are within 2% of each other. * Update _auto_cluster docstring renamed reachability_ordering --> ordering for consistency * changes fro jnothman changes unit tests to check for specific error message (instead of 'a' error); minor updates to documentation. This also fixes a bug in the extract_dbscan function whereby some core points were erroneously marked as periphery... this is fixed by reverting to a previous extraction code block that initalizes all points as core and then demotes noise and periphery points during the extract scan. Parameterized unit test. * fix spelling error in tests * contingency_matrix test New invarient test between optics and dbscan * small unit test updates per jnothman * unit test typo fix * extract dbscan updates Vectorized extract dbscan function to see if would improve performance of periphery point labeling; it did not, but the function is vectorized. Changed unit test with min_samples=1 to min_samples=3, as at min_samples=1 the test isn't meaningful (no noise is possible, all points are marked core). Parameterized parity test. Changed parity test to assert ~5% or better mismatch, instead of 5 points (this is needed for larger clusters, as the starting point mismatch effect scales with cluster size). * updated documentation comparing OPTICS/DBSCAN * DOC: phrasing and whats_new * MISC: small mem footprint in OPTICS
2018-07-16 19:10:42 +08:00
"""
===================================
Demo of OPTICS clustering algorithm
===================================
Finds core samples of high density and expands clusters from them.
This example uses data that is generated so that the clusters have
different densities.
The clustering is first used in its automatic settings, which is the
:class:`sklearn.cluster.OPTICS` algorithm, and then setting specific
thresholds on the reachability, which corresponds to DBSCAN.
We can see that the different clusters of OPTICS can be recovered with
different choices of thresholds in DBSCAN.
"""
# Authors: Shane Grigsby <refuge@rocktalus.com>
# Amy X. Zhang <axz@mit.edu>
# License: BSD 3 clause
from sklearn.cluster import OPTICS
import matplotlib.gridspec as gridspec
import numpy as np
import matplotlib.pyplot as plt
# Generate sample data
np.random.seed(0)
n_points_per_cluster = 250
C1 = [-5, -2] + .8 * np.random.randn(n_points_per_cluster, 2)
C2 = [4, -1] + .1 * np.random.randn(n_points_per_cluster, 2)
C3 = [1, -2] + .2 * np.random.randn(n_points_per_cluster, 2)
C4 = [-2, 3] + .3 * np.random.randn(n_points_per_cluster, 2)
C5 = [3, -2] + 1.6 * np.random.randn(n_points_per_cluster, 2)
C6 = [5, 6] + 2 * np.random.randn(n_points_per_cluster, 2)
X = np.vstack((C1, C2, C3, C4, C5, C6))
clust = OPTICS(min_samples=9, rejection_ratio=0.5)
# Run the fit
clust.fit(X)
_, labels_025 = clust.extract_dbscan(0.25)
_, labels_075 = clust.extract_dbscan(0.75)
space = np.arange(len(X))
reachability = clust.reachability_[clust.ordering_]
labels = clust.labels_[clust.ordering_]
plt.figure(figsize=(10, 7))
G = gridspec.GridSpec(2, 3)
ax1 = plt.subplot(G[0, :])
ax2 = plt.subplot(G[1, 0])
ax3 = plt.subplot(G[1, 1])
ax4 = plt.subplot(G[1, 2])
# Reachability plot
color = ['g.', 'r.', 'b.', 'y.', 'c.']
for k, c in zip(range(0, 5), color):
Xk = space[labels == k]
Rk = reachability[labels == k]
ax1.plot(Xk, Rk, c, alpha=0.3)
ax1.plot(space[labels == -1], reachability[labels == -1], 'k.', alpha=0.3)
ax1.plot(space, np.full_like(space, 0.75, dtype=float), 'k-', alpha=0.5)
ax1.plot(space, np.full_like(space, 0.25, dtype=float), 'k-.', alpha=0.5)
OPTICS (#11547) * OPTICS clustering algorithm Equivalent results to DBSCAN, but allows execution on arbitrarily large datasets. After initial construction, allows multiple 'scans' to quickly extract DBSCAN clusters at variable epsilon distances * Create plot_optics Shows example usage of OPTICS to extract clustering structure * pep8 fixes Mainly issues in long lines, long comments * fixed conditional to be pep8 * updated to match sklearn API OPTICS object, with fit method. Documentation updates. extract method * removed extra files * plotting example updated, small changes new plot example that matches the updated API updated n_cluster attribute with reruns of extract removed scaling factor on first ‘fit’ run * updated OPTICS.labels to OPTICS.labels_ should pass unit test now? * additional labels_ changes * added stability warning Scales eps to give stable results from first input distance. Extraction above scaled eps is not allowed; extraction below scaled eps but greater than input distance prints stability warning but still will run clustering extraction. All distances below initial input distance are stable; no warning printed * Noise fix; updated example plot Fixed noise points from being initialized as type ‘core point’ Fixed initialization for first ‘fit’ call Decoupled eps and eps_prime (deep copy) Matched plot example to same random state as dbscan Added second plot to show ‘extract’ usage * Changed to match Sklearn API eps is not modified in init; kwargs fix * Forcing 2 parameters Why do I have to do this? * Conforming to API Fit now returns self; test fix for unit test fail in ball tree (asarray problem) * Fixing plot example labels to labels_ * Fixed issue with sparse matrices * Another attempt at fixing the sparse matrix error Temporary fix until balltree can be updated to deal with sparse matrices. * Better checking of sparse arrays Using ‘check_array’ * General cleanup Removed extraneous lines and comments (old commented out code has been removed) * Added unit tests for extract function Added the unit tests in test_optics for fit, extract, and declaration. Should bring coverage to ~100%. Additionally, fixed a small bug that cropped up in the extract function during testing. * Attempting for near 100% coverage Removed unused imports, added to get warning. * Fixed error in unit tests * Trimmed extraneous 'if-else' check see title * forcing to check for a warning. Result should be 100% coverage * Updates to doc strings All public methods now have doc strings; forcing a rebuild of OPTICS so that build tests pass (last round failed due to external module) * Style / pep8 changes 99% pep8 now… line 138 isn’t, but reads better with the long variable names * Added Narrative Documentation Includes general description, discussion of algorithm output, and comparison with DBSCAN. References and implementation notes are included * Vectorized nneighbors lookups Following suggestion from jnothman for doing nneighbors queries enmass. Added OPTICS to cluster init * fixing init build error * reverting init * All code now vectorized …at least all code that can be ;) Some general pruning and cleanup as well * Style changes Now 100% pep8 * Changing parameter style matching DBSCAN * Extraction change; Authors update —initialize all points as ‘not core’ (fixes bug when plotting at epsPrime larger than eps) —added Sean Freeman to authors list * Changed eps scaling to 5x instead of 10x 10x scaling is too conservative…eps scaling at 5x is perfectly stable, and much faster as well. * Fixing unit test Should be ‘None’ for this; initialization previously at 1 for ‘is_core’ was incorrect * Actually fixing unit test Null comparison doesn’t work, using size * Making ordering_ and other attributes public renamed core_samples to core_sample_indices_ (as in DBSCAN). Used attribute naming conventions (trailing _ character), and made ordering_, reachability_, and core_dists_ public. * Updates for Documentation includes attribute strings for now public attributes * Pep8 cleanup Minor pep8 and pyflakes fixes * updating plot example to match new attribute name * CamelCase fixes conforming to sklearn API on CamelCase * adding hierarchical extraction function Added hierarchical extraction from #2043 #2043 is BSD licensed and hasn’t had any activity for 10 months, so this seems pretty kosher; authors are cited in the code * added hierarchical switch to extract Additional style and documentation changes as well * cluster order bug fix ensured that ordered list is always same length as input data * removed hierarchical cluster extraction Code from FredrikAppelros is totally unable to handle noise— as currently written every point is assigned to a cluster, except the first point of that cluster. May include later as a third method * initial import of automatic cluster extraction code Adding the excellent (and working) automatic cluster extraction code from Amy X. Zhang. Some minor formatting changes on import to conform to pep8 * wrapper for 'auto' extraction additional fixes to style (camelCase, etc.), comments; made all helper functions private * test and example updates Much better example data to showcase ‘auto’ extraction. Unit tests now test both extraction methods. Set ‘auto’ as default, as it doesn’t require any parameters and gives a better result. Pruned references to hierarchical clustering * fixing unit coverage; pruning unused functions probably could still get a better unit test for ‘auto’ clustering… * Added 'filter' fuction Allows density-based filtering. Useful for cases where only a ‘noise’/‘no noise’ classification is desired. Function does not require fitting prior to running, although it can be run after a fit if desired * Vectorizing auto_cluster generalizes input to multiple dimensions as well… * updated filter function * fixing test error setting minPts to < data size returns None (with error message) * removing exception handling in favor of conditional check * Updated unit tests Coverage to 90%. PEP8 fixes * Additional unit test Now at 94%; auto extract method is very hard to test… this new unit test adds a more robust dataset to trigger more branches for testing. Some of the remaining conditionals are pretty rare … :-/ * Fix unit test bug / python 3 compat None type comparison problem… * Fixed annoying deprecation warning for 1d array in BallTree PEP8 Fixes * More PEP8 and remove print statements fromt est * 70 to 80% faster, fixed distance metrics Modified to remove extraneous sorts, nneighbors query, and reduced pairwise distance calculations to the upper triangle of a distance matrix (instead of the full matrix). It appears that in ‘full’ scans (i.e., when epsilon is set to inf, or the width of the data set), OPTICS now actually outperforms DBSCAN… also, should be easy to run distance calculations in parallel now for large datasets, with proper heuristics. * Fixing unit test failure * Exposed auto_cluster parameters as public documentation and API update so that users can tweek the auto method * Fixed def with missing ':' * Fix bugs from api change... make sure that arguments are being called correctly with the new extract_auto() method * pep8 / pyflakes changes * Updated example / plot Correctly generates figure with subplots for DBSCAN / OPTICS comparison * Tuning plot example / pep8 change * Bug fix for commit 0b4cbdd (enforce stable sort) the returned index from “sp.argmin(setofobjects.reachability_[n_pr])” assumes that entires are stably sorted by distance from query point (i.e., that ties in the argmin return the closest point to the input point). Still overall faster compared to the pre 0b4cbdd version, since we’re filtering processed points before sorting by distance… and also only calculating distances for non-processed points, instead of all within the epsilon query. * Code review fixes (style) Fixes coding style (test comments, camel case, author list, relative imports, etc). Included a new unit test and .npy file to explicitly test reachability distances (test coverage at 99%). * fixing new unit test can’t import reach_values.npy in current directory….just placed testing values in script directly (~200 lines for 1500 testing values) * refactored min_heap to c extension reduced optics file by about 100 lines of code… new c extension for speedup (needs further optimization…6-10X speedup still possible) * minor fixes.. * small cython optimizations * cython fixes * Add foo.txt * Remove foo.txt * fix compilation error * last optimizations cython/numpy MinHeap is only called once…so it’s faster to do a simple linear scan here. The np.argmin() function does *almost* the exact same thing, but the custom quick scan function is needed for cases where reachability distances are tied (and the next point is selected based on which of the points tied in reachability are closest to the querying point). OPTICS is now faster than DBSCAN for medium-to-large number of input points, and has better worst case run time (eps=inf). * fix pyflakes errors; change default eps value * _ * API changes from agramfort Removed ‘filter method’, changed print statements to exceptions and warnings as needed, remove ‘processed’ flag and replaced with fit_check method, changed array copies to parameters, updated unit tests, changed inline documentation to match proper doc string formatting, made private class private. Probably some other changes too… * fixed core samples bug * added fit_predict conforming to scipy api * updates to variable names; update plot * refactor to remove balltree specific code * major refactor finished decoupling balltree; lots of changes * fixed bugs; test all pass again :-) * fixed weird cython bug …not at all sure why pairwise_distances doesn’t automatically return np.float arrays. Makes no sense to me. I could understand cases in which the metric call returns int’s (i.e., city block)… but why it would return float.32 instead of float.64 seems super odd. * major refactor deleted extract and auto_extract methods; added optics function; added extract_dbscan method; added extract_dbscan and extract_optics functions; updated unit tests; renamed `eps` to `max_bounds`; flake8 corrections; added types: int —> labels, bool —> is_core, enforced X to be ‘float’ (fixes cython type errors) * Updated Documentation! Updated plot with reachability plot, as well as documentation :) * fix flake8 error * added optics to cluster comparison Don't like the figure since it doesn't use black for noise, but added OPTICS for consistency * Updated comparison plot to transpose ...kinda kludgy fix for the transpose * small fix * flake8 error * reverting transpose of cluster comparison (seperate PR #9739) * fixes from agramfort's review public/private changes, numpydocstring fixes, a few pep8 fixes that flake8 didn't flag for some reason... * fix for error message unit test should pass now * force cluster_id's to start at 0 * Fix sp. error and flake8 warning(s) * Updated documentation responses to reviews * Removed extraneous files also fixed small typo * fixing lgtm alert * changes from jnothman small fixes for docs, plots, and tests * Fixes from jnothman's review Reorded parameters, updated documentation, removed unneeded else statement, changes to varible names. * Fixing flake8 error * Removed neighbors / balltree inheritance Also decouples n_jobs -- can set n_jobs for just the kneighbors lookup, while keeping pairwise lookups to single job * Made nbrs private and moved initiation to fit() also renamed core_dists to core_distances. * fixed non-standard characters * Response to TomDLT review narrative changes to tests. Normalized reachability distances for significant_min parameter. Cleaned up plots; small changes to documentation with optics_.py * Fixed labeling bug also minor documentation updates * update unit test since labels are 0 indexed, max of labels is (1 - total number of clusters). We can't take len(set(clusters)) because of noise (will be 4, not 3). * Simple fixes per jnothman Fixed float division, condensed variable names to be shorter, renamed bools to True and False, removed un-needed code block. Still need to add tests for cluster tree extraction :-( * Auto-cluster tests coverage should be complete now; fixed minor bug; removed un-needed check. * fixing test error * removed python loop also fixed documentation link * Fixing test error * Fix typo in unit test entry was supposed to be '1.0' not '10'; the test is supposed to posit 3 clusters, 2 of which are too small and are merged. Old version posited 4 clusters, two of which were merged as intended, and two of which were discarded (cluster merging requires one of the clusters to be large enough to be an independent cluster; with 4 instead of three, this case did not happen for either of the first two clusters). * documentation updates * Post-merge doctest fix merge conflict in clustering.rst in previous commit; this push updates the doctest values to current correct values, and resolves the conflict * DBSCAN / OPTICS invariant test Restructured documentation. Small unit test fixes. Added test to ensure clustering metrics between OPTICS dbscan_extract and DBSCAN are within 2% of each other. * Update _auto_cluster docstring renamed reachability_ordering --> ordering for consistency * changes fro jnothman changes unit tests to check for specific error message (instead of 'a' error); minor updates to documentation. This also fixes a bug in the extract_dbscan function whereby some core points were erroneously marked as periphery... this is fixed by reverting to a previous extraction code block that initalizes all points as core and then demotes noise and periphery points during the extract scan. Parameterized unit test. * fix spelling error in tests * contingency_matrix test New invarient test between optics and dbscan * small unit test updates per jnothman * unit test typo fix * extract dbscan updates Vectorized extract dbscan function to see if would improve performance of periphery point labeling; it did not, but the function is vectorized. Changed unit test with min_samples=1 to min_samples=3, as at min_samples=1 the test isn't meaningful (no noise is possible, all points are marked core). Parameterized parity test. Changed parity test to assert ~5% or better mismatch, instead of 5 points (this is needed for larger clusters, as the starting point mismatch effect scales with cluster size). * updated documentation comparing OPTICS/DBSCAN * DOC: phrasing and whats_new * MISC: small mem footprint in OPTICS
2018-07-16 19:10:42 +08:00
ax1.set_ylabel('Reachability (epsilon distance)')
ax1.set_title('Reachability Plot')
# OPTICS
color = ['g.', 'r.', 'b.', 'y.', 'c.']
for k, c in zip(range(0, 5), color):
Xk = X[clust.labels_ == k]
ax2.plot(Xk[:, 0], Xk[:, 1], c, alpha=0.3)
ax2.plot(X[clust.labels_ == -1, 0], X[clust.labels_ == -1, 1], 'k+', alpha=0.1)
ax2.set_title('Automatic Clustering\nOPTICS')
# DBSCAN at 0.25
color = ['g', 'greenyellow', 'olive', 'r', 'b', 'c']
for k, c in zip(range(0, 6), color):
Xk = X[labels_025 == k]
ax3.plot(Xk[:, 0], Xk[:, 1], c, alpha=0.3, marker='.')
ax3.plot(X[labels_025 == -1, 0], X[labels_025 == -1, 1], 'k+', alpha=0.1)
ax3.set_title('Clustering at 0.25 epsilon cut\nDBSCAN')
# DBSCAN at 0.75
color = ['g.', 'm.', 'y.', 'c.']
for k, c in zip(range(0, 4), color):
Xk = X[labels_075 == k]
ax4.plot(Xk[:, 0], Xk[:, 1], c, alpha=0.3)
ax4.plot(X[labels_075 == -1, 0], X[labels_075 == -1, 1], 'k+', alpha=0.1)
ax4.set_title('Clustering at 0.75 epsilon cut\nDBSCAN')
plt.tight_layout()
plt.show()