Selecting the Number of Clusters $K$ with a Stability Trade-off: an
  Internal Validation Criterion
  Model selection is a major challenge in non-parametric clustering. There is
no universally admitted way to evaluate clustering results for the obvious
reason that there is no ground truth against which results could be tested, as
in supervised learning. The difficulty to find a universal evaluation criterion
is a direct consequence of the fundamentally ill-defined objective of
clustering. In this perspective, clustering stability has emerged as a natural
and model-agnostic principle: an algorithm should find stable structures in the
data. If data sets are repeatedly sampled from the same underlying
distribution, an algorithm should find similar partitions. However, it turns
out that stability alone is not a well-suited tool to determine the number of
clusters. For instance, it is unable to detect if the number of clusters is too
small. We propose a new principle for clustering validation: a good clustering
should be stable, and within each cluster, there should exist no stable
partition. This principle leads to a novel internal clustering validity
criterion based on between-cluster and within-cluster stability, overcoming
limitations of previous stability-based methods. We empirically show the
superior ability of additive noise to discover structures, compared with
sampling-based perturbation. We demonstrate the effectiveness of our method for
selecting the number of clusters through a large number of experiments and
compare it with existing evaluation methods.
