Data segmentation based on the local intrinsic dimension
  One of the founding paradigms of machine learning is that a small number of
variables is often sufficient to describe high-dimensional data. The minimum
number of variables required is called the intrinsic dimension (ID) of the
data. Contrary to common intuition, there are cases where the ID varies within
the same data set. This fact has been highlighted in technical discussions, but
seldom exploited to analyze large data sets and obtain insight into their
structure. Here we develop a robust approach to discriminate regions with
different local IDs and segment the points accordingly. Our approach is
computationally efficient and can be proficiently used even on large data sets.
We find that many real-world data sets contain regions with widely
heterogeneous dimensions. These regions host points differing in core
properties: folded vs unfolded configurations in a protein molecular dynamics
trajectory, active vs non-active regions in brain imaging data, and firms with
different financial risk in company balance sheets. A simple topological
feature, the local ID, is thus sufficient to achieve an unsupervised
segmentation of high-dimensional data, complementary to the one given by
clustering algorithms.
