Intrinsic dimension estimation for locally undersampled data
  High-dimensional data are ubiquitous in contemporary science and finding
methods to compress them is one of the primary goals of machine learning. Given
a dataset lying in a high-dimensional space (in principle hundreds to several
thousands of dimensions), it is often useful to project it onto a
lower-dimensional manifold, without loss of information. Identifying the
minimal dimension of such manifold is a challenging problem known in the
literature as intrinsic dimension estimation (IDE). Traditionally, most IDE
algorithms are either based on multiscale principal component analysis (PCA) or
on the notion of correlation dimension (and more in general on
k-nearest-neighbors distances). These methods are affected, in different ways,
by a severe curse of dimensionality. In particular, none of the existing
algorithms can provide accurate ID estimates in the extreme locally
undersampled regime, i.e. in the limit where the number of samples in any local
patch of the manifold is less than (or of the same order of) the ID of the
dataset. Here we introduce a new ID estimator that leverages on simple
properties of the tangent space of a manifold to overcome these shortcomings.
The method is based on the full correlation integral, going beyond the limit of
small radius used for the estimation of the correlation dimension. Our
estimator alleviates the extreme undersampling problem, intractable with other
methods. Based on this insight, we explore a multiscale generalization of the
algorithm. We show that it is capable of (i) identifying multiple
dimensionalities in a dataset, and (ii) providing accurate estimates of the ID
of extremely curved manifolds. In particular, we test the method on manifolds
generated from global transformations of high-contrast images, relevant for
invariant object recognition and considered a challenge for state-of-the-art ID
estimators.
