Monitoring the shape of weather, soundscapes, and dynamical systems: a
  new statistic for dimension-driven data analysis on large data sets
  Dimensionality-reduction methods are a fundamental tool in the analysis of
large data sets. These algorithms work on the assumption that the "intrinsic
dimension" of the data is generally much smaller than the ambient dimension in
which it is collected. Alongside their usual purpose of mapping data into a
smaller dimension with minimal information loss, dimensionality-reduction
techniques implicitly or explicitly provide information about the dimension of
the data set.
  In this paper, we propose a new statistic that we call the $\kappa$-profile
for analysis of large data sets. The $\kappa$-profile arises from a
dimensionality-reduction optimization problem: namely that of finding a
projection into $k$-dimensions that optimally preserves the secants between
points in the data set. From this optimal projection we extract $\kappa,$ the
norm of the shortest projected secant from among the set of all normalized
secants. This $\kappa$ can be computed for any $k$; thus the tuple of $\kappa$
values (indexed by dimension) becomes a $\kappa$-profile. Algorithms such as
the Secant-Avoidance Projection algorithm and the Hierarchical Secant-Avoidance
Projection algorithm, provide a computationally feasible means of estimating
the $\kappa$-profile for large data sets, and thus a method of understanding
and monitoring their behavior. As we demonstrate in this paper, the
$\kappa$-profile serves as a useful statistic in several representative
settings: weather data, soundscape data, and dynamical systems data.
