Towards Modeling and Resolving Singular Parameter Spaces using
  Stratifolds
  When analyzing parametric statistical models, a useful approach consists in
modeling geometrically the parameter space. However, even for very simple and
commonly used hierarchical models like statistical mixtures or stochastic deep
neural networks, the smoothness assumption of manifolds is violated at singular
points which exhibit non-smooth neighborhoods in the parameter space. These
singular models have been analyzed in the context of learning dynamics, where
singularities can act as attractors on the learning trajectory and, therefore,
negatively influence the convergence speed of models. We propose a general
approach to circumvent the problem arising from singularities by using
stratifolds, a concept from algebraic topology, to formally model singular
parameter spaces. We use the property that specific stratifolds are equipped
with a resolution method to construct a smooth manifold approximation of the
singular space. We empirically show that using (natural) gradient descent on
the smooth manifold approximation instead of the singular space allows us to
avoid the attractor behavior and therefore improve the convergence speed in
learning.
