High-dimensional Penalty Selection via Minimum Description Length
  Principle
  We tackle the problem of penalty selection of regularization on the basis of
the minimum description length (MDL) principle. In particular, we consider that
the design space of the penalty function is high-dimensional. In this
situation, the luckiness-normalized-maximum-likelihood(LNML)-minimization
approach is favorable, because LNML quantifies the goodness of regularized
models with any forms of penalty functions in view of the minimum description
length principle, and guides us to a good penalty function through the
high-dimensional space. However, the minimization of LNML entails two major
challenges: 1) the computation of the normalizing factor of LNML and 2) its
minimization in high-dimensional spaces. In this paper, we present a novel
regularization selection method (MDL-RS), in which a tight upper bound of LNML
(uLNML) is minimized with local convergence guarantee. Our main contribution is
the derivation of uLNML, which is a uniform-gap upper bound of LNML in an
analytic expression. This solves the above challenges in an approximate manner
because it allows us to accurately approximate LNML and then efficiently
minimize it. The experimental results show that MDL-RS improves the
generalization performance of regularized estimates specifically when the model
has redundant parameters.
