Improved Bilevel Model: Fast and Optimal Algorithm with Theoretical
  Guarantee
  Due to the hierarchical structure of many machine learning problems, bilevel
programming is becoming more and more important recently, however, the
complicated correlation between the inner and outer problem makes it extremely
challenging to solve. Although several intuitive algorithms based on the
automatic differentiation have been proposed and obtained success in some
applications, not much attention has been paid to finding the optimal
formulation of the bilevel model. Whether there exists a better formulation is
still an open problem. In this paper, we propose an improved bilevel model
which converges faster and better compared to the current formulation. We
provide theoretical guarantee and evaluation results over two tasks: Data
Hyper-Cleaning and Hyper Representation Learning. The empirical results show
that our model outperforms the current bilevel model with a great margin.
\emph{This is a concurrent work with \citet{liu2020generic} and we submitted to
ICML 2020. Now we put it on the arxiv for record.}
