Local Stochastic Bilevel Optimization with Momentum-Based Variance
  Reduction
  Bilevel Optimization has witnessed notable progress recently with new
emerging efficient algorithms and has been applied to many machine learning
tasks such as data cleaning, few-shot learning, and neural architecture search.
However, little attention has been paid to solve the bilevel problems under
distributed setting. Federated learning (FL) is an emerging paradigm which
solves machine learning tasks over distributed-located data. FL problems are
challenging to solve due to the heterogeneity and communication bottleneck.
However, it is unclear how these challenges will affect the convergence of
Bilevel Optimization algorithms. In this paper, we study Federated Bilevel
Optimization problems. Specifically, we first propose the FedBiO, a
deterministic gradient-based algorithm and we show it requires
$O(\epsilon^{-2})$ number of iterations to reach an $\epsilon$-stationary
point. Then we propose FedBiOAcc to accelerate FedBiO with the momentum-based
variance-reduction technique under the stochastic scenario. We show FedBiOAcc
has complexity of $O(\epsilon^{-1.5})$. Finally, we validate our proposed
algorithms via the important Fair Federated Learning task. More specifically,
we define a bilevel-based group fair FL objective. Our algorithms show superior
performances compared to other baselines in numerical experiments.
