Federated Hyperparameter Tuning: Challenges, Baselines, and Connections
  to Weight-Sharing
  Tuning hyperparameters is a crucial but arduous part of the machine learning
pipeline. Hyperparameter optimization is even more challenging in federated
learning, where models are learned over a distributed network of heterogeneous
devices; here, the need to keep data on device and perform local training makes
it difficult to efficiently train and evaluate configurations. In this work, we
investigate the problem of federated hyperparameter tuning. We first identify
key challenges and show how standard approaches may be adapted to form
baselines for the federated setting. Then, by making a novel connection to the
neural architecture search technique of weight-sharing, we introduce a new
method, FedEx, to accelerate federated hyperparameter tuning that is applicable
to widely-used federated optimization methods such as FedAvg and recent
variants. Theoretically, we show that a FedEx variant correctly tunes the
on-device learning rate in the setting of online convex optimization across
devices. Empirically, we show that FedEx can outperform natural baselines for
federated hyperparameter tuning by several percentage points on the
Shakespeare, FEMNIST, and CIFAR-10 benchmarks, obtaining higher accuracy using
the same training budget.
