Single-shot Hyper-parameter Optimization for Federated Learning: A
  General Algorithm & Analysis
  We address the relatively unexplored problem of hyper-parameter optimization
(HPO) for federated learning (FL-HPO). We introduce Federated Loss SuRface
Aggregation (FLoRA), a general FL-HPO solution framework that can address use
cases of tabular data and any Machine Learning (ML) model including gradient
boosting training algorithms and therefore further expands the scope of FL-HPO.
FLoRA enables single-shot FL-HPO: identifying a single set of good
hyper-parameters that are subsequently used in a single FL training. Thus, it
enables FL-HPO solutions with minimal additional communication overhead
compared to FL training without HPO. We theoretically characterize the
optimality gap of FL-HPO, which explicitly accounts for the heterogeneous
non-IID nature of the parties' local data distributions, a dominant
characteristic of FL systems. Our empirical evaluation of FLoRA for multiple ML
algorithms on seven OpenML datasets demonstrates significant model accuracy
improvements over the considered baseline, and robustness to increasing number
of parties involved in FL-HPO training.
