Bitwidth Heterogeneous Federated Learning with Progressive Weight
  Dequantization
  In practical federated learning scenarios, the participating devices may have
different bitwidths for computation and memory storage by design. However,
despite the progress made in device-heterogeneous federated learning scenarios,
the heterogeneity in the bitwidth specifications in the hardware has been
mostly overlooked. We introduce a pragmatic FL scenario with bitwidth
heterogeneity across the participating devices, dubbed as Bitwidth
Heterogeneous Federated Learning (BHFL). BHFL brings in a new challenge, that
the aggregation of model parameters with different bitwidths could result in
severe performance degeneration, especially for high-bitwidth models. To tackle
this problem, we propose ProWD framework, which has a trainable weight
dequantizer at the central server that progressively reconstructs the
low-bitwidth weights into higher bitwidth weights, and finally into
full-precision weights. ProWD further selectively aggregates the model
parameters to maximize the compatibility across bit-heterogeneous weights. We
validate ProWD against relevant FL baselines on the benchmark datasets, using
clients with varying bitwidths. Our ProWD largely outperforms the baseline FL
algorithms as well as naive approaches (e.g. grouped averaging) under the
proposed BHFL scenario.
