On the Convergence of Differentially Private Federated Learning on
  Non-Lipschitz Objectives, and with Normalized Client Updates
  There is a dearth of convergence results for differentially private federated
learning (FL) with non-Lipschitz objective functions (i.e., when gradient norms
are not bounded). The primary reason for this is that the clipping operation
(i.e., projection onto an $\ell_2$ ball of a fixed radius called the clipping
threshold) for bounding the sensitivity of the average update to each client's
update introduces bias depending on the clipping threshold and the number of
local steps in FL, and analyzing this is not easy. For Lipschitz functions, the
Lipschitz constant serves as a trivial clipping threshold with zero bias.
However, Lipschitzness does not hold in many practical settings; moreover,
verifying it and computing the Lipschitz constant is hard. Thus, the choice of
the clipping threshold is non-trivial and requires a lot of tuning in practice.
In this paper, we provide the first convergence result for private FL on smooth
\textit{convex} objectives \textit{for a general clipping threshold} --
\textit{without assuming Lipschitzness}. We also look at a simpler alternative
to clipping (for bounding sensitivity) which is \textit{normalization} -- where
we use only a scaled version of the unit vector along the client updates,
completely discarding the magnitude information. {The resulting
normalization-based private FL algorithm is theoretically shown to have better
convergence than its clipping-based counterpart on smooth convex functions. We
corroborate our theory with synthetic experiments as well as experiments on
benchmarking datasets.
