Gradient Leakage Attack Resilient Deep Learning
  Gradient leakage attacks are considered one of the wickedest privacy threats
in deep learning as attackers covertly spy gradient updates during iterative
training without compromising model training quality, and yet secretly
reconstruct sensitive training data using leaked gradients with high attack
success rate. Although deep learning with differential privacy is a defacto
standard for publishing deep learning models with differential privacy
guarantee, we show that differentially private algorithms with fixed privacy
parameters are vulnerable against gradient leakage attacks. This paper
investigates alternative approaches to gradient leakage resilient deep learning
with differential privacy (DP). First, we analyze existing implementation of
deep learning with differential privacy, which use fixed noise variance to
injects constant noise to the gradients in all layers using fixed privacy
parameters. Despite the DP guarantee provided, the method suffers from low
accuracy and is vulnerable to gradient leakage attacks. Second, we present a
gradient leakage resilient deep learning approach with differential privacy
guarantee by using dynamic privacy parameters. Unlike fixed-parameter
strategies that result in constant noise variance, different dynamic parameter
strategies present alternative techniques to introduce adaptive noise variance
and adaptive noise injection which are closely aligned to the trend of gradient
updates during differentially private model training. Finally, we describe four
complementary metrics to evaluate and compare alternative approaches.
