Denoising Noisy Neural Networks: A Bayesian Approach with Compensation
  Deep neural networks (DNNs) with noisy weights, which we refer to as noisy
neural networks (NoisyNNs), arise from the training and inference of DNNs in
the presence of noise. NoisyNNs emerge in many new applications, including the
wireless transmission of DNNs, the efficient deployment or storage of DNNs in
analog devices, and the truncation or quantization of DNN weights. This paper
studies a fundamental problem of NoisyNNs: how to reconstruct the DNN weights
from their noisy manifestations. While all prior works relied on the maximum
likelihood (ML) estimation, this paper puts forth a denoising approach to
reconstruct DNNs with the aim of maximizing the inference accuracy of the
reconstructed models. The superiority of our denoiser is rigorously proven in
two small-scale problems, wherein we consider a quadratic neural network
function and a shallow feedforward neural network, respectively. When applied
to advanced learning tasks with modern DNN architectures, our denoiser exhibits
significantly better performance than the ML estimator. Consider the average
test accuracy of the denoised DNN model versus the weight variance to noise
power ratio (WNR) performance. When denoising a noisy ResNet34 model arising
from noisy inference, our denoiser outperforms ML estimation by up to 4.1 dB to
achieve a test accuracy of 60%.When denoising a noisy ResNet18 model arising
from noisy training, our denoiser outperforms ML estimation by 13.4 dB and 8.3
dB to achieve test accuracies of 60% and 80%, respectively.
