Impact of L1 Batch Normalization on Analog Noise Resistant Property of Deep Learning Models
Analog hardware has become a popular choice for machine learning on
resource-constrained devices recently due to its fast execution and energy
efficiency. However, the inherent presence of noise in analog hardware and the
negative impact of the noise on deployed deep neural network (DNN) models limit
their usage. The degradation in performance due to the noise calls for the
novel design of DNN models that have excellent noiseresistant property,
leveraging the properties of the fundamental building block of DNN models. In
this work, the use of L1 or TopK BatchNorm type, a fundamental DNN model
building block, in designing DNN models with excellent noise-resistant property
is proposed. Specifically, a systematic study has been carried out by training
DNN models with L1/TopK BatchNorm type, and the performance is compared with
DNN models with L2 BatchNorm types. The resulting model noise-resistant
property is tested by injecting additive noise to the model weights and
evaluating the new model inference accuracy due to the noise. The results show
that L1 and TopK BatchNorm type has excellent noise-resistant property, and
there is no sacrifice in performance due to the change in the BatchNorm type
from L2 to L1/TopK BatchNorm type.