Suppressing the Unusual: towards Robust CNNs using Symmetric Activation
  Functions
  Many deep Convolutional Neural Networks (CNN) make incorrect predictions on
adversarial samples obtained by imperceptible perturbations of clean samples.
We hypothesize that this is caused by a failure to suppress unusual signals
within network layers. As remedy we propose the use of Symmetric Activation
Functions (SAF) in non-linear signal transducer units. These units suppress
signals of exceptional magnitude. We prove that SAF networks can perform
classification tasks to arbitrary precision in a simplified situation. In
practice, rather than use SAFs alone, we add them into CNNs to improve their
robustness. The modified CNNs can be easily trained using popular strategies
with the moderate training load. Our experiments on MNIST and CIFAR-10 show
that the modified CNNs perform similarly to plain ones on clean samples, and
are remarkably more robust against adversarial and nonsense samples.
