Unreasonable Effectiveness of Last Hidden Layer Activations for
  Adversarial Robustness
  In standard Deep Neural Network (DNN) based classifiers, the general
convention is to omit the activation function in the last (output) layer and
directly apply the softmax function on the logits to get the probability scores
of each class. In this type of architectures, the loss value of the classifier
against any output class is directly proportional to the difference between the
final probability score and the label value of the associated class. Standard
White-box adversarial evasion attacks, whether targeted or untargeted, mainly
try to exploit the gradient of the model loss function to craft adversarial
samples and fool the model. In this study, we show both mathematically and
experimentally that using some widely known activation functions in the output
layer of the model with high temperature values has the effect of zeroing out
the gradients for both targeted and untargeted attack cases, preventing
attackers from exploiting the model's loss function to craft adversarial
samples. We've experimentally verified the efficacy of our approach on MNIST
(Digit), CIFAR10 datasets. Detailed experiments confirmed that our approach
substantially improves robustness against gradient-based targeted and
untargeted attack threats. And, we showed that the increased non-linearity at
the output layer has some additional benefits against some other attack methods
like Deepfool attack.
