Where Does the Robustness Come from? A Study of the Transformation-based
  Ensemble Defence
  This paper aims to provide a thorough study on the effectiveness of the
transformation-based ensemble defence for image classification and its reasons.
It has been empirically shown that they can enhance the robustness against
evasion attacks, while there is little analysis on the reasons. In particular,
it is not clear whether the robustness improvement is a result of
transformation or ensemble. In this paper, we design two adaptive attacks to
better evaluate the transformation-based ensemble defence. We conduct
experiments to show that 1) the transferability of adversarial examples exists
among the models trained on data records after different reversible
transformations; 2) the robustness gained through transformation-based ensemble
is limited; 3) this limited robustness is mainly from the irreversible
transformations rather than the ensemble of a number of models; and 4) blindly
increasing the number of sub-models in a transformation-based ensemble does not
bring extra robustness gain.
