Joint rotational invariance and adversarial training of a dual-stream
  Transformer yields state of the art Brain-Score for Area V4
  Modern high-scoring models of vision in the brain score competition do not
stem from Vision Transformers. However, in this paper, we provide evidence
against the unexpected trend of Vision Transformers (ViT) being not
perceptually aligned with human visual representations by showing how a
dual-stream Transformer, a CrossViT$~\textit{a la}$ Chen et al. (2021), under a
joint rotationally-invariant and adversarial optimization procedure yields 2nd
place in the aggregate Brain-Score 2022 competition(Schrimpf et al., 2020b)
averaged across all visual categories, and at the time of the competition held
1st place for the highest explainable variance of area V4. In addition, our
current Transformer-based model also achieves greater explainable variance for
areas V4, IT and Behaviour than a biologically-inspired CNN (ResNet50) that
integrates a frontal V1-like computation module (Dapello et al.,2020). To
assess the contribution of the optimization scheme with respect to the CrossViT
architecture, we perform several additional experiments on differently
optimized CrossViT's regarding adversarial robustness, common corruption
benchmarks, mid-ventral stimuli interpretation and feature inversion. Against
our initial expectations, our family of results provides tentative support for
an $\textit{"All roads lead to Rome"}$ argument enforced via a joint
optimization rule even for non biologically-motivated models of vision such as
Vision Transformers. Code is available at
https://github.com/williamberrios/BrainScore-Transformers
