Can Attention Enable MLPs To Catch Up With CNNs?
  In the first week of May, 2021, researchers from four different institutions:
Google, Tsinghua University, Oxford University and Facebook, shared their
latest work [16, 7, 12, 17] on arXiv.org almost at the same time, each
proposing new learning architectures, consisting mainly of linear layers,
claiming them to be comparable, or even superior to convolutional-based models.
This sparked immediate discussion and debate in both academic and industrial
communities as to whether MLPs are sufficient, many thinking that learning
architectures are returning to MLPs. Is this true? In this perspective, we give
a brief history of learning architectures, including multilayer perceptrons
(MLPs), convolutional neural networks (CNNs) and transformers. We then examine
what the four newly proposed architectures have in common. Finally, we give our
views on challenges and directions for new learning architectures, hoping to
inspire future research.
