How to Understand Masked Autoencoders
  "Masked Autoencoders (MAE) Are Scalable Vision Learners" revolutionizes the
self-supervised learning method in that it not only achieves the
state-of-the-art for image pre-training, but is also a milestone that bridges
the gap between visual and linguistic masked autoencoding (BERT-style)
pre-trainings. However, to our knowledge, to date there are no theoretical
perspectives to explain the powerful expressivity of MAE. In this paper, we,
for the first time, propose a unified theoretical framework that provides a
mathematical understanding for MAE. Specifically, we explain the patch-based
attention approaches of MAE using an integral kernel under a non-overlapping
domain decomposition setting. To help the research community to further
comprehend the main reasons of the great success of MAE, based on our
framework, we pose five questions and answer them with mathematical rigor using
insights from operator theory.
