DynaMixer: A Vision MLP Architecture with Dynamic Mixing
  Recently, MLP-like vision models have achieved promising performances on
mainstream visual recognition tasks. In contrast with vision transformers and
CNNs, the success of MLP-like models shows that simple information fusion
operations among tokens and channels can yield a good representation power for
deep recognition models. However, existing MLP-like models fuse tokens through
static fusion operations, lacking adaptability to the contents of the tokens to
be mixed. Thus, customary information fusion procedures are not effective
enough. To this end, this paper presents an efficient MLP-like network
architecture, dubbed DynaMixer, resorting to dynamic information fusion.
Critically, we propose a procedure, on which the DynaMixer model relies, to
dynamically generate mixing matrices by leveraging the contents of all the
tokens to be mixed. To reduce the time complexity and improve the robustness, a
dimensionality reduction technique and a multi-segment fusion mechanism are
adopted. Our proposed DynaMixer model (97M parameters) achieves 84.3\% top-1
accuracy on the ImageNet-1K dataset without extra training data, performing
favorably against the state-of-the-art vision MLP models. When the number of
parameters is reduced to 26M, it still achieves 82.7\% top-1 accuracy,
surpassing the existing MLP-like models with a similar capacity. The
implementation of DynaMixer will be made available to the public.
