Revisiting Deep Learning Models for Tabular Data
  The existing literature on deep learning for tabular data proposes a wide
range of novel architectures and reports competitive results on various
datasets. However, the proposed models are usually not properly compared to
each other and existing works often use different benchmarks and experiment
protocols. As a result, it is unclear for both researchers and practitioners
what models perform best. Additionally, the field still lacks effective
baselines, that is, the easy-to-use models that provide competitive performance
across different problems.
  In this work, we perform an overview of the main families of DL architectures
for tabular data and raise the bar of baselines in tabular DL by identifying
two simple and powerful deep architectures. The first one is a ResNet-like
architecture which turns out to be a strong baseline that is often missing in
prior works. The second model is our simple adaptation of the Transformer
architecture for tabular data, which outperforms other solutions on most tasks.
Both models are compared to many existing architectures on a diverse set of
tasks under the same training and tuning protocols. We also compare the best DL
models with Gradient Boosted Decision Trees and conclude that there is still no
universally superior solution.
