The GatedTabTransformer. An enhanced deep learning architecture for
  tabular modeling
  There is an increasing interest in the application of deep learning
architectures to tabular data. One of the state-of-the-art solutions is
TabTransformer which incorporates an attention mechanism to better track
relationships between categorical features and then makes use of a standard MLP
to output its final logits. In this paper we propose multiple modifications to
the original TabTransformer performing better on binary classification tasks
for three separate datasets with more than 1% AUROC gains. Inspired by gated
MLP, linear projections are implemented in the MLP block and multiple
activation functions are tested. We also evaluate the importance of specific
hyper parameters during training.
