PALBERT: Teaching ALBERT to Ponder
  Currently, pre-trained models can be considered the default choice for a wide
range of NLP tasks. Despite their SoTA results, there is practical evidence
that these models may require a different number of computing layers for
different input sequences, since evaluating all layers leads to overconfidence
on wrong predictions (namely overthinking). This problem can potentially be
solved by implementing adaptive computation time approaches, which were first
designed to improve inference speed. Recently proposed PonderNet may be a
promising solution for performing an early exit by treating the exit layers
index as a latent variable. However, the originally proposed exit criterion,
relying on sampling from trained posterior distribution on the probability of
exiting from i-th layer, introduces major variance in model outputs,
significantly reducing the resulting models performance. In this paper, we
propose Ponder ALBERT (PALBERT): an improvement to PonderNet with a novel
deterministic Q-exit criterion and a revisited model architecture. We compared
PALBERT with recent methods for performing an early exit. We observed that the
proposed changes can be considered significant improvements on the original
PonderNet architecture and outperform PABEE on a wide range of GLUE tasks. In
addition, we also performed an in-depth ablation study of the proposed
architecture to further understand Lambda layers and their performance.
