How Reliable are Model Diagnostics?
  In the pursuit of a deeper understanding of a model's behaviour, there is
recent impetus for developing suites of probes aimed at diagnosing models
beyond simple metrics like accuracy or BLEU. This paper takes a step back and
asks an important and timely question: how reliable are these diagnostics in
providing insight into models and training setups? We critically examine three
recent diagnostic tests for pre-trained language models, and find that
likelihood-based and representation-based model diagnostics are not yet as
reliable as previously assumed. Based on our empirical findings, we also
formulate recommendations for practitioners and researchers.
