Low-Rank Softmax Can Have Unargmaxable Classes in Theory but Rarely in
  Practice
  Classifiers in natural language processing (NLP) often have a large number of
output classes. For example, neural language models (LMs) and machine
translation (MT) models both predict tokens from a vocabulary of thousands. The
Softmax output layer of these models typically receives as input a dense
feature representation, which has much lower dimensionality than the output. In
theory, the result is some words may be impossible to be predicted via argmax,
irrespective of input features, and empirically, there is evidence this happens
in small language models. In this paper we ask whether it can happen in
practical large language models and translation models. To do so, we develop
algorithms to detect such \emph{unargmaxable} tokens in public models. We find
that 13 out of 150 models do indeed have such tokens; however, they are very
infrequent and unlikely to impact model quality. We release our code so that
others can inspect their models.
