Money on the Table: Statistical information ignored by Softmax can
  improve classifier accuracy
  Softmax is a standard final layer used in Neural Nets (NNs) to summarize
information encoded in the trained NN and return a prediction. However, Softmax
leverages only a subset of the class-specific structure encoded in the trained
model and ignores potentially valuable information: During training, models
encode an array $D$ of class response distributions, where $D_{ij}$ is the
distribution of the $j^{th}$ pre-Softmax readout neuron's responses to the
$i^{th}$ class. Given a test sample, Softmax implicitly uses only the row of
this array $D$ that corresponds to the readout neurons' responses to the
sample's true class. Leveraging more of this array $D$ can improve classifier
accuracy, because the likelihoods of two competing classes can be encoded in
other rows of $D$.
  To explore this potential resource, we develop a hybrid classifier
(Softmax-Pooling Hybrid, $SPH$) that uses Softmax on high-scoring samples, but
on low-scoring samples uses a log-likelihood method that pools the information
from the full array $D$. We apply $SPH$ to models trained on a vectorized MNIST
dataset to varying levels of accuracy. $SPH$ replaces only the final Softmax
layer in the trained NN, at test time only. All training is the same as for
Softmax. Because the pooling classifier performs better than Softmax on
low-scoring samples, $SPH$ reduces test set error by 6% to 23%, using the exact
same trained model, whatever the baseline Softmax accuracy. This reduction in
error reflects hidden capacity of the trained NN that is left unused by
Softmax.
