Robust Document Representations using Latent Topics and Metadata
  Task specific fine-tuning of a pre-trained neural language model using a
custom softmax output layer is the de facto approach of late when dealing with
document classification problems. This technique is not adequate when labeled
examples are not available at training time and when the metadata artifacts in
a document must be exploited. We address these challenges by generating
document representations that capture both text and metadata artifacts in a
task agnostic manner. Instead of traditional auto-regressive or auto-encoding
based training, our novel self-supervised approach learns a soft-partition of
the input space when generating text embeddings. Specifically, we employ a
pre-learned topic model distribution as surrogate labels and construct a loss
function based on KL divergence. Our solution also incorporates metadata
explicitly rather than just augmenting them with text. The generated document
embeddings exhibit compositional characteristics and are directly used by
downstream classification tasks to create decision boundaries from a small
number of labeled examples, thereby eschewing complicated recognition methods.
We demonstrate through extensive evaluation that our proposed cross-model
fusion solution outperforms several competitive baselines on multiple datasets.
