Topic Discovery via Latent Space Clustering of Pretrained Language Model
  Representations
  Topic models have been the prominent tools for automatic topic discovery from
text corpora. Despite their effectiveness, topic models suffer from several
limitations including the inability of modeling word ordering information in
documents, the difficulty of incorporating external linguistic knowledge, and
the lack of both accurate and efficient inference methods for approximating the
intractable posterior. Recently, pretrained language models (PLMs) have brought
astonishing performance improvements to a wide variety of tasks due to their
superior representations of text. Interestingly, there have not been standard
approaches to deploy PLMs for topic discovery as better alternatives to topic
models. In this paper, we begin by analyzing the challenges of using PLM
representations for topic discovery, and then propose a joint latent space
learning and clustering framework built upon PLM embeddings. In the latent
space, topic-word and document-topic distributions are jointly modeled so that
the discovered topics can be interpreted by coherent and distinctive terms and
meanwhile serve as meaningful summaries of the documents. Our model effectively
leverages the strong representation power and superb linguistic features
brought by PLMs for topic discovery, and is conceptually simpler than topic
models. On two benchmark datasets in different domains, our model generates
significantly more coherent and diverse topics than strong topic models, and
offers better topic-wise document representations, based on both automatic and
human evaluations.
