Modeling Topical Coherence in Discourse without Supervision
  Coherence of text is an important attribute to be measured for both manually
and automatically generated discourse; but well-defined quantitative metrics
for it are still elusive. In this paper, we present a metric for scoring
topical coherence of an input paragraph on a real-valued scale by analyzing its
underlying topical structure. We first extract all possible topics that the
sentences of a paragraph of text are related to. Coherence of this text is then
measured by computing: (a) the degree of uncertainty of the topics with respect
to the paragraph, and (b) the relatedness between these topics. All components
of our modular framework rely only on unlabeled data and WordNet, thus making
it completely unsupervised, which is an important feature for general-purpose
usage of any metric. Experiments are conducted on two datasets - a publicly
available dataset for essay grading (representing human discourse), and a
synthetic dataset constructed by mixing content from multiple paragraphs
covering diverse topics. Our evaluation shows that the measured coherence
scores are positively correlated with the ground truth for both the datasets.
Further validation to our coherence scores is provided by conducting human
evaluation on the synthetic data, showing a significant agreement of 79.3%
