Evaluating Text Coherence at Sentence and Paragraph Levels
  In this paper, to evaluate text coherence, we propose the paragraph ordering
task as well as conducting sentence ordering. We collected four distinct
corpora from different domains on which we investigate the adaptation of
existing sentence ordering methods to a paragraph ordering task. We also
compare the learnability and robustness of existing models by artificially
creating mini datasets and noisy datasets respectively and verifying the
efficiency of established models under these circumstances. Furthermore, we
carry out human evaluation on the rearranged passages from two competitive
models and confirm that WLCS-l is a better metric performing significantly
higher correlations with human rating than tau, the most prevalent metric used
before. Results from these evaluations show that except for certain extreme
conditions, the recurrent graph neural network-based model is an optimal choice
for coherence modeling.
