Czech News Dataset for Semantic Textual Similarity
  This paper describes a novel dataset consisting of sentences with semantic
similarity annotations. The data originate from the journalistic domain in the
Czech language. We describe the process of collecting and annotating the data
in detail. The dataset contains 138,556 human annotations divided into train
and test sets. In total, 485 journalism students participated in the creation
process. To increase the reliability of the test set, we compute the annotation
as an average of 9 individual annotations. We evaluate the quality of the
dataset by measuring inter and intra annotation annotators' agreements. Beside
agreement numbers, we provide detailed statistics of the collected dataset. We
conclude our paper with a baseline experiment of building a system for
predicting the semantic similarity of sentences. Due to the massive number of
training annotations (116 956), the model can perform significantly better than
an average annotator (0,92 versus 0,86 of Person's correlation coefficients).
