Measuring What Counts: The case of Rumour Stance Classification
  Stance classification can be a powerful tool for understanding whether and
which users believe in online rumours. The task aims to automatically predict
the stance of replies towards a given rumour, namely support, deny, question,
or comment. Numerous methods have been proposed and their performance compared
in the RumourEval shared tasks in 2017 and 2019. Results demonstrated that this
is a challenging problem since naturally occurring rumour stance data is highly
imbalanced. This paper specifically questions the evaluation metrics used in
these shared tasks. We re-evaluate the systems submitted to the two RumourEval
tasks and show that the two widely adopted metrics -- accuracy and macro-F1 --
are not robust for the four-class imbalanced task of rumour stance
classification, as they wrongly favour systems with highly skewed accuracy
towards the majority class. To overcome this problem, we propose new evaluation
metrics for rumour stance detection. These are not only robust to imbalanced
data but also score higher systems that are capable of recognising the two most
informative minority classes (support and deny).
