Human-in-the-Loop Disinformation Detection: Stance, Sentiment, or
  Something Else?
  Both politics and pandemics have recently provided ample motivation for the
development of machine learning-enabled disinformation (a.k.a. fake news)
detection algorithms. Existing literature has focused primarily on the
fully-automated case, but the resulting techniques cannot reliably detect
disinformation on the varied topics, sources, and time scales required for
military applications. By leveraging an already-available analyst as a
human-in-the-loop, however, the canonical machine learning techniques of
sentiment analysis, aspect-based sentiment analysis, and stance detection
become plausible methods to use for a partially-automated disinformation
detection system. This paper aims to determine which of these techniques is
best suited for this purpose and how each technique might best be used towards
this end. Training datasets of the same size and nearly identical neural
architectures (a BERT transformer as a word embedder with a single feed-forward
layer thereafter) are used for each approach, which are then tested on
sentiment- and stance-specific datasets to establish a baseline of how well
each method can be used to do the other tasks. Four different datasets relating
to COVID-19 disinformation are used to test the ability of each technique to
detect disinformation on a topic that did not appear in the training data set.
Quantitative and qualitative results from these tests are then used to provide
insight into how best to employ these techniques in practice.
