Joint Multi-Dimensional Model for Global and Time-Series Annotations
  Crowdsourcing is a popular approach to collect annotations for unlabeled data
instances. It involves collecting a large number of annotations from several,
often naive untrained annotators for each data instance which are then combined
to estimate the ground truth. Further, annotations for constructs such as
affect are often multi-dimensional with annotators rating multiple dimensions,
such as valence and arousal, for each instance. Most annotation fusion schemes
however ignore this aspect and model each dimension separately. In this work we
address this by proposing a generative model for multi-dimensional annotation
fusion, which models the dimensions jointly leading to more accurate ground
truth estimates. The model we propose is applicable to both global and time
series annotation fusion problems and treats the ground truth as a latent
variable distorted by the annotators. The model parameters are estimated using
the Expectation-Maximization algorithm and we evaluate its performance using
synthetic data and real emotion corpora as well as on an artificial task with
human annotations
