Applying Cooperative Machine Learning to Speed Up the Annotation of
  Social Signals in Large Multi-modal Corpora
  Scientific disciplines, such as Behavioural Psychology, Anthropology and
recently Social Signal Processing are concerned with the systematic exploration
of human behaviour. A typical work-flow includes the manual annotation (also
called coding) of social signals in multi-modal corpora of considerable size.
For the involved annotators this defines an exhausting and time-consuming task.
In the article at hand we present a novel method and also provide the tools to
speed up the coding procedure. To this end, we suggest and evaluate the use of
Cooperative Machine Learning (CML) techniques to reduce manual labelling
efforts by combining the power of computational capabilities and human
intelligence. The proposed CML strategy starts with a small number of labelled
instances and concentrates on predicting local parts first. Afterwards, a
session-independent classification model is created to finish the remaining
parts of the database. Confidence values are computed to guide the manual
inspection and correction of the predictions. To bring the proposed approach
into application we introduce NOVA - an open-source tool for collaborative and
machine-aided annotations. In particular, it gives labellers immediate access
to CML strategies and directly provides visual feedback on the results. Our
experiments show that the proposed method has the potential to significantly
reduce human labelling efforts.
