A Pitfall of Learning from User-generated Data: In-depth Analysis of
  Subjective Class Problem
  Research in the supervised learning algorithms field implicitly assumes that
training data is labeled by domain experts or at least semi-professional
labelers accessible through crowdsourcing services like Amazon Mechanical Turk.
With the advent of the Internet, data has become abundant and a large number of
machine learning based systems started being trained with user-generated data,
using categorical data as true labels. However, little work has been done in
the area of supervised learning with user-defined labels where users are not
necessarily experts and might be motivated to provide incorrect labels in order
to improve their own utility from the system. In this article, we propose two
types of classes in user-defined labels: subjective class and objective class -
showing that the objective classes are as reliable as if they were provided by
domain experts, whereas the subjective classes are subject to bias and
manipulation by the user. We define this as a subjective class issue and
provide a framework for detecting subjective labels in a dataset without
querying oracle. Using this framework, data mining practitioners can detect a
subjective class at an early stage of their projects, and avoid wasting their
precious time and resources by dealing with subjective class problem with
traditional machine learning techniques.
