Addressing practical challenges in Active Learning via a hybrid query
  strategy
  Active Learning (AL) is a powerful tool to address modern machine learning
problems with significantly fewer labeled training instances. However,
implementation of traditional AL methodologies in practical scenarios is
accompanied by multiple challenges due to the inherent assumptions. There are
several hindrances, such as unavailability of labels for the AL algorithm at
the beginning; unreliable external source of labels during the querying
process; or incompatible mechanisms to evaluate the performance of Active
Learner. Inspired by these practical challenges, we present a hybrid query
strategy-based AL framework that addresses three practical challenges
simultaneously: cold-start, oracle uncertainty and performance evaluation of
Active Learner in the absence of ground truth. While a pre-clustering approach
is employed to address the cold-start problem, the uncertainty surrounding the
expertise of labeler and confidence in the given labels is incorporated to
handle oracle uncertainty. The heuristics obtained during the querying process
serve as the fundamental premise for accessing the performance of Active
Learner. The robustness of the proposed AL framework is evaluated across three
different environments and industrial settings. The results demonstrate the
capability of the proposed framework to tackle practical challenges during AL
implementation in real-world scenarios.
