Intentional Control of Type I Error over Unconscious Data Distortion: a
  Neyman-Pearson Approach to Text Classification
  This paper addresses the challenges in classifying textual data obtained from
open online platforms, which are vulnerable to distortion. Most existing
classification methods minimize the overall classification error and may yield
an undesirably large type I error (relevant textual messages are classified as
irrelevant), particularly when available data exhibit an asymmetry between
relevant and irrelevant information. Data distortion exacerbates this situation
and often leads to fallacious prediction. To deal with inestimable data
distortion, we propose the use of the Neyman-Pearson (NP) classification
paradigm, which minimizes type II error under a user-specified type I error
constraint. Theoretically, we show that the NP oracle is unaffected by data
distortion when the class conditional distributions remain the same.
Empirically, we study a case of classifying posts about worker strikes obtained
from a leading Chinese microblogging platform, which are frequently prone to
extensive, unpredictable and inestimable censorship. We demonstrate that, even
though the training and test data are susceptible to different distortion and
therefore potentially follow different distributions, our proposed NP methods
control the type I error on test data at the targeted level. The methods and
implementation pipeline proposed in our case study are applicable to many other
problems involving data distortion.
