Algebraic Ground Truth Inference: Non-Parametric Estimation of Sample
  Errors by AI Algorithms
  Binary classification is widely used in ML production systems. Monitoring
classifiers in a constrained event space is well known. However, real world
production systems often lack the ground truth these methods require. Privacy
concerns may also require that the ground truth needed to evaluate the
classifiers cannot be made available. In these autonomous settings,
non-parametric estimators of performance are an attractive solution. They do
not require theoretical models about how the classifiers made errors in any
given sample. They just estimate how many errors there are in a sample of an
industrial or robotic datastream. We construct one such non-parametric
estimator of the sample errors for an ensemble of weak binary classifiers. Our
approach uses algebraic geometry to reformulate the self-assessment problem for
ensembles of binary classifiers as an exact polynomial system. The polynomial
formulation can then be used to prove - as an algebraic geometry algorithm -
that no general solution to the self-assessment problem is possible. However,
specific solutions are possible in settings where the engineering context puts
the classifiers close to independent errors. The practical utility of the
method is illustrated on a real-world dataset from an online advertising
campaign and a sample of common classification benchmarks. The accuracy
estimators in the experiments where we have ground truth are better than one
part in a hundred. The online advertising campaign data, where we do not have
ground truth data, is verified by an internal consistency approach whose
validity we conjecture as an algebraic geometry theorem. We call this approach
- algebraic ground truth inference.
