Using Simulation to Aid the Design and Optimization of Intelligent User
  Interfaces for Quality Assurance Processes in Machine Learning
  Many mission-critical applications of machine learning (ML) in the real-world
require a quality assurance (QA) process before the decisions or predictions of
an ML model can be deployed. Because QA4ML users have to view a non-trivial
amount of data and perform many input actions to correct errors made by the ML
model, an optimally-designed user interface (UI) can reduce the cost of
interactions significantly. A UI's effectiveness can be affected by many
factors, such as the number of data objects processed concurrently, the types
of commands for correcting errors, and the availability of algorithms for
assisting users. We propose using simulation to aid the design and optimization
of intelligent user interfaces for QA4ML processes. In particular, we focus on
simulating the combined effects of human intelligence in selecting appropriate
commands and algorithms, and machine intelligence in providing a collection of
general-purpose algorithms for reordering data objects to be quality-assured.
