Theory and Practice of Quality Assurance for Machine Learning Systems An
  Experiment Driven Approach
  The crafting of machine learning (ML) based systems requires statistical
control throughout its life cycle. Careful quantification of business
requirements and identification of key factors that impact the business
requirements reduces the risk of a project failure. The quantification of
business requirements results in the definition of random variables
representing the system key performance indicators that need to be analyzed
through statistical experiments. In addition, available data for training and
experiment results impact the design of the system. Once the system is
developed, it is tested and continually monitored to ensure it meets its
business requirements. This is done through the continued application of
statistical experiments to analyze and control the key performance indicators.
This book teaches the art of crafting and developing ML based systems. It
advocates an "experiment first" approach stressing the need to define
statistical experiments from the beginning of the project life cycle. It also
discusses in detail how to apply statistical control on the ML based system
throughout its lifecycle.
