SUPERNOVA: Automating Test Selection and Defect Prevention in AAA Video
  Games Using Risk Based Testing and Machine Learning
  Testing video games is an increasingly difficult task as traditional methods
fail to scale with growing software systems. Manual testing is a very
labor-intensive process, and therefore quickly becomes cost prohibitive. Using
scripts for automated testing is affordable, however scripts are ineffective in
non-deterministic environments, and knowing when to run each test is another
problem altogether. The modern game's complexity, scope, and player
expectations are rapidly increasing where quality control is a big portion of
the production cost and delivery risk. Reducing this risk and making production
happen is a big challenge for the industry currently. To keep production costs
realistic up-to and after release, we are focusing on preventive quality
assurance tactics alongside testing and data analysis automation. We present
SUPERNOVA (Selection of tests and Universal defect Prevention in External
Repositories for Novel Objective Verification of software Anomalies), a system
responsible for test selection and defect prevention while also functioning as
an automation hub. By integrating data analysis functionality with machine and
deep learning capability, SUPERNOVA assists quality assurance testers in
finding bugs and developers in reducing defects, which improves stability
during the production cycle and keeps testing costs under control. The direct
impact of this has been observed to be a reduction in 55% or more testing hours
for an undisclosed sports game title that has shipped, which was using these
test selection optimizations. Furthermore, using risk scores generated by a
semi-supervised machine learning model, we are able to detect with 71%
precision and 77% recall the probability of a change-list being bug inducing,
and provide a detailed breakdown of this inference to developers. These efforts
improve workflow and reduce testing hours required on game titles in
development.
