Using Machine Learning at Scale in HPC Simulations with SmartSim: An
  Application to Ocean Climate Modeling
  We demonstrate the first climate-scale, numerical ocean simulations improved
through distributed, online inference of Deep Neural Networks (DNN) using
SmartSim. SmartSim is a library dedicated to enabling online analysis and
Machine Learning (ML) for traditional HPC simulations. In this paper, we detail
the SmartSim architecture and provide benchmarks including online inference
with a shared ML model on heterogeneous HPC systems. We demonstrate the
capability of SmartSim by using it to run a 12-member ensemble of global-scale,
high-resolution ocean simulations, each spanning 19 compute nodes, all
communicating with the same ML architecture at each simulation timestep. In
total, 970 billion inferences are collectively served by running the ensemble
for a total of 120 simulated years. Finally, we show our solution is stable
over the full duration of the model integrations, and that the inclusion of
machine learning has minimal impact on the simulation runtimes.
