Predicting batch queue job wait times for informed scheduling of urgent
  HPC workloads
  There is increasing interest in the use of HPC machines for urgent workloads
to help tackle disasters as they unfold. Whilst batch queue systems are not
ideal in supporting such workloads, many disadvantages can be worked around by
accurately predicting when a waiting job will start to run. However there are
numerous challenges in achieving such a prediction with high accuracy, not
least because the queue's state can change rapidly and depend upon many
factors. In this work we explore a novel machine learning approach for
predicting queue wait times, hypothesising that such a model can capture the
complex behaviour resulting from the queue policy and other interactions to
generate accurate job start times.
  For ARCHER2 (HPE Cray EX), Cirrus (HPE 8600) and 4-cabinet (HPE Cray EX) we
explore how different machine learning approaches and techniques improve the
accuracy of our predictions, comparing against the estimation generated by
Slurm. We demonstrate that our techniques deliver the most accurate predictions
across our machines of interest, with the result of this work being the ability
to predict job start times within one minute of the actual start time for
around 65\% of jobs on ARCHER2 and 4-cabinet, and 76\% of jobs on Cirrus. When
compared against what Slurm can deliver, this represents around 3.8 times
better accuracy on ARCHER2 and 18 times better for Cirrus. Furthermore our
approach can accurately predicting the start time for three quarters of all job
within ten minutes of the actual start time on ARCHER2 and 4-cabinet, and for
90\% of jobs on Cirrus. Whilst the driver of this work has been to better
facilitate placement of urgent workloads across HPC machines, the insights
gained can be used to provide wider benefits to users and also enrich existing
batch queue systems and inform policy too.
