Model-free and Bayesian Ensembling Model-based Deep Reinforcement
  Learning for Particle Accelerator Control Demonstrated on the FERMI FEL
  Reinforcement learning holds tremendous promise in accelerator controls. The
primary goal of this paper is to show how this approach can be utilised on an
operational level on accelerator physics problems. Despite the success of
model-free reinforcement learning in several domains, sample-efficiency still
is a bottle-neck, which might be encompassed by model-based methods. We compare
well-suited purely model-based to model-free reinforcement learning applied to
the intensity optimisation on the FERMI FEL system. We find that the
model-based approach demonstrates higher representational power and
sample-efficiency, while the asymptotic performance of the model-free method is
slightly superior. The model-based algorithm is implemented in a DYNA-style
using an uncertainty aware model, and the model-free algorithm is based on
tailored deep Q-learning. In both cases, the algorithms were implemented in a
way, which presents increased noise robustness as omnipresent in accelerator
control problems. Code is released in
https://github.com/MathPhysSim/FERMI_RL_Paper.
