Model-predictive control and reinforcement learning in multi-energy
  system case studies
  Model-predictive-control (MPC) offers an optimal control technique to
establish and ensure that the total operation cost of multi-energy systems
remains at a minimum while fulfilling all system constraints. However, this
method presumes an adequate model of the underlying system dynamics, which is
prone to modelling errors and is not necessarily adaptive. This has an
associated initial and ongoing project-specific engineering cost. In this
paper, we present an on- and off-policy multi-objective reinforcement learning
(RL) approach, that does not assume a model a priori, benchmarking this against
a linear MPC (LMPC - to reflect current practice, though non-linear MPC
performs better) - both derived from the general optimal control problem,
highlighting their differences and similarities. In a simple multi-energy
system (MES) configuration case study, we show that a twin delayed deep
deterministic policy gradient (TD3) RL agent offers potential to match and
outperform the perfect foresight LMPC benchmark (101.5%). This while the
realistic LMPC, i.e. imperfect predictions, only achieves 98%. While in a more
complex MES system configuration, the RL agent's performance is generally lower
(94.6%), yet still better than the realistic LMPC (88.9%). In both case
studies, the RL agents outperformed the realistic LMPC after a training period
of 2 years using quarterly interactions with the environment. We conclude that
reinforcement learning is a viable optimal control technique for multi-energy
systems given adequate constraint handling and pre-training, to avoid unsafe
interactions and long training periods, as is proposed in fundamental future
work.
