Steady-State Error Compensation in Reference Tracking and Disturbance
  Rejection Problems for Reinforcement Learning-Based Control
  Reinforcement learning (RL) is a promising, upcoming topic in automatic
control applications. Where classical control approaches require a priori
system knowledge, data-driven control approaches like RL allow a model-free
controller design procedure, rendering them emergent techniques for systems
with changing plant structures and varying parameters. While it was already
shown in various applications that the transient control behavior for complex
systems can be sufficiently handled by RL, the challenge of non-vanishing
steady-state control errors remains, which arises from the usage of control
policy approximations and finite training times. To overcome this issue, an
integral action state augmentation (IASA) for actor-critic-based RL controllers
is introduced that mimics an integrating feedback, which is inspired by the
delta-input formulation within model predictive control. This augmentation does
not require any expert knowledge, leaving the approach model free. As a result,
the RL controller learns how to suppress steady-state control deviations much
more effectively. Two exemplary applications from the domain of electrical
energy engineering validate the benefit of the developed method both for
reference tracking and disturbance rejection. In comparison to a standard deep
deterministic policy gradient (DDPG) setup, the suggested IASA extension allows
to reduce the steady-state error by up to 52 $\%$ within the considered
validation scenarios.
