Cloud-Edge Training Architecture for Sim-to-Real Deep Reinforcement
  Learning
  Deep reinforcement learning (DRL) is a promising approach to solve complex
control tasks by learning policies through interactions with the environment.
However, the training of DRL policies requires large amounts of training
experiences, making it impractical to learn the policy directly on physical
systems. Sim-to-real approaches leverage simulations to pretrain DRL policies
and then deploy them in the real world. Unfortunately, the direct real-world
deployment of pretrained policies usually suffers from performance
deterioration due to the different dynamics, known as the reality gap. Recent
sim-to-real methods, such as domain randomization and domain adaptation, focus
on improving the robustness of the pretrained agents. Nevertheless, the
simulation-trained policies often need to be tuned with real-world data to
reach optimal performance, which is challenging due to the high cost of
real-world samples.
  This work proposes a distributed cloud-edge architecture to train DRL agents
in the real world in real-time. In the architecture, the inference and training
are assigned to the edge and cloud, separating the real-time control loop from
the computationally expensive training loop. To overcome the reality gap, our
architecture exploits sim-to-real transfer strategies to continue the training
of simulation-pretrained agents on a physical system. We demonstrate its
applicability on a physical inverted-pendulum control system, analyzing
critical parameters. The real-world experiments show that our architecture can
adapt the pretrained DRL agents to unseen dynamics consistently and
efficiently.
