Continuous-Discrete Reinforcement Learning for Hybrid Control in
  Robotics
  Many real-world control problems involve both discrete decision variables -
such as the choice of control modes, gear switching or digital outputs - as
well as continuous decision variables - such as velocity setpoints, control
gains or analogue outputs. However, when defining the corresponding optimal
control or reinforcement learning problem, it is commonly approximated with
fully continuous or fully discrete action spaces. These simplifications aim at
tailoring the problem to a particular algorithm or solver which may only
support one type of action space. Alternatively, expert heuristics are used to
remove discrete actions from an otherwise continuous space. In contrast, we
propose to treat hybrid problems in their 'native' form by solving them with
hybrid reinforcement learning, which optimizes for discrete and continuous
actions simultaneously. In our experiments, we first demonstrate that the
proposed approach efficiently solves such natively hybrid reinforcement
learning problems. We then show, both in simulation and on robotic hardware,
the benefits of removing possibly imperfect expert-designed heuristics. Lastly,
hybrid reinforcement learning encourages us to rethink problem definitions. We
propose reformulating control problems, e.g. by adding meta actions, to improve
exploration or reduce mechanical wear and tear.
