HyAR: Addressing Discrete-Continuous Action Reinforcement Learning via
  Hybrid Action Representation
  Discrete-continuous hybrid action space is a natural setting in many
practical problems, such as robot control and game AI. However, most previous
Reinforcement Learning (RL) works only demonstrate the success in controlling
with either discrete or continuous action space, while seldom take into account
the hybrid action space. One naive way to address hybrid action RL is to
convert the hybrid action space into a unified homogeneous action space by
discretization or continualization, so that conventional RL algorithms can be
applied. However, this ignores the underlying structure of hybrid action space
and also induces the scalability issue and additional approximation
difficulties, thus leading to degenerated results. In this paper, we propose
Hybrid Action Representation (HyAR) to learn a compact and decodable latent
representation space for the original hybrid action space. HyAR constructs the
latent space and embeds the dependence between discrete action and continuous
parameter via an embedding table and conditional Variantional Auto-Encoder
(VAE). To further improve the effectiveness, the action representation is
trained to be semantically smooth through unsupervised environmental dynamics
prediction. Finally, the agent then learns its policy with conventional DRL
algorithms in the learned representation space and interacts with the
environment by decoding the hybrid action embeddings to the original action
space. We evaluate HyAR in a variety of environments with discrete-continuous
action space. The results demonstrate the superiority of HyAR when compared
with previous baselines, especially for high-dimensional action spaces.
