Balance Between Efficient and Effective Learning: Dense2Sparse Reward
  Shaping for Robot Manipulation with Environment Uncertainty
  Efficient and effective learning is one of the ultimate goals of the deep
reinforcement learning (DRL), although the compromise has been made in most of
the time, especially for the application of robot manipulations. Learning is
always expensive for robot manipulation tasks and the learning effectiveness
could be affected by the system uncertainty. In order to solve above
challenges, in this study, we proposed a simple but powerful reward shaping
method, namely Dense2Sparse. It combines the advantage of fast convergence of
dense reward and the noise isolation of the sparse reward, to achieve a balance
between learning efficiency and effectiveness, which makes it suitable for
robot manipulation tasks. We evaluated our Dense2Sparse method with a series of
ablation experiments using the state representation model with system
uncertainty. The experiment results show that the Dense2Sparse method obtained
higher expected reward compared with the ones using standalone dense reward or
sparse reward, and it also has a superior tolerance of system uncertainty.
