Switch Trajectory Transformer with Distributional Value Approximation
  for Multi-Task Reinforcement Learning
  We propose SwitchTT, a multi-task extension to Trajectory Transformer but
enhanced with two striking features: (i) exploiting a sparsely activated model
to reduce computation cost in multi-task offline model learning and (ii)
adopting a distributional trajectory value estimator that improves policy
performance, especially in sparse reward settings. These two enhancements make
SwitchTT suitable for solving multi-task offline reinforcement learning
problems, where model capacity is critical for absorbing the vast quantities of
knowledge available in the multi-task dataset. More specifically, SwitchTT
exploits switch transformer model architecture for multi-task policy learning,
allowing us to improve model capacity without proportional computation cost.
Also, SwitchTT approximates the distribution rather than the expectation of
trajectory value, mitigating the effects of the Monte-Carlo Value estimator
suffering from poor sample complexity, especially in the sparse-reward setting.
We evaluate our method using the suite of ten sparse-reward tasks from the
gym-mini-grid environment.We show an improvement of 10% over Trajectory
Transformer across 10-task learning and obtain up to 90% increase in offline
model training speed. Our results also demonstrate the advantage of the switch
transformer model for absorbing expert knowledge and the importance of value
distribution in evaluating the trajectory.
