User-Oriented Robust Reinforcement Learning
  Recently, improving the robustness of policies across different environments
attracts increasing attention in the reinforcement learning (RL) community.
Existing robust RL methods mostly aim to achieve the max-min robustness by
optimizing the policy's performance in the worst-case environment. However, in
practice, a user that uses an RL policy may have different preferences over its
performance across environments. Clearly, the aforementioned max-min robustness
is oftentimes too conservative to satisfy user preference. Therefore, in this
paper, we integrate user preference into policy learning in robust RL, and
propose a novel User-Oriented Robust RL (UOR-RL) framework. Specifically, we
define a new User-Oriented Robustness (UOR) metric for RL, which allocates
different weights to the environments according to user preference and
generalizes the max-min robustness metric. To optimize the UOR metric, we
develop two different UOR-RL training algorithms for the scenarios with or
without a priori known environment distribution, respectively. Theoretically,
we prove that our UOR-RL training algorithms converge to near-optimal policies
even with inaccurate or completely no knowledge about the environment
distribution. Furthermore, we carry out extensive experimental evaluations in 4
MuJoCo tasks. The experimental results demonstrate that UOR-RL is comparable to
the state-of-the-art baselines under the average and worst-case performance
metrics, and more importantly establishes new state-of-the-art performance
under the UOR metric.
