Expected Scalarised Returns Dominance: A New Solution Concept for
  Multi-Objective Decision Making
  In many real-world scenarios, the utility of a user is derived from the
single execution of a policy. In this case, to apply multi-objective
reinforcement learning, the expected utility of the returns must be optimised.
Various scenarios exist where a user's preferences over objectives (also known
as the utility function) are unknown or difficult to specify. In such
scenarios, a set of optimal policies must be learned. However, settings where
the expected utility must be maximised have been largely overlooked by the
multi-objective reinforcement learning community and, as a consequence, a set
of optimal solutions has yet to be defined. In this paper we address this
challenge by proposing first-order stochastic dominance as a criterion to build
solution sets to maximise expected utility. We also propose a new dominance
criterion, known as expected scalarised returns (ESR) dominance, that extends
first-order stochastic dominance to allow a set of optimal policies to be
learned in practice. We then define a new solution concept called the ESR set,
which is a set of policies that are ESR dominant. Finally, we define a new
multi-objective distributional tabular reinforcement learning (MOT-DRL)
algorithm to learn the ESR set in a multi-objective multi-armed bandit setting.
