Multi-Agent Inverse Reinforcement Learning: Suboptimal Demonstrations
  and Alternative Solution Concepts
  Multi-agent inverse reinforcement learning (MIRL) can be used to learn reward
functions from agents in social environments. To model realistic social
dynamics, MIRL methods must account for suboptimal human reasoning and
behavior. Traditional formalisms of game theory provide computationally
tractable behavioral models, but assume agents have unrealistic cognitive
capabilities. This research identifies and compares mechanisms in MIRL methods
which a) handle noise, biases and heuristics in agent decision making and b)
model realistic equilibrium solution concepts. MIRL research is systematically
reviewed to identify solutions for these challenges. The methods and results of
these studies are analyzed and compared based on factors including performance
accuracy, efficiency, and descriptive quality. We found that the primary
methods for handling noise, biases and heuristics in MIRL were extensions of
Maximum Entropy (MaxEnt) IRL to multi-agent settings. We also found that many
successful solution concepts are generalizations of the traditional Nash
Equilibrium (NE). These solutions include the correlated equilibrium, logistic
stochastic best response equilibrium and entropy regularized mean field NE.
Methods which use recursive reasoning or updating also perform well, including
the feedback NE and archive multi-agent adversarial IRL. Success in modeling
specific biases and heuristics in single-agent IRL and promising results using
a Theory of Mind approach in MIRL imply that modeling specific biases and
heuristics may be useful. Flexibility and unbiased inference in the identified
alternative solution concepts suggest that a solution concept which has both
recursive and generalized characteristics may perform well at modeling
realistic social interactions.
