Value Variance Minimization for Learning Approximate Equilibrium in
  Aggregation Systems
  For effective matching of resources (e.g., taxis, food, bikes, shopping
items) to customer demand, aggregation systems have been extremely successful.
In aggregation systems, a central entity (e.g., Uber, Food Panda, Ofo)
aggregates supply (e.g., drivers, delivery personnel) and matches demand to
supply on a continuous basis (sequential decisions). Due to the objective of
the central entity to maximize its profits, individual suppliers get sacrificed
thereby creating incentive for individuals to leave the system. In this paper,
we consider the problem of learning approximate equilibrium solutions (win-win
solutions) in aggregation systems, so that individuals have an incentive to
remain in the aggregation system.
  Unfortunately, such systems have thousands of agents and have to consider
demand uncertainty and the underlying problem is a (Partially Observable)
Stochastic Game. Given the significant complexity of learning or planning in a
stochastic game, we make three key contributions: (a) To exploit
infinitesimally small contribution of each agent and anonymity (reward and
transitions between agents are dependent on agent counts) in interactions, we
represent this as a Multi-Agent Reinforcement Learning (MARL) problem that
builds on insights from non-atomic congestion games model; (b) We provide a
novel variance reduction mechanism for moving joint solution towards Nash
Equilibrium that exploits the infinitesimally small contribution of each agent;
and finally (c) We provide detailed results on three different domains to
demonstrate the utility of our approach in comparison to state-of-the-art
methods.
