Optimizing Mixed Autonomy Traffic Flow With Decentralized Autonomous
  Vehicles and Multi-Agent RL
  We study the ability of autonomous vehicles to improve the throughput of a
bottleneck using a fully decentralized control scheme in a mixed autonomy
setting. We consider the problem of improving the throughput of a scaled model
of the San Francisco-Oakland Bay Bridge: a two-stage bottleneck where four
lanes reduce to two and then reduce to one. Although there is extensive work
examining variants of bottleneck control in a centralized setting, there is
less study of the challenging multi-agent setting where the large number of
interacting AVs leads to significant optimization difficulties for
reinforcement learning methods. We apply multi-agent reinforcement algorithms
to this problem and demonstrate that significant improvements in bottleneck
throughput, from 20\% at a 5\% penetration rate to 33\% at a 40\% penetration
rate, can be achieved. We compare our results to a hand-designed feedback
controller and demonstrate that our results sharply outperform the feedback
controller despite extensive tuning. Additionally, we demonstrate that the
RL-based controllers adopt a robust strategy that works across penetration
rates whereas the feedback controllers degrade immediately upon penetration
rate variation. We investigate the feasibility of both action and observation
decentralization and demonstrate that effective strategies are possible using
purely local sensing. Finally, we open-source our code at
https://github.com/eugenevinitsky/decentralized_bottlenecks.
