Developing an OpenAI Gym-compatible framework and simulation environment
  for testing Deep Reinforcement Learning agents solving the Ambulance Location
  Problem
  Background and motivation: Deep Reinforcement Learning (Deep RL) is a rapidly
developing field. Historically most application has been made to games (such as
chess, Atari games, and go). Deep RL is now reaching the stage where it may
offer value in real world problems, including optimisation of healthcare
systems. One such problem is where to locate ambulances between calls in order
to minimise time from emergency call to ambulance on-scene. This is known as
the Ambulance Location problem.
  Aim: To develop an OpenAI Gym-compatible framework and simulation environment
for testing Deep RL agents.
  Methods: A custom ambulance dispatch simulation environment was developed
using OpenAI Gym and SimPy. Deep RL agents were built using PyTorch. The
environment is a simplification of the real world, but allows control over the
number of clusters of incident locations, number of possible dispatch
locations, number of hospitals, and creating incidents that occur at different
locations throughout each day.
  Results: A range of Deep RL agents based on Deep Q networks were tested in
this custom environment. All reduced time to respond to emergency calls
compared with random allocation to dispatch points. Bagging Noisy Duelling Deep
Q networks gave the most consistence performance. All methods had a tendency to
lose performance if trained for too long, and so agents were saved at their
optimal performance (and tested on independent simulation runs).
  Conclusions: Deep RL agents, developed using simulated environments, have the
potential to offer a novel approach to optimise the Ambulance Location problem.
Creating open simulation environments should allow more rapid progress in this
field.
