Deep Reinforcement Learning for Closed-Loop Blood Glucose Control
  People with type 1 diabetes (T1D) lack the ability to produce the insulin
their bodies need. As a result, they must continually make decisions about how
much insulin to self-administer to adequately control their blood glucose
levels. Longitudinal data streams captured from wearables, like continuous
glucose monitors, can help these individuals manage their health, but currently
the majority of the decision burden remains on the user. To relieve this
burden, researchers are working on closed-loop solutions that combine a
continuous glucose monitor and an insulin pump with a control algorithm in an
`artificial pancreas.' Such systems aim to estimate and deliver the appropriate
amount of insulin. Here, we develop reinforcement learning (RL) techniques for
automated blood glucose control. Through a series of experiments, we compare
the performance of different deep RL approaches to non-RL approaches. We
highlight the flexibility of RL approaches, demonstrating how they can adapt to
new individuals with little additional data. On over 2.1 million hours of data
from 30 simulated patients, our RL approach outperforms baseline control
algorithms: leading to a decrease in median glycemic risk of nearly 50% from
8.34 to 4.24 and a decrease in total time hypoglycemic of 99.8%, from 4,610
days to 6. Moreover, these approaches are able to adapt to predictable meal
times (decreasing average risk by an additional 24% as meals increase in
predictability). This work demonstrates the potential of deep RL to help people
with T1D manage their blood glucose levels without requiring expert knowledge.
All of our code is publicly available, allowing for replication and extension.
