Semi-Markov Offline Reinforcement Learning for Healthcare
  Reinforcement learning (RL) tasks are typically framed as Markov Decision
Processes (MDPs), assuming that decisions are made at fixed time intervals.
However, many applications of great importance, including healthcare, do not
satisfy this assumption, yet they are commonly modelled as MDPs after an
artificial reshaping of the data. In addition, most healthcare (and similar)
problems are offline by nature, allowing for only retrospective studies. To
address both challenges, we begin by discussing the Semi-MDP (SMDP) framework,
which formally handles actions of variable timings. We next present a formal
way to apply SMDP modifications to nearly any given value-based offline RL
method. We use this theory to introduce three SMDP-based offline RL algorithms,
namely, SDQN, SDDQN, and SBCQ. We then experimentally demonstrate that only
these SMDP-based algorithms learn the optimal policy in variable-time
environments, whereas their MDP counterparts do not. Finally, we apply our new
algorithms to a real-world offline dataset pertaining to warfarin dosing for
stroke prevention and demonstrate similar results.
