Recurrent Sum-Product-Max Networks for Decision Making in
  Perfectly-Observed Environments
  Recent investigations into sum-product-max networks (SPMN) that generalize
sum-product networks (SPN) offer a data-driven alternative for decision making,
which has predominantly relied on handcrafted models. SPMNs computationally
represent a probabilistic decision-making problem whose solution scales
linearly in the size of the network. However, SPMNs are not well suited for
sequential decision making over multiple time steps. In this paper, we present
recurrent SPMNs (RSPMN) that learn from and model decision-making data over
time. RSPMNs utilize a template network that is unfolded as needed depending on
the length of the data sequence. This is significant as RSPMNs not only inherit
the benefits of SPMNs in being data driven and mostly tractable, they are also
well suited for sequential problems. We establish conditions on the template
network, which guarantee that the resulting SPMN is valid, and present a
structure learning algorithm to learn a sound template network. We demonstrate
that the RSPMNs learned on a testbed of sequential decision-making data sets
generate MEUs and policies that are close to the optimal on perfectly-observed
domains. They easily improve on a recent batch-constrained reinforcement
learning method, which is important because RSPMNs offer a new model-based
approach to offline reinforcement learning.
