Sequential Stochastic Optimization in Separable Learning Environments
  We consider a class of sequential decision-making problems under uncertainty
that can encompass various types of supervised learning concepts. These
problems have a completely observed state process and a partially observed
modulation process, where the state process is affected by the modulation
process only through an observation process, the observation process only
observes the modulation process, and the modulation process is exogenous to
control. We model this broad class of problems as a partially observed Markov
decision process (POMDP). The belief function for the modulation process is
control invariant, thus separating the estimation of the modulation process
from the control of the state process. We call this specially structured POMDP
the separable POMDP, or SEP-POMDP, and show it (i) can serve as a model for a
broad class of application areas, e.g., inventory control, finance, healthcare
systems, (ii) inherits value function and optimal policy structure from a set
of completely observed MDPs, (iii) can serve as a bridge between classical
models of sequential decision making under uncertainty having fully specified
model artifacts and such models that are not fully specified and require the
use of predictive methods from statistics and machine learning, and (iv) allows
for specialized approximate solution procedures.
