Structural Estimation of Partially Observable Markov Decision Processes
  In many practical settings control decisions must be made under
partial/imperfect information about the evolution of a relevant state variable.
Partially Observable Markov Decision Processes (POMDPs) is a relatively
well-developed framework for modeling and analyzing such problems. In this
paper we consider the structural estimation of the primitives of a POMDP model
based upon the observable history of the process. We analyze the structural
properties of POMDP model with random rewards and specify conditions under
which the model is identifiable without knowledge of the state dynamics. We
consider a soft policy gradient algorithm to compute a maximum likelihood
estimator and provide a finite-time characterization of convergence to a
stationary point. We illustrate the estimation methodology with an application
to optimal equipment replacement. In this context, replacement decisions must
be made under partial/imperfect information on the true state (i.e. condition
of the equipment). We use synthetic and real data to highlight the robustness
of the proposed methodology and characterize the potential for misspecification
when partial state observability is ignored.
