Optimal Control as Variational Inference
In this article we address the stochastic and risk sensitive optimal control
problem probabilistically and decompose and solve the probabilistic models
using principles from variational inference. We demonstrate how this culminates
into two separate probabilistic inference procedures that allow to iteratively
infer the deterministic optimal policy. More formally a sequence of belief
policies, as a probabilistic proxy for the deterministic optimal policy, is
specified through a fixed point iteration with the equilibrium point coinciding
with the deterministic solution. These results re-establish the paradigm of
Control as Inference, a concept explored and exploited originally by the
Reinforcement Learning community anticipating deep rooted connections between
optimal estimation and control. Although the Control as Inference paradigm
already resulted in the development of several Reinforcement Learning
algorithms, until now the underlying mechanism were only partially understood.
For that very reason control as inference has not been well received by the
control community. By exposing the underlying mechanism we aim to contribute to
its general acceptance as a framework superseding optimal control. In order to
exhibit its general relevance we discuss parallels with path integral control
and discuss a wide range of possible applications.