Towards Verified Stochastic Variational Inference for Probabilistic
  Programs
  Probabilistic programming is the idea of writing models from statistics and
machine learning using program notations and reasoning about these models using
generic inference engines. Recently its combination with deep learning has been
explored intensely, leading to the development of deep probabilistic
programming languages such as Pyro. At the core of this development lie
inference engines based on stochastic variational inference algorithms. When
asked to find information about the posterior distribution of a model written
in such a language, these algorithms convert this posterior-inference query
into an optimisation problem and solve it approximately by gradient ascent. In
this paper, we analyse one of the most fundamental and versatile variational
inference algorithms, called score estimator, using tools from denotational
semantics and program analysis. We formally express what this algorithm does on
models denoted by programs, and expose implicit assumptions made by the
algorithm. The violation of these assumptions may lead to an undefined
optimisation objective or the loss of convergence guarantee of the optimisation
process. We then describe rules for proving these assumptions, which can be
automated by static program analyses. Some of our rules use nontrivial facts
from continuous mathematics, and let us replace requirements about integrals in
the assumptions, by conditions involving differentiation or boundedness, which
are much easier to prove automatically. Following our general methodology, we
have developed a static program analysis for Pyro that aims at discharging the
assumption about what we call model-guide support match. Applied to the eight
representative model-guide pairs from the Pyro webpage, our analysis finds a
bug in one of these cases, reveals a non-standard use of an inference engine in
another, and shows the assumptions are met in the remaining cases.
