Differentiable probabilistic models of scientific imaging with the
  Fourier slice theorem
  Scientific imaging techniques such as optical and electron microscopy and
computed tomography (CT) scanning are used to study the 3D structure of an
object through 2D observations. These observations are related to the original
3D object through orthogonal integral projections. For common 3D reconstruction
algorithms, computational efficiency requires the modeling of the 3D structures
to take place in Fourier space by applying the Fourier slice theorem. At
present, it is unclear how to differentiate through the projection operator,
and hence current learning algorithms can not rely on gradient based methods to
optimize 3D structure models. In this paper we show how back-propagation
through the projection operator in Fourier space can be achieved. We
demonstrate the validity of the approach with experiments on 3D reconstruction
of proteins. We further extend our approach to learning probabilistic models of
3D objects. This allows us to predict regions of low sampling rates or estimate
noise. A higher sample efficiency can be reached by utilizing the learned
uncertainties of the 3D structure as an unsupervised estimate of the model fit.
Finally, we demonstrate how the reconstruction algorithm can be extended with
an amortized inference scheme on unknown attributes such as object pose.
Through empirical studies we show that joint inference of the 3D structure and
the object pose becomes more difficult when the ground truth object contains
more symmetries. Due to the presence of for instance (approximate) rotational
symmetries, the pose estimation can easily get stuck in local optima,
inhibiting a fine-grained high-quality estimate of the 3D structure.
