Set Prediction without Imposing Structure as Conditional Density
  Estimation
  Set prediction is about learning to predict a collection of unordered
variables with unknown interrelations. Training such models with set losses
imposes the structure of a metric space over sets. We focus on stochastic and
underdefined cases, where an incorrectly chosen loss function leads to
implausible predictions. Example tasks include conditional point-cloud
reconstruction and predicting future states of molecules. In this paper, we
propose an alternative to training via set losses by viewing learning as
conditional density estimation. Our learning framework fits deep energy-based
models and approximates the intractable likelihood with gradient-guided
sampling. Furthermore, we propose a stochastically augmented prediction
algorithm that enables multiple predictions, reflecting the possible variations
in the target set. We empirically demonstrate on a variety of datasets the
capability to learn multi-modal densities and produce different plausible
predictions. Our approach is competitive with previous set prediction models on
standard benchmarks. More importantly, it extends the family of addressable
tasks beyond those that have unambiguous predictions.
