Improving Joint Training of Inference Networks and Structured Prediction
  Energy Networks
  Deep energy-based models are powerful, but pose challenges for learning and
inference (Belanger and McCallum, 2016). Tu and Gimpel (2018) developed an
efficient framework for energy-based models by training "inference networks" to
approximate structured inference instead of using gradient descent. However,
their alternating optimization approach suffers from instabilities during
training, requiring additional loss terms and careful hyperparameter tuning. In
this paper, we contribute several strategies to stabilize and improve this
joint training of energy functions and inference networks for structured
prediction. We design a compound objective to jointly train both cost-augmented
and test-time inference networks along with the energy function. We propose
joint parameterizations for the inference networks that encourage them to
capture complementary functionality during learning. We empirically validate
our strategies on two sequence labeling tasks, showing easier paths to strong
performance than prior work, as well as further improvements with global energy
terms.
