Learning Constraints for Structured Prediction Using Rectifier Networks
  Various natural language processing tasks are structured prediction problems
where outputs are constructed with multiple interdependent decisions. Past work
has shown that domain knowledge, framed as constraints over the output space,
can help improve predictive accuracy. However, designing good constraints often
relies on domain expertise. In this paper, we study the problem of learning
such constraints. We frame the problem as that of training a two-layer
rectifier network to identify valid structures or substructures, and show a
construction for converting a trained network into a system of linear
constraints over the inference variables. Our experiments on several NLP tasks
show that the learned constraints can improve the prediction accuracy,
especially when the number of training examples is small.
