Structure Regularization for Structured Prediction: Theories and
  Experiments
  While there are many studies on weight regularization, the study on structure
regularization is rare. Many existing systems on structured prediction focus on
increasing the level of structural dependencies within the model. However, this
trend could have been misdirected, because our study suggests that complex
structures are actually harmful to generalization ability in structured
prediction. To control structure-based overfitting, we propose a structure
regularization framework via \emph{structure decomposition}, which decomposes
training samples into mini-samples with simpler structures, deriving a model
with better generalization power. We show both theoretically and empirically
that structure regularization can effectively control overfitting risk and lead
to better accuracy. As a by-product, the proposed method can also substantially
accelerate the training speed. The method and the theoretical results can apply
to general graphical models with arbitrary structures. Experiments on
well-known tasks demonstrate that our method can easily beat the benchmark
systems on those highly-competitive tasks, achieving state-of-the-art
accuracies yet with substantially faster training speed.
