Complex Structure Leads to Overfitting: A Structure Regularization
  Decoding Method for Natural Language Processing
  Recent systems on structured prediction focus on increasing the level of
structural dependencies within the model. However, our study suggests that
complex structures entail high overfitting risks. To control the
structure-based overfitting, we propose to conduct structure regularization
decoding (SR decoding). The decoding of the complex structure model is
regularized by the additionally trained simple structure model. We
theoretically analyze the quantitative relations between the structural
complexity and the overfitting risk. The analysis shows that complex structure
models are prone to the structure-based overfitting. Empirical evaluations show
that the proposed method improves the performance of the complex structure
models by reducing the structure-based overfitting. On the sequence labeling
tasks, the proposed method substantially improves the performance of the
complex neural network models. The maximum F1 error rate reduction is 36.4% for
the third-order model. The proposed method also works for the parsing task. The
maximum UAS improvement is 5.5% for the tri-sibling model. The results are
competitive with or better than the state-of-the-art results.
