Synthetic vs. Real Reference Strings for Citation Parsing, and the
  Importance of Re-training and Out-Of-Sample Data for Meaningful Evaluations:
  Experiments with GROBID, GIANT and Cora
  Citation parsing, particularly with deep neural networks, suffers from a lack
of training data as available datasets typically contain only a few thousand
training instances. Manually labelling citation strings is very time-consuming,
hence synthetically created training data could be a solution. However, as of
now, it is unknown if synthetically created reference-strings are suitable to
train machine learning algorithms for citation parsing. To find out, we train
Grobid, which uses Conditional Random Fields, with a) human-labelled reference
strings from 'real' bibliographies and b) synthetically created reference
strings from the GIANT dataset. We find that both synthetic and organic
reference strings are equally suited for training Grobid (F1 = 0.74). We
additionally find that retraining Grobid has a notable impact on its
performance, for both synthetic and real data (+30% in F1). Having as many
types of labelled fields as possible during training also improves
effectiveness, even if these fields are not available in the evaluation data
(+13.5% F1). We conclude that synthetic data is suitable for training (deep)
citation parsing models. We further suggest that in future evaluations of
reference parsers both evaluation data similar and dissimilar to the training
data should be used for more meaningful evaluations.
