SemEval-2021 Task 11: NLPContributionGraph -- Structuring Scholarly NLP
  Contributions for a Research Knowledge Graph
  There is currently a gap between the natural language expression of scholarly
publications and their structured semantic content modeling to enable
intelligent content search. With the volume of research growing exponentially
every year, a search feature operating over semantically structured content is
compelling. The SemEval-2021 Shared Task NLPContributionGraph (a.k.a. 'the NCG
task') tasks participants to develop automated systems that structure
contributions from NLP scholarly articles in the English language. Being the
first-of-its-kind in the SemEval series, the task released structured data from
NLP scholarly articles at three levels of information granularity, i.e. at
sentence-level, phrase-level, and phrases organized as triples toward Knowledge
Graph (KG) building. The sentence-level annotations comprised the few sentences
about the article's contribution. The phrase-level annotations were scientific
term and predicate phrases from the contribution sentences. Finally, the
triples constituted the research overview KG. For the Shared Task,
participating systems were then expected to automatically classify contribution
sentences, extract scientific terms and relations from the sentences, and
organize them as KG triples.
  Overall, the task drew a strong participation demographic of seven teams and
27 participants. The best end-to-end task system classified contribution
sentences at 57.27% F1, phrases at 46.41% F1, and triples at 22.28% F1. While
the absolute performance to generate triples remains low, in the conclusion of
this article, the difficulty of producing such data and as a consequence of
modeling it is highlighted.
