Orthogonal Attention: A Cloze-Style Approach to Negation Scope
  Resolution
  Negation Scope Resolution is an extensively researched problem, which is used
to locate the words affected by a negation cue in a sentence. Recent works have
shown that simply finetuning transformer-based architectures yield
state-of-the-art results on this task. In this work, we look at Negation Scope
Resolution as a Cloze-Style task, with the sentence as the Context and the cue
words as the Query. We also introduce a novel Cloze-Style Attention mechanism
called Orthogonal Attention, which is inspired by Self Attention. First, we
propose a framework for developing Orthogonal Attention variants, and then
propose 4 Orthogonal Attention variants: OA-C, OA-CA, OA-EM, and OA-EMB. Using
these Orthogonal Attention layers on top of an XLNet backbone, we outperform
the finetuned XLNet state-of-the-art for Negation Scope Resolution, achieving
the best results to date on all 4 datasets we experiment with: BioScope
Abstracts, BioScope Full Papers, SFU Review Corpus and the *sem 2012 Dataset
(Sherlock).
