Salient Phrase Aware Dense Retrieval: Can a Dense Retriever Imitate a
  Sparse One?
  Despite their recent popularity and well-known advantages, dense retrievers
still lag behind sparse methods such as BM25 in their ability to reliably match
salient phrases and rare entities in the query and to generalize to
out-of-domain data. It has been argued that this is an inherent limitation of
dense models. We rebut this claim by introducing the Salient Phrase Aware
Retriever (SPAR), a dense retriever with the lexical matching capacity of a
sparse model. We show that a dense Lexical Model {\Lambda} can be trained to
imitate a sparse one, and SPAR is built by augmenting a standard dense
retriever with {\Lambda}. Empirically, SPAR shows superior performance on a
range of tasks including five question answering datasets, MS MARCO passage
retrieval, as well as the EntityQuestions and BEIR benchmarks for out-of-domain
evaluation, exceeding the performance of state-of-the-art dense and sparse
retrievers.
