On the Reliability of Test Collections for Evaluating Systems of
  Different Types
  As deep learning based models are increasingly being used for information
retrieval (IR), a major challenge is to ensure the availability of test
collections for measuring their quality. Test collections are generated based
on pooling results of various retrieval systems, but until recently this did
not include deep learning systems. This raises a major challenge for reusable
evaluation: Since deep learning based models use external resources (e.g. word
embeddings) and advanced representations as opposed to traditional methods that
are mainly based on lexical similarity, they may return different types of
relevant document that were not identified in the original pooling. If so, test
collections constructed using traditional methods are likely to lead to biased
and unfair evaluation results for deep learning (neural) systems. This paper
uses simulated pooling to test the fairness and reusability of test
collections, showing that pooling based on traditional systems only can lead to
biased evaluation of deep learning systems.
