The BOSARIS Toolkit: Theory, Algorithms and Code for Surviving the New
  DCF
  The change of two orders of magnitude in the 'new DCF' of NIST's SRE'10,
relative to the 'old DCF' evaluation criterion, posed a difficult challenge for
participants and evaluator alike. Initially, participants were at a loss as to
how to calibrate their systems, while the evaluator underestimated the required
number of evaluation trials. After the fact, it is now obvious that both
calibration and evaluation require very large sets of trials. This poses the
challenges of (i) how to decide what number of trials is enough, and (ii) how
to process such large data sets with reasonable memory and CPU requirements.
After SRE'10, at the BOSARIS Workshop, we built solutions to these problems
into the freely available BOSARIS Toolkit. This paper explains the principles
and algorithms behind this toolkit. The main contributions of the toolkit are:
1. The Normalized Bayes Error-Rate Plot, which analyses likelihood- ratio
calibration over a wide range of DCF operating points. These plots also help in
judging the adequacy of the sizes of calibration and evaluation databases. 2.
Efficient algorithms to compute DCF and minDCF for large score files, over the
range of operating points required by these plots. 3. A new score file format,
which facilitates working with very large trial lists. 4. A faster logistic
regression optimizer for fusion and calibration. 5. A principled way to define
EER (equal error rate), which is of practical interest when the absolute error
count is small.
