Designing off-sample performance metrics
  Modern machine learning systems are traditionally designed and tested with
the overall goal of achieving the best possible performance on average. In this
work, we consider an approach to building learning systems which treats the
question of "how should we quantify good off-sample performance?" as a key
design decision. We describe this proposal using a simple and general
formulation, place the current dominant paradigm within the proper historical
context, and then survey the literature for more recent developments that
depart from tradition and can be viewed as special cases of our proposed
methodology.
