The Efficiency Misnomer
  Model efficiency is a critical aspect of developing and deploying machine
learning models. Inference time and latency directly affect the user
experience, and some applications have hard requirements. In addition to
inference costs, model training also have direct financial and environmental
impacts. Although there are numerous well-established metrics (cost indicators)
for measuring model efficiency, researchers and practitioners often assume that
these metrics are correlated with each other and report only few of them. In
this paper, we thoroughly discuss common cost indicators, their advantages and
disadvantages, and how they can contradict each other. We demonstrate how
incomplete reporting of cost indicators can lead to partial conclusions and a
blurred or incomplete picture of the practical considerations of different
models. We further present suggestions to improve reporting of efficiency
metrics.
