ISP4ML: Understanding the Role of Image Signal Processing in Efficient
  Deep Learning Vision Systems
  Convolutional neural networks (CNNs) are now predominant components in a
variety of computer vision (CV) systems. These systems typically include an
image signal processor (ISP), even though the ISP is traditionally designed to
produce images that look appealing to humans. In CV systems, it is not clear
what the role of the ISP is, or if it is even required at all for accurate
prediction. In this work, we investigate the efficacy of the ISP in CNN
classification tasks, and outline the system-level trade-offs between
prediction accuracy and computational cost. To do so, we build software models
of a configurable ISP and an imaging sensor in order to train CNNs on ImageNet
with a range of different ISP settings and functionality. Results on ImageNet
show that an ISP improves accuracy by 4.6%-12.2% on MobileNet architectures of
different widths. Results using ResNets demonstrate that these trends also
generalize to deeper networks. An ablation study of the various processing
stages in a typical ISP reveals that the tone mapper is the most significant
stage when operating on high dynamic range (HDR) images, by providing 5.8%
average accuracy improvement alone. Overall, the ISP benefits system efficiency
because the memory and computational costs of the ISP is minimal compared to
the cost of using a larger CNN to achieve the same accuracy.
