NetCut: Real-Time DNN Inference Using Layer Removal
  Deep Learning plays a significant role in assisting humans in many aspects of
their lives. As these networks tend to get deeper over time, they extract more
features to increase accuracy at the cost of additional inference latency. This
accuracy-performance trade-off makes it more challenging for Embedded Systems,
as resource-constrained processors with strict deadlines, to deploy them
efficiently. This can lead to selection of networks that can prematurely meet a
specified deadline with excess slack time that could have potentially
contributed to increased accuracy.
  In this work, we propose: (i) the concept of layer removal as a means of
constructing TRimmed Networks (TRNs) that are based on removing
problem-specific features of a pretrained network used in transfer learning,
and (ii) NetCut, a methodology based on an empirical or an analytical latency
estimator, which only proposes and retrains TRNs that can meet the
application's deadline, hence reducing the exploration time significantly.
  We demonstrate that TRNs can expand the Pareto frontier that trades off
latency and accuracy to provide networks that can meet arbitrary deadlines with
potential accuracy improvement over off-the-shelf networks. Our experimental
results show that such utilization of TRNs, while transferring to a simpler
dataset, in combination with NetCut, can lead to the proposal of networks that
can achieve relative accuracy improvement of up to 10.43% among existing
off-the-shelf neural architectures while meeting a specific deadline, and 27x
speedup in exploration time.
