Doubly Nested Network for Resource-Efficient Inference
  We propose doubly nested network(DNNet) where all neurons represent their own
sub-models that solve the same task. Every sub-model is nested both layer-wise
and channel-wise. While nesting sub-models layer-wise is straight-forward with
deep-supervision as proposed in \cite{xie2015holistically}, channel-wise
nesting has not been explored in the literature to our best knowledge.
Channel-wise nesting is non-trivial as neurons between consecutive layers are
all connected to each other. In this work, we introduce a technique to solve
this problem by sorting channels topologically and connecting neurons
accordingly. For the purpose, channel-causal convolutions are used. Slicing
doubly nested network gives a working sub-network. The most notable application
of our proposed network structure with slicing operation is resource-efficient
inference. At test time, computing resources such as time and memory available
for running the prediction algorithm can significantly vary across devices and
applications. Given a budget constraint, we can slice the network accordingly
and use a sub-model for inference within budget, requiring no additional
computation such as training or fine-tuning after deployment. We demonstrate
the effectiveness of our approach in several practical scenarios of utilizing
available resource efficiently.
