Runtime Deep Model Multiplexing for Reduced Latency and Energy
  Consumption Inference
  We propose a learning algorithm to design a light-weight neural multiplexer
that given the input and computational resource requirements, calls the model
that will consume the minimum compute resources for a successful inference.
Mobile devices can use the proposed algorithm to offload the hard inputs to the
cloud while inferring the easy ones locally. Besides, in the large scale
cloud-based intelligent applications, instead of replicating the most-accurate
model, a range of small and large models can be multiplexed from depending on
the input's complexity which will save the cloud's computational resources. The
input complexity or hardness is determined by the number of models that can
predict the correct label. For example, if no model can predict the label
correctly, then the input is considered as the hardest. The proposed algorithm
allows the mobile device to detect the inputs that can be processed locally and
the ones that require a larger model and should be sent a cloud server.
Therefore, the mobile user benefits from not only the local processing but also
from an accurate model hosted on a cloud server. Our experimental results show
that the proposed algorithm improves mobile's model accuracy by 8.52% which is
because of those inputs that are properly selected and offloaded to the cloud
server. In addition, it saves the cloud providers' compute resources by a
factor of 2.85x as small models are chosen for easier inputs.
