SensiX++: Bringing MLOPs and Multi-tenant Model Serving to Sensory Edge
  Devices
  We present SensiX++ - a multi-tenant runtime for adaptive model execution
with integrated MLOps on edge devices, e.g., a camera, a microphone, or IoT
sensors. SensiX++ operates on two fundamental principles - highly modular
componentisation to externalise data operations with clear abstractions and
document-centric manifestation for system-wide orchestration. First, a data
coordinator manages the lifecycle of sensors and serves models with correct
data through automated transformations. Next, a resource-aware model server
executes multiple models in isolation through model abstraction, pipeline
automation and feature sharing. An adaptive scheduler then orchestrates the
best-effort executions of multiple models across heterogeneous accelerators,
balancing latency and throughput. Finally, microservices with REST APIs serve
synthesised model predictions, system statistics, and continuous deployment.
Collectively, these components enable SensiX++ to serve multiple models
efficiently with fine-grained control on edge devices while minimising data
operation redundancy, managing data and device heterogeneity, reducing resource
contention and removing manual MLOps. We benchmark SensiX++ with ten different
vision and acoustics models across various multi-tenant configurations on
different edge accelerators (Jetson AGX and Coral TPU) designed for sensory
devices. We report on the overall throughput and quantified benefits of various
automation components of SensiX++ and demonstrate its efficacy to significantly
reduce operational complexity and lower the effort to deploy, upgrade,
reconfigure and serve embedded models on edge devices.
