Diagonal Memory Optimisation for Machine Learning on Micro-controllers
  As machine learning spreads into more and more application areas, micro
controllers and low power CPUs are increasingly being used to perform inference
with machine learning models. The capability to deploy onto these limited
hardware targets is enabling machine learning models to be used across a
diverse range of new domains. Optimising the inference process on these targets
poses different challenges from either desktop CPU or GPU implementations,
where the small amounts of RAM available on these targets sets limits on size
of models which can be executed. Analysis of the memory use patterns of eleven
machine learning models was performed. Memory load and store patterns were
observed using a modified version of the Valgrind debugging tool, identifying
memory areas holding values necessary for the calculation as inference
progressed. These analyses identified opportunities optimise the memory use of
these models by overlapping the input and output buffers of individual tensor
operations. Three methods are presented which can calculate the safe overlap of
input and output buffers for tensor operations. Ranging from a computationally
expensive approach with the ability to operate on compiled layer operations, to
a versatile analytical solution which requires access to the original source
code of the layer. The diagonal memory optimisation technique is described and
shown to achieve memory savings of up to 34.5% when applied to eleven common
models. Micro-controller targets are identified where it is only possible to
deploy some models if diagonal memory optimisation is used.
