Pairwise Neural Networks (PairNets) with Low Memory for Fast On-Device
  Applications
  A traditional artificial neural network (ANN) is normally trained slowly by a
gradient descent algorithm, such as the backpropagation algorithm, since a
large number of hyperparameters of the ANN need to be fine-tuned with many
training epochs. Since a large number of hyperparameters of a deep neural
network, such as a convolutional neural network, occupy much memory, a
memory-inefficient deep learning model is not ideal for real-time Internet of
Things (IoT) applications on various devices, such as mobile phones. Thus, it
is necessary to develop fast and memory-efficient Artificial Intelligence of
Things (AIoT) systems for real-time on-device applications. We created a novel
wide and shallow 4-layer ANN called "Pairwise Neural Network" ("PairNet") with
high-speed non-gradient-descent hyperparameter optimization. The PairNet is
trained quickly with only one epoch since its hyperparameters are directly
optimized one-time via simply solving a system of linear equations by using the
multivariate least squares fitting method. In addition, an n-input space is
partitioned into many n-input data subspaces, and a local PairNet is built in a
local n-input subspace. This divide-and-conquer approach can train the local
PairNet using specific local features to improve model performance. Simulation
results indicate that the three PairNets with incremental learning have smaller
average prediction mean squared errors, and achieve much higher speeds than
traditional ANNs. An important future work is to develop better and faster
non-gradient-descent hyperparameter optimization algorithms to generate
effective, fast, and memory-efficient PairNets with incremental learning on
optimal subspaces for real-time AIoT on-device applications.
