Forward and Inverse models in HCI:Physical simulation and deep learning
  for inferring 3D finger pose
  We outline the role of forward and inverse modelling approaches in the design
of human--computer interaction systems. Causal, forward models tend to be
easier to specify and simulate, but HCI requires solutions of the inverse
problem. We infer finger 3D position $(x,y,z)$ and pose (pitch and yaw) on a
mobile device using capacitive sensors which can sense the finger up to 5cm
above the screen. We use machine learning to develop data-driven models to
infer position, pose and sensor readings, based on training data from: 1. data
generated by robots, 2. data from electrostatic simulators 3. human-generated
data. Machine learned emulation is used to accelerate the electrostatic
simulation performance by a factor of millions. We combine a Conditional
Variational Autoencoder with domain expertise/models experimentally collected
data. We compare forward and inverse model approaches to direct inference of
finger pose. The combination gives the most accurate reported results on
inferring 3D position and pose with a capacitive sensor on a mobile device.
