Learning Intuitive Physics with Multimodal Generative Models
  Predicting the future interaction of objects when they come into contact with
their environment is key for autonomous agents to take intelligent and
anticipatory actions. This paper presents a perception framework that fuses
visual and tactile feedback to make predictions about the expected motion of
objects in dynamic scenes. Visual information captures object properties such
as 3D shape and location, while tactile information provides critical cues
about interaction forces and resulting object motion when it makes contact with
the environment. Utilizing a novel See-Through-your-Skin (STS) sensor that
provides high resolution multimodal sensing of contact surfaces, our system
captures both the visual appearance and the tactile properties of objects. We
interpret the dual stream signals from the sensor using a Multimodal
Variational Autoencoder (MVAE), allowing us to capture both modalities of
contacting objects and to develop a mapping from visual to tactile interaction
and vice-versa. Additionally, the perceptual system can be used to infer the
outcome of future physical interactions, which we validate through simulated
and real-world experiments in which the resting state of an object is predicted
from given initial conditions.
