ComPhy: Compositional Physical Reasoning of Objects and Events from Videos
Objects' motions in nature are governed by complex interactions and their
properties. While some properties, such as shape and material, can be
identified via the object's visual appearances, others like mass and electric
charge are not directly visible. The compositionality between the visible and
hidden properties poses unique challenges for AI models to reason from the
physical world, whereas humans can effortlessly infer them with limited
observations. Existing studies on video reasoning mainly focus on visually
observable elements such as object appearance, movement, and contact
interaction. In this paper, we take an initial step to highlight the importance
of inferring the hidden physical properties not directly observable from visual
appearances, by introducing the Compositional Physical Reasoning (ComPhy)
dataset. For a given set of objects, ComPhy includes few videos of them moving
and interacting under different initial conditions. The model is evaluated
based on its capability to unravel the compositional hidden properties, such as
mass and charge, and use this knowledge to answer a set of questions posted on
one of the videos. Evaluation results of several state-of-the-art video
reasoning models on ComPhy show unsatisfactory performance as they fail to
capture these hidden properties. We further propose an oracle neural-symbolic
framework named Compositional Physics Learner (CPL), combining visual
perception, physical property learning, dynamic prediction, and symbolic
execution into a unified framework. CPL can effectively identify objects'
physical properties from their interactions and predict their dynamics to
answer questions.