Phy-Q: A Testbed for Physical Reasoning
  Humans are well-versed in reasoning about the behaviors of physical objects
and choosing actions accordingly to accomplish tasks, while it remains a major
challenge for AI. To facilitate research addressing this problem, we propose a
new testbed that requires an agent to reason about physical scenarios and take
an action appropriately. Inspired by the physical knowledge acquired in infancy
and the capabilities required for robots to operate in real-world environments,
we identify 15 essential physical scenarios. For each scenario, we create a
wide variety of distinct task templates, and we ensure all the task templates
within the same scenario can be solved by using one specific strategic physical
rule. By having such a design, we evaluate two distinct levels of
generalization, namely the local generalization and the broad generalization.
We conduct an extensive evaluation with human players, learning agents with
varying input types and architectures, and heuristic agents with different
strategies. Inspired by how human IQ is calculated, we define the physical
reasoning quotient (Phy-Q score) that reflects the physical reasoning
intelligence of an agent. Our evaluation shows that 1) all agents are far below
human performance, and 2) learning agents, even with good local generalization
ability, struggle to learn the underlying physical reasoning rules and fail to
generalize broadly. We encourage the development of intelligent agents that can
reach the human level Phy-Q score. Website: https://github.com/phy-q/benchmark
