EST: Evaluating Scientific Thinking in Artificial Agents
Theoretical ideas and empirical research have shown us a seemingly surprising
result: children, even very young toddlers, demonstrate learning and thinking
in a strikingly similar manner to scientific reasoning in formal research.
Encountering a novel phenomenon, children make hypotheses against data, conduct
causal inference from observation, test their theory via experimentation, and
correct the proposition if inconsistency arises. Rounds of such processes
continue until the underlying mechanism is found. Towards building machines
that can learn and think like people, one natural question for us to ask is:
whether the intelligence we achieve today manages to perform such a scientific
thinking process, and if any, at what level. In this work, we devise the EST
environment for evaluating the scientific thinking ability in artificial
agents. Motivated by the stream of research on causal discovery, we build our
interactive EST environment based on Blicket detection. Specifically, in each
episode of EST, an agent is presented with novel observations and asked to
figure out all objects' Blicketness. At each time step, the agent proposes new
experiments to validate its hypothesis and updates its current belief. By
evaluating Reinforcement Learning (RL) agents on both a symbolic and visual
version of this task, we notice clear failure of today's learning methods in
reaching a level of intelligence comparable to humans. Such inefficacy of
learning in scientific thinking calls for future research in building humanlike
intelligence.