Structured, flexible, and robust: benchmarking and improving large language models towards more human-like behavior in out-of-distribution reasoning tasks
Human language offers a powerful window into our thoughts -- we tell stories,
give explanations, and express our beliefs and goals through words. Abundant
evidence also suggests that language plays a developmental role in structuring
our learning. Here, we ask: how much of human-like thinking can be captured by
learning statistical patterns in language alone? We first contribute a new
challenge benchmark for comparing humans and distributional large language
models (LLMs). Our benchmark contains two problem-solving domains (planning and
explanation generation) and is designed to require generalization to new,
out-of-distribution problems expressed in language. We find that humans are far
more robust than LLMs on this benchmark. Next, we propose a hybrid
Parse-and-Solve model, which augments distributional LLMs with a structured
symbolic reasoning module. We find that this model shows more robust adaptation
to out-of-distribution planning problems, demonstrating the promise of hybrid
AI models for more human-like reasoning.