Program synthesis performance constrained by non-linear spatial
  relations in Synthetic Visual Reasoning Test
  Despite remarkable advances in automated visual recognition by machines, some
visual tasks remain challenging for machines. Fleuret et al. (2011) introduced
the Synthetic Visual Reasoning Test (SVRT) to highlight this point, which
required classification of images consisting of randomly generated shapes based
on hidden abstract rules using only a few examples. Ellis et al. (2015)
demonstrated that a program synthesis approach could solve some of the SVRT
problems with unsupervised, few-shot learning, whereas they remained
challenging for several convolutional neural networks trained with thousands of
examples. Here we re-considered the human and machine experiments, because they
followed different protocols and yielded different statistics. We thus proposed
a quantitative reintepretation of the data between the protocols, so that we
could make fair comparison between human and machine performance. We improved
the program synthesis classifier by correcting the image parsings, and compared
the results to the performance of other machine agents and human subjects. We
grouped the SVRT problems into different types by the two aspects of the core
characteristics for classification: shape specification and location relation.
We found that the program synthesis classifier could not solve problems
involving shape distances, because it relied on symbolic computation which
scales poorly with input dimension and adding distances into such computation
would increase the dimension combinatorially with the number of shapes in an
image. Therefore, although the program synthesis classifier is capable of
abstract reasoning, its performance is highly constrained by the accessible
information in image parsings.
