SHOP-VRB: A Visual Reasoning Benchmark for Object Perception
  In this paper we present an approach and a benchmark for visual reasoning in
robotics applications, in particular small object grasping and manipulation.
The approach and benchmark are focused on inferring object properties from
visual and text data. It concerns small household objects with their
properties, functionality, natural language descriptions as well as
question-answer pairs for visual reasoning queries along with their
corresponding scene semantic representations. We also present a method for
generating synthetic data which allows to extend the benchmark to other objects
or scenes and propose an evaluation protocol that is more challenging than in
the existing datasets. We propose a reasoning system based on symbolic program
execution. A disentangled representation of the visual and textual inputs is
obtained and used to execute symbolic programs that represent a 'reasoning
process' of the algorithm. We perform a set of experiments on the proposed
benchmark and compare to results for the state of the art methods. These
results expose the shortcomings of the existing benchmarks that may lead to
misleading conclusions on the actual performance of the visual reasoning
systems.
