Integrated Benchmarking and Design for Reproducible and Accessible
  Evaluation of Robotic Agents
  As robotics matures and increases in complexity, it is more necessary than
ever that robot autonomy research be reproducible. Compared to other sciences,
there are specific challenges to benchmarking autonomy, such as the complexity
of the software stacks, the variability of the hardware and the reliance on
data-driven techniques, amongst others. In this paper, we describe a new
concept for reproducible robotics research that integrates development and
benchmarking, so that reproducibility is obtained "by design" from the
beginning of the research/development processes. We first provide the overall
conceptual objectives to achieve this goal and then a concrete instance that we
have built: the DUCKIENet. One of the central components of this setup is the
Duckietown Autolab, a remotely accessible standardized setup that is itself
also relatively low-cost and reproducible. When evaluating agents, careful
definition of interfaces allows users to choose among local versus remote
evaluation using simulation, logs, or remote automated hardware setups. We
validate the system by analyzing the repeatability of experiments conducted
using the infrastructure and show that there is low variance across different
robot hardware and across different remote labs.
