Omnipush: accurate, diverse, real-world dataset of pushing dynamics with
  RGB-D video
  Pushing is a fundamental robotic skill. Existing work has shown how to
exploit models of pushing to achieve a variety of tasks, including grasping
under uncertainty, in-hand manipulation and clearing clutter. Such models,
however, are approximate, which limits their applicability. Learning-based
methods can reason directly from raw sensory data with accuracy, and have the
potential to generalize to a wider diversity of scenarios. However, developing
and testing such methods requires rich-enough datasets. In this paper we
introduce Omnipush, a dataset with high variety of planar pushing behavior. In
particular, we provide 250 pushes for each of 250 objects, all recorded with
RGB-D and a high precision tracking system. The objects are constructed so as
to systematically explore key factors that affect pushing -- the shape of the
object and its mass distribution -- which have not been broadly explored in
previous datasets, and allow to study generalization in model learning.
Omnipush includes a benchmark for meta-learning dynamic models, which requires
algorithms that make good predictions and estimate their own uncertainty. We
also provide an RGB video prediction benchmark and propose other relevant tasks
that can be suited with this dataset.
  Data and code are available at
\url{https://web.mit.edu/mcube/omnipush-dataset/}.
