Multimodal Material Classification for Robots using Spectroscopy and
  High Resolution Texture Imaging
  Material recognition can help inform robots about how to properly interact
with and manipulate real-world objects. In this paper, we present a multimodal
sensing technique, leveraging near-infrared spectroscopy and close-range high
resolution texture imaging, that enables robots to estimate the materials of
household objects. We release a dataset of high resolution texture images and
spectral measurements collected from a mobile manipulator that interacted with
144 household objects. We then present a neural network architecture that
learns a compact multimodal representation of spectral measurements and texture
images. When generalizing material classification to new objects, we show that
this multimodal representation enables a robot to recognize materials with
greater performance as compared to prior state-of-the-art approaches. Finally,
we present how a robot can combine this high resolution local sensing with
images from the robot's head-mounted camera to achieve accurate material
classification over a scene of objects on a table.
