{'fraction_non_alphanumeric': 0.02818578801111552, 'fraction_numerical': 0.005160778086542279, 'mean_word_length': 5.395939086294416, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Toward Autonoumous Visual-tactile Exploration and Manipulation

The sense of touch allows humans and higher animals to perform coordinated and efficient interactions within their environment. In last years, the resolution and sensitivity of tactile sensors only sufficed for basic force feedback during blind grasping [1]. However, tactile sensor arrays providing high spatial and temporal resolution as well as high sensitivity [2], [5] emerged recently. We argue, that successful dexterous manipulation strongly depends on tight feedback loops integrating visual and tactile feedback. Due to the lack of appropriate tactile sensor devices, research so far focused mainly on planning-based approaches using few or no feedback at all. However, the advent of new tactile sensor devices asks for new control strategies to exploit this important and valuable sensory channel for grasping and manipulation tasks. In our work, we implemented a control framework to realize a whole set of tactile and visual servoing tasks. This includes such simple tasks like tracking a touched object, maintaining both contact location and contact force, as well as more elaborate tasks like tracking an object’s pose, tactile object exploration, or in-hand manipulation. Aiming to handle unknown objects, all control primitives make as parsimonious assumptions about available prior knowledge as possible: Neither the object properties (shape, weight) nor contact properties (friction coefficients, softness) are assumed to be available. As an example application we consider in-hand manipulation of an unknown object, emphasizing coordinated manipulation motions of all fingers and surface exploration of a single finger for regrasping. In previous work [3] we have shown in-hand manipulation in physics simulation only. The present paper extends this work to real-world experiments using 16×16 tactile sensor arrays as large fingertips mounted on two KUKA LWR arms (see Fig. 1). In the first group of experiments, we demonstrate the following tasks: 1) contact point tracking, 2) tracking of object edge orientation, 3) following of an unknown object edge, and 4) exploring an unknown object surface. The second experiment is a direct transfer of our in-hand manipulation results from simulation to real-world: Visually tracking the object’s pose employing a fiducial marker, we show how the tactile feedback can be exploited to realize robust “dual-finger” object manipulation. All experiments are documented in the accompanying video.