{'fraction_non_alphanumeric': 0.023818561396849496, 'fraction_numerical': 0.008567632784751783, 'mean_word_length': 5.126318412092019, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Towards Automated Sample Collection and Return in Extreme Underwater Environments

In this report, we present the system design, operational strategy, and results of coordinated multi-vehicle field demonstrations of autonomous marine robotic technologies in search-for-life missions within the Pacific shelf margin of Costa Rica and the Santorini-Kolumbo caldera complex, which serve as analogs to environments that may exist in oceans beyond Earth. This report focuses on the automation of ROV manipulator operations for targeted biological sample-collection-and-return from the seafloor. In the context of future extraterrestrial exploration missions to ocean worlds, an ROV is an analog to a planetary lander, which must be capable of high-level autonomy. Our field trials involve two underwater vehicles, the SuBastian ROV and the Nereid Under Ice (NUI) hybrid ROV for mixed initiative (i.e., teleoperated or autonomous) missions, both equipped 7-DoF hydraulic manipulators. We describe an adaptable, hardware-independent computer vision architecture that enables high-level automated manipulation. The vision system provides a 3D understanding of the workspace to inform manipulator motion planning in complex unstructured environments. We demonstrate the effectiveness of the vision system and control framework through field trials in increasingly challenging environments, including the automated collection and return of biological samples from within the active undersea volcano, Kolumbo. Based on our experiences in the field, we discuss the performance of our system and identify promising directions for future research.


Introduction
A growing body of evidence suggests that the Earth is not unique in containing liquid water (Lewis, 1971;Gaeman et al., 2012;Malin and Edgett, 2000;Khurana et al., 1998), an essential ingredient for carbon-based life. Recent indications of water geysers emanating from moons of Saturn and Jupiter, including Enceladus (Nimmo et al., 2007) and Europa (Arnold et al., 2019), suggest that they may contain subsurface oceans with active hydrothermal venting (Lowell and DuBose, 2005;Hsu et al., 2015). Here on Earth, ocean floor hydrothermal systems and cold seep sites have long been known to host diverse chemosynthetic ecosystems that rely on the redox potentials of deep Earth fluids emitted from these sites to derive biochemical energy (Jannasch and Wirsen, 1979;Brooks et al., 1987), and may serve as analogs for oases of life elsewhere in our solar system and beyond. However, exploration for life within the distant oceans of Europa and Enceladus remains a daunting technological challenge. Robotic submersible vehicles equipped with manipulators provide a practical means for sample analysis and collection, enabling flexibility and dexterity without requiring precise and energetically costly positioning of the vehicle. Planetary landers such as the Mars Rovers have historically relied on human teleoperated manipulation using manually generated scripts (Lehner et al., 2018;Fong et al., 2001;Leger et al., 2005) to collect samples. However, teleoperation of robotic subsea vehicles within these putative ocean worlds is impractical because of high communication latencies (e.g., on the order of an hour for Europa). Thus, robotic missions must be capable of fully automated manipulation.
Marine robotic platforms such as remotely operated vehicles (ROVs) equipped with manipulators provide a useful testbed to develop automated manipulation and sampling technologies as analogs for space missions. Although Earth's gravitational constant is higher than Europa and Enceladus, these moons' estimated ice thicknesses of up to 30 km (Iess et al., 2014;Billings and Kattenhorn, 2005) are expected to present operational challenges, such as extreme pressure, near-freezing temperatures, and corrosion that are similar to Earth's deep ocean environments. While autonomous underwater vehicles (AUVs) have been used for under-ice surveys for nearly 50 years (Francois and Nodland, 1972), deep ocean missions that require sample collection and return using manipulator arms are generally conducted using ROVs under direct human piloted control with cable-tethered communication. Only limited attempts at autonomous manipulation have been made in natural ocean environments (Marani et al., 2009;Shim et al., 2010;De Novi et al., 2010;Sanz et al., 2013;Peñalver et al., 2015;Sivčev et al., 2018b). The comparative lag in subsea manipulator autonomy behind terrestrial systems can be at least partly attributed to commercial systems being historically designed for direct teleoperation, with limited command modes and feedback, low control loop frequency, and poor repeatability Sivčev et al. (2018a). Despite these challenges, we demonstrate an automation framework that is compatible with existing commercial manipulator systems and that automates many high level tasks, while reducing risk through visual based scene understanding and pilot supervision.
In this paper, we consider the challenge of automated subsea manipulation and sample collection using existing ROV platforms as a technology analog for an under-ice exploration missions to Europa or Enceladus. We discuss the challenges that deep seafloor environments pose to automated robotic intervention and propose an architecture that overcomes many of these challenges. The system that we describe can be integrated on existing ROVs with minimal hardware requirements, namely, a vehicle-mounted stereo camera and a manipulator-mounted fisheye camera. We investigate the practical use of our perception methods to estimate the vehicle configuration, dynamically localize tools, and ground the transform between the natural scene reconstruction and the structured vehicle workspace. The manipulator control and vision processes that serve as the basis of this automation framework can be readily adapted to a variety of hardware configurations, making them suitable for a wide range of robotic platforms, including space flight systems. We demonstrate the flexibility of this framework through separate field trials performed with two different classes of ROVs equipped with substantively different manipulators. Figure 1 shows a conceptual diagram of how our system integrates with an ROV. In the current system implementation, a topside machine performs all processing using camera and manipulator data streamed from the vehicle over a high-bandwidth tether.
We conducted testing and field trials in progressively challenging environments, initially in laboratory settings and tank testing, followed by 11 dive missions at the Central American Pacific shelf margin of Costa Rica Figure 1: Conceptual graphic of the our control system for an underwater intervention vehicle. The autonomy system runs on a topside desktop computer with visual sensor data and manipulator coms streamed over a high bandwidth tether from the vehicle. Solid red flow lines represent standard teleoperated control from a surface ship. Blue flow lines represent our automated system. Red dashed lines represent interfacing between the pilot and the autonomous system, where, in this work, the pilot acts as the high level task planner and interfaces with the automated system through a graphical scene representation and task level controller. Eventually, the pilot would be replaced with an automated mission planner that could issue high level tasks.
to operational depths of approximately 1800 m. This area of the Costa Rican accretionary prism is a well studied region with localized ocean floor fluid expulsion sites that host diverse assemblages of extremophile organisms (Hensen et al., 2004;Krause et al., 2014;Levin et al., 2015;Sahling et al., 2008;Silver et al., 2000). Following the completion of the Costa Rica expedition, our team conducted a series of five dive missions to depths of 500 m within the potentially hazardous craters of the Kolumbo and Santorini calderas. These sites of active volcanism contain localized areas of high-temperature hydrothermal venting causing environmental hypercapnia (Camilli et al., 2015;Carey et al., 2013), which host non-calcifying chemosynthetic organisms that may resemble those that arose early on Earth, prior to the advent of its oxidizing atmosphere.
The paper is organized as follows. We begin in Section 2 with an overview of previous work on automated underwater manipulation. Section 3 briefly outlines our strategy for mission planning and operations, and describes in detail the architecture of our automated manipulation system. Section 4 examines the results of experimental missions during field demonstrations using the SuBastian ROV (Campbell et al., 2019) and Nereid Under Ice (NUI) Hybrid Remotely Operated Vehicle (HROV) (Bowen et al., 2014). Section 5 draws on these field results and experiences to discuss advances as well as the limitations and potential failure modes of our perception and control methods and examines how this research may help to advance both automated ROV operations here on Earth and future space flight missions to explore for life within ocean worlds elsewhere. Section 6 identifies promising directions for future research.

Background
There is a rich body of literature on underwater vehicle manipulator system (UVMS) control. This section provides a brief review of the work most related to our approach which have demonstrated their methods in experimental trials. For a thorough discussion of the prior work on UVMS systems, we refer the reader to Sivčev et al. (2018a). Ishimi et al. (1991) and Broome et al. (1995) describe some of the pioneering work on automating UVMSs, where demonstrations included 3D graphical renderings of an ROV's configuration and workspace, real-time visualization of manipulator motion plans, and Cartesian space end-effector control. Yann et al. (2005) demonstrated free-floating stereovision based servoing of the Twin-Burger 2 AUV outfitted with an unconventional 2-DoF manipulator to collect vacuum samples of free-floating targets in a pool. More recent works under the large-scale research projects RAUVI (De Novi et al., 2010), TRIDENT (Sanz et al., 2013), TRI-TON (Peñalver et al., 2015), PANDORA (Cieslak et al., 2015), and MARIS (Simetti et al., 2017) focus on tightly coupled control of the 140 kg displacement Girona 500 AUV outfitted with a customized electric manipulator to perform free-floating intervention tasks. The PANDORA project explores the ability to learn the vehicle and manipulator trajectories by demonstration. The other projects combine vehicle and manipulator motion generation under a task priority framework, where the manipulator control law is a function of the vehicle velocity. Building on these works, the MERBOTS project (Youakim et al., 2017) offers a significant advancement towards automated UVMS control by integrating the ROS-based MoveIt! motion planning framework with the intervention AUV to generate combined vehicle and manipulator motion trajectories in Cartesian space for free-floating intervention tasks. In very recent work, Nishida et al. (2019) demonstrated free-floating autonomous sample collection of shells from the seafloor using the 400 kg displacement Tuna-Sand2 AUV outfitted with a custom extendable suction device. While this body of work provides key advancements towards automated free-floating intervention, limitations make it difficult for many actively operated UVMSs to adopt these methods. Among them, integrating such a tightly coupled control system with existing UVMS platforms would require significant modification to the software architecture, which is particularly problematic for commercial systems. Additionally, the dynamic coupling effect between the vehicle and manipulator during free-floating intervention can strongly affect the trajectory tracking performance, necessitating very slow actuation of the vehicle and manipulator. Lastly, this control approach is designed for high-precision electric manipulators that support velocity-based control, whereas most manipulators on operational UVMS platforms are hydraulic and support only position set point commands with limited precision and repeatability.
Hydraulic manipulators have orders of magnitude higher power-to-weight ratios compared to their electric counterparts and are generally more reliable, making them the manipulator of choice for commercial ROV systems. Though recent commercial electric manipulators have entered the market, their significantly higher power requirements make them practical only for ROVs that have power supplied over a tether. For vehicles like the NUI HROV, which carries all power onboard, low-power hydraulic manipulators remain the most practical choice. However, the limited precision and feedback of hydraulic manipulators present challenges for automation, and little work exists that addresses these challenges. Hildebrandt et al. (2009) demonstrated precision control of a hydraulic manipulator to plug a deep-sea connector. Shim et al. (2010) perform preprogrammed motion following and operator control of a hydraulic work class manipulator. Using Cartesian space end-effector control, they demonstrate operator-guided push-core sampling in the deep ocean. Zhang et al. (2019) perform visual servoing and target grasping with a custom 7-DoF hydraulic manipulator. Sivčev et al. (2018b) demonstrate impressive visual servoing of a working class hydraulic manipulator using position-based control, with feedback provided by fiducials detected from a wrist-mounted camera. Their results include grasping and turning T-bar valves and tracking targets in motion with the end-effector.
Our system builds on these prior approaches to UVMS control, where we demonstrate the effective integration of the MoveIt! motion planning framework (Coleman et al., 2014) with a work class ROV manipulator system for automated planning and control in obstructed scenes. We take a decoupled approach to manipulator control that assumes the vehicle holds station (i.e., rests on the bottom) during the manipulation task. This assumption is motivated by the goal of having the system widely transferable among existing ROV systems. This decoupled approach enables our manipulator control system to be integrated externally from the existing UVMS control systems, providing high-level autonomy with flexibility to be integrated onto a wide array of ROV classes and manipulator arms, including both electric and hydraulic systems.
Important to automating UVMSs are the problems of visual scene understanding and target localization, whether the target be a tool to grasp, a valve to turn, or a sample location in an unstructured environment. Subsea perception is a particularly challenging problem for a number of reasons: turbidity degrades image quality; evenly lighting the scene is very difficult; variable wavelength-dependent absorption and scattering properties of the water column attenuate light and reduce color and photometric contrast; and gathering underwater datasets for developing computer vision methods is expensive. Despite these challenges, computer vision remains the primary means of performing target localization for automated UVMS platforms. Most prior works on UVMS automation rely on fiducials or known geometric shapes that retain high contrast underwater. Marani et al. (2009) use large spherical markers attached to a target and a circle shape edge detector algorithm to localize the marker from a video feed. Sanz et al. (2013) localize a black box object on a harbor seabed by first constructing a visual mosaic from a pre-intervention survey dive with a downwardfacing stereo camera, and then matching an image template of the black box to the mosaic. Cieslak et al. (2015) localize a known panel during intervention operations by registering interest points against a template image. They then estimate the orientation of valves on the panel based on edge detection. Peñalver et al. (2015) and Youakim et al. (2017) use fiducial markers to localize a panel with a priori known relative positions of the turn valves and connector plugs. Youakim et al. (2017) also use fiducials on the end-effector of the manipulator to update the manipulator calibration in real-time. Simetti et al. (2017) use color and geometric shape segmentation of RGB images to detect the pose of a cylindrical pipe of known size. Under the DexROV project, Birk et al. (2018) process stereo point clouds into a 3D occupancy map, while also using fiducial markers to detect and localize a panel with known structure that was projected into the planning scene.
Building on the long history of fiducials as a robust visual cue for underwater computer vision methods, our work extends the use of fiducials to detect the pose of graspable tools carried on-board the ROV, estimate dynamic vehicle configurations in real-time, and ground the relative reference frames in the planning environment. We demonstrate the use of fiducials in a way that is practical for field deployments with an underlying vision system that can effectively localize tools and target objects within the workspace, as well as reconstruct the workspace for obstructed motion planning. Fiducials also enable the collection of annotated image datasets in natural deep seabed environments that support the development of advanced perception methods for scene reconstruction, and target detection and localization.

System Overview
The following sections provide an overview of the different components of our system, including the mission architecture for field operations and the methods for perception and control.

Mission and Vehicle Platform Architecture
Field demonstration and validation include two research cruises, conducted east of the Cocos and Caribbean tectonic subduction zone along Central America's Pacific continental margin (9.0 N 84.5 W), and within the Kolumbo and Santorini Calderas of the Hellenic volcanic arc in the southern Aegean Sea (36.52 N 25.48 E and 36.45 N 25.39 E,respectively). The sites, which are known to host oases of chemosynthetic communities associated with hydrothermal and seafloor hydrocarbon seeps, were chosen as NASA TRL-6 demonstration locations for analog astrobiology exploration missions. These campaigns utilized a sequentially nested survey method with a coordinated team of heterogeneous robotic platforms that relied on automated planing tools to rapidly synthesize vehicle missions in response to newly acquired information (Vrolijk et al., 2021;Ayton et al., 2019). To better approximate an analog space flight mission scenario, surface ships operated as orbiters, conducting multibeam sonar bathymetric mapping of the Pacific (Vrolijk et al., 2019) and Aegean  campaign sites, with coverage areas of 2.000 km 2 at 30 m resolution and 48 km 2 at 10 m resolution, respectively. These maps informed the mission planning for autonomous underwater gliders (AUG), which acted as long-range in-situ reconnaissance drones, conceptually similar to NASA's Ingenuity and Dragonfly vehicles, conducting reconnaissance missions of between 1 km and 500 km in length at standoff distances to within 15 m of ocean floor obstacles in order to identify potential areas of scientific interest (Vrolijk et al., 2021;Duguid and Camilli, 2020). Automated AUG mission planning considered resource (e.g., time and power) and risk constraints (Timmons et al., 2016(Timmons et al., , 2019, and adaptively replanned missions based on inferred sites of scientific interest that correlated with the presence of active ocean floor hydrocarbon cold seeps and hydrothermal vents. Using information gained by the surface ship sonar and AUG missions, the automated planning process then generated viable mission sequences that the ROV used to investigate areas of highest estimated information gain (Vrolijk et al., 2021). During these missions, the ROV acted as a lander, outfitted with a manipulator for automated sample collection and return. The hazardous deep ocean environments explored as part of these ROV missions are considered probable analogs for environments ( Fig. 2) that may exist on other ocean worlds. The two ROVs used for these demonstration campaigns, SuBastian and NUI are substantially different in design and purpose (Table 1). SuBastian is an exemplar of modern deep ocean work class ROVs, with its power, communications, and navigation net provided via an armored cable by its attendant surface ship, the R/V Falkor. SuBastian is equipped with twin 7-DoF Schilling Titan-4 hydraulic manipulator arms (Schilling Robotics, Davis, California) and is a fully teleoperated vehicle that can operate at horizontal excursions of up to 500 m laterally from the R/V Falkor. In contrast, NUI is a HROV that relies on its own battery power and uses an un-armored fiber optic link (roughly the diameter of a human hair) for optional communication with an attendant surface ship, and can operate as both an ROV and an AUV. When in tethered ROV mode, NUI 's power and telemetry architecture enables lateral excursions of up to 20 km from the attendant surface ship. To aid hydrodynamic efficiency, NUI has articulating bow doors that can be closed and act as a fairing during transits and AUV missions. In contrast to SuBastian's twin Titan-4 architecture, which is configured to maximize ROV work area and dexterity, NUI's starboard door is equipped with a single, custom 7-DoF hydraulic manipulator (Kraft Telerobotics, Overland Park, Kansas) that is optimized for energy efficiency. This emphasis on efficiency comes at the expense of reductions in available payload and work space, the   usable range of motion, control precision, lighting field, and available viewing perspectives.

Perception
A vehicle capable of automated intervention must have an effective means to self-localize within the environment and visually reconstruct the workspace to complete the mission tasks. We adopt a vision system consisting of computer vision cameras, that takes into consideration three primary criteria. First, the system must be capable of generating a 3D reconstruction of the manipulator workspace, enabling the motion planner to avoid obstacles and generate safe, collision-free paths. Second, the system must be able to localize a set of known objects, such as tools, and guide the manipulator to grasp them. Third, the system must easily integrate with existing robotic platforms. Our vision system is composed of a vehicle chassis-mounted stereo camera pair with a fixed-baseline and a manipulator wrist-mounted fisheye camera (Fig. 3). The stereo pair observes the manipulator workspace, including part of the tool tray and the scene working area. The system uses the stereo to generate 3D point clouds of the workspace for scene reconstruction, assist with localizing tools in the tool tray, and visually track dynamic vehicle reference frames that are otherwise not observable (e.g., the position of the NUI HROV doors). The wrist-mounted fisheye camera provides a wide-angle view of the scene, and is used to detect objects and acquire dynamic viewpoints of the scene, which may be occluded or outside the field-of-view of the stereo pair. The wide field-of-view of the fisheye compared to a perspective camera enables clear views of objects and scene context at both close and far range ( Fig. 4), which is advantageous for manipulation.
All three cameras are Blackfly model BFLY-PGE-50S5C-C (FLIR, Wilsonville, OR). The stereo cameras use the VS Technology SV-0614H 6 mm f/1.4 lens (VS Technology Corporation, Tokyo, JP), and the fisheye lens is the Fujinon FE185C086HA-1 2.7 mm f/1.8 (Fujinon, Tokyo, JP). The camera housings are custom fabricated with titanium shells and dome viewports (Sexton Corporation, Salem, OR), with a depth rating of 6000 m. A hardware trigger synchronizes the cameras. We calibrate the cameras using images of a checkerboard that the ROV manipulator moves throughout each camera's field-of-view while the vehicle is submerged. We calibrate the stereo cameras using the ROS stereo camera calibration package. We calibrate the fisheye camera using the Kalibr toolbox (Kannala and Brandt, 2006). Because the usable field-of-view for the fisheye camera is less than 180 • due to occlusions from the housing, we use the pinhole projection model with equidistant distortion. We verify that both the stereo and fisheye calibrations achieve sub-pixel reprojection errors for the checkerboard corners.

Tool Handle Pose Estimation
Tools carried by the ROV must be localized by the vision system before they can be grasped. It is general practice in ROV operations to use a single type of handle on every tool to provide consistency for ROV pilots. Given a known type of tool and its model, the vision system need only localize the handle for a tool to be grasped and manipulated.
Using data collected with our vision system during the field trials, we developed a novel deep learningbased method, SilhoNet (Billings and Johnson-Roberson, 2019) and SilhoNet-Fisheye (Billings and Johnson-Roberson, 2020), that estimates the pose of tool handles detected from the wrist-mounted fisheye camera, without the need for fiducials. SilhoNet uses an intermediate silhouette representation to regress the detected object poses. This silhouette representation improves pose regression performance and facilitates training the network on synthetic data, which is especially beneficial when real training data is limited, as is the case for underwater environments. This method achieves promising results on the recorded datasets, but was not ready for integration with the system during the field trials.
During our field demonstrations, we relied on AprilTag markers (Olson, 2011) to localize the tool handles.
Our choice of the AprilTag marker was motivated by the results of dos Santos Cesar et al. (2015), which show that AprilTags yield the best performance in underwater environments, with the lowest sensitivity to turbidity and variable lighting conditions in comparison to other popular fiducial markers. In this study, the minimum marker size detectable in an image was approximately 20 pixels, which, for the 50 × 50 mm markers used in our system, equates to an expected maximum detection range of approximately 1.0 m for the  Figure 4: A comparison of (top) the full view of the wrist-mounted fisheye camera in an underwater scene at close and far range compared to (bottom) a 60 • perspective rectification, which illustrates the significant increase in the field-of-view provided by a fisheye lens compared to a conventional perspective lens. This increased field-of-view provides significantly better contextual awareness to the vision and manipulation systems, especially when working at close range to the target, which is typical for manipulation tasks.
fisheye camera and 2.4 m for the stereo cameras. These distances are within the typical working ranges of the manipulators used in our demonstrations. We designed 3D printed mounts that screw onto the t-handle bases and hold AprilTag vinyl stickers (Fig. 5, right).
We use the ROS TagSLAM package (Pfrommer and Daniilidis, 2019) to detect the fiducials from the wristmounted fisheye camera. TagSLAM is built on the GTSAM (Dellaert, 2012) factor graph library and uses the ISAM2 (Kaess et al., 2012) incremental optimizer for efficient run-time performance. TagSLAM operates in a transform tree completely separate from the world planning environment. Within the TagSLAM environment, the fisheye camera is set as the origin, while the tools with the tag mounts are set as dynamic objects. We optimize the pose of each detected tool with respect to the fisheye camera frame using TagSLAM.  Figure 5: A single type of t-handle was used to manipulate the different tools. The vision system localizes the t-handles using (a) AprilTags affixed to 3D-printed mounts located beneath the t-handle. These tags are detected in (b) images of the ROV tool tray from the wrist-mounted fisheye camera.
The optimized tool pose with respect to the fisheye frame is projected into the world frame through the manipulator kinematics. If the fisheye camera loses sight of a tool, the tool pose within the world scene remains static until the tool is tracked again with TagSLAM.

Control
While many existing methods tightly couple vehicle and manipulator motion planning and control, our approach decouples the manipulator and imaging system from other systems on the ROV. This makes it easier to integrate the system with different ROVs and also minimizes risk to the vehicle, as the automation system runs independently of the vehicle's software stack. This approach also mimics standard ROV operation procedures, in which one pilot controls the vehicle while another pilot controls the manipulator. Our system seeks to replace the direct pilot control of the manipulator with a high-level automation interface that naturally integrates with standard ROV operational procedures. A current limitation of this control approach is a fixed-base assumption while the manipulator is activated. During manipulator operations, the ROV is assumed to be set down on the seabed and essentially acts as a fixed-base manipulator platform during a sampling tasks. When a manipulator command is executed, our system assumes that the scene state remains static until the activation is completed. This assumption of fixing the vehicle position before activating the manipulator follows the standard practice for operating work-class ROVs. Figure 6 shows a diagram of our system architecture. We use the MoveIt! Motion Planning Framework (Coleman et al., 2014) to integrate the outputs of the perception system into the planning scene and to generate collision-free motion plans. MoveIt! directly supports a diverse set of state-of-the-art motion planners and inverse kinematic (IK) solvers. For this work, we used the RRT * planner (Karaman and Frazzoli, 2011) with the KDL IK solver. We visualize the planning scene using RVIZ, with out-of-the-box integration with MoveIt!. We generated a kinematic description of the manipulator and vehicle from CAD models, and configured a motion planning environment with MoveIt!. A low-level driver for the manipulator exposes a position trajectory control interface to MoveIt! and interprets motion plans as command packets that it sends Figure 6: A diagram of the overall system, where rectangular blocks represent processes and diamond-shaped blocks represent hardware. Blocks in blue relate to perception. Blocks in red relate to (left) high-and (right) low-level control. Blocks in green are part of the MoveIt! framework around which our system is built. Our system uses the stereo camera to estimate the vehicle configuration (e.g., the pose of the doors on the NUI HROV), generate point clouds of the scene that can be fused to produce a 3D reconstruction of the scene, and assist with tool localization. The fisheye camera is used to localize tools and obtain dynamic viewpoints of the workspace. For low-level control, a driver implements a position-based trajectory controller, which integrates between MoveIt! and the manipulator valve controller. For high-level control, we implemented an automation interface to MoveIt! that supports high-level commands. In this work, we implement this interface using a graphical front-end as well as a preliminary demonstration using natural language.
to the manipulator. Most work class hydraulic manipulators support only position setpoint commands. For this work, the system encodes the target joint positions and sends them directly to the manipulator valve controller. Figure 7 illustrates the coordinate frame transforms that must be calibrated for motion planning and kinematic-based control of the manipulator. The end-effector pose follows from the kinematic chain of transforms from the manipulator base frame through each consecutive link, where each transform is parameterized by the joint angle. Hydraulic manipulators generally provide limited joint feedback from position sensors like potentiometers or resolvers, which must be calibrated to the kinematic model. For this work, we assume a linear interpolation between the feedback values at the joint limits. We calibrate the hand-eye transform between the fisheye camera and the wrist link by detecting the fisheye-relative pose of an AprilTag positioned at a fixed location relative to the manipulator base. We perform these detections for a set of different kinematic configurations and then optimize the hand-eye transform using the ROS easy_handeye package (Tsai et al., 1989). When the manipulator base is rigidly fixed relative to the stereo pair, as would be the case for most ROV configurations, the transformation between the stereo pair and the manipulator base frame is calibrated by detecting the pose of a vehicle-affixed AprilTag in the scene in both the left stereo and the wrist-mounted fisheye cameras. We transform the pose of the fiducial from the fisheye camera frame to the manipulator base frame through the kinematic chain, giving the stereo-to-base frame transform as the difference in the tag pose between the two frames. For the NUI vehicle, where the stereo is not fixed relative to the manipulator, we used a different approach to estimate the stereo-to-base transform in real-time (see Section 4.3).

Calibration Procedure
Figure 7: (right) An image of our testbed consisting of a Kraft TeleRobotics manipulator, a fisheye camera mounted to the end-effector, and a overhead stereo camera. Together with the manipulator base frame, there are four references frames (left) which must be calibrated in order to fuse sensor data into a common reference frame and to plan the motion of the arm. Calibration is performed in the order shown on the left, where each transform enables calibrating the next in a bootstrapping manner. The fiducial in the image is included to indicate that AprilTags were placed statically in the workspace to obtain the Gripper-to-Fisheye and Base-to-Stereo calibrations.

Pick-and-Place Interface
Our autonomy framework targets manipulation operations that involve pick-and-place tasks, such as taking a push-core sample. We implement a simple front-end interface (Fig. 8) that allows a user to step through a pick-and-place state machine that automates each step of the process, while maintaining a high level of safety through human oversight. The interface visualizes the manipulator motion plan at each step and only proceeds to execute the plan after the operator provides confirmation. The interface allows the user to select a target among a set of tools detected in the scene and then activate a sequence of automated steps to grasp and manipulate the target using pre-defined grasp points. An interactive marker enables the user to indicate the desired sample location in the 3D planning scene. Besides the pick-and-place state machine controller, the interface enables one-click planning of the manipulator to a set of pre-defined poses, immediate stopping of any manipulator motion, and opening and closing of the gripper. The MoveIt! planning environment also allows the operator to command the manipulator to an arbitrary configuration within the workspace through an interactive 3D visualization.

System Precision
The maximum precision of our system is limited by both kinematic and visual factors. The KRAFT manipulator uses 11 bit encoders, for an approximate per-joint angular resolution of 0.176 • . When the arm is fully extended to 1.3 m, the angular resolution for the shoulder joints equates to a metric arc length resolution of 4 mm at the end-effector. However, this estimate does not account for non-linear effects in the hydraulic actuators, bias in the joint actuation, inaccurate feedback from the joint sensors, or flexing of the arm's mounting base/vehicle door, any of which may significantly reduce the kinematic accuracy of the system. The visual factors that limit precision include the accuracy of localizing the AprilTags from the fisheye camera and the resolution of the stereo reconstruction. Visual precision is dependent on the metric Figure 8: A simple interface to the automated system allows the user to configure and step through the automated pick-and-place pipeline. The motion plan for each step is visualized in the planning scene and is only executed upon confirmation by the user, which provides a high-level of safety for the system to be deployed on ocean-going systems.
resolution of a pixel projected into the world. For a tag that is 1 m from the fisheye camera, the pixel metric resolution is 1.3 mm, which is the expected best precision for localizing the tags. High distortion of tags near the edges of the fisheye image is expected and has been observed to reduce the localization accuracy. When processing the stereo images to produce depth maps, the maximum working distance can be tuned based on the maximum disparity over which a feature match is searched across a rectified image pair. In our system, the maximum practical distance we target for stereo reconstruction is 3 m, which is well beyond the manipulator reach and beyond which lighting and haze effects severely degraded the image quality. For a viewing range of 3 m, the metric pixel resolution in the stereo view is 1.7 mm. Due to feature smoothing by the SGM correlation window, the actual reconstructed spatial resolution is coarser. In practice, we have observed that the kinematic accuracy is the limiting factor on the precision of our system, due to the many sources of kinematic error in hydraulic manipulator systems.

Automated Pick-and-Place Demonstration on Testbed
To prove the viability of our system before deploying it in the field, we demonstrated the full pick-and-place pipeline on a hardware testbed (Fig. 9) that mimics the configuration of the vision system and manipulator as they would be mounted on an ROV. The testbed includes a Kraft TeleRobotics manipulator identical to the one that we use for the field deployments with the NUI vehicle. The planning environment simulates the manipulator being mounted on the NUI HROV. The stereo point cloud is projected into the planning scene : We demonstrated fully autonomous pick-and-place with a t-handle on a testbed with the same camera and manipulator hardware used on the NUI HROV. First, (a) the t-handle was detected from the fisheye camera using the AprilTags, and the handle pose was projected into the planning scene. Next, (b) the manipulator was commanded to grasp the t-handle via the autonomy interface. Subsequently, (c) a sample location was set in the planning scene with an interactive marker based on the projected stereo point cloud, the manipulator planned a motion to reach the sample location, and executed the plan after the user verified it. The manipulator was then (d) commanded to return the t-handle to the location where it was first grasped. The rock in the environment was placed in a delicate balance on its end, yet the manipulator was controlled with enough precision to bring the tool into direct contact without knocking it over.
to inform placement of the sample marker. As described previously, we estimate the t-handle pose from the wrist-mounted fisheye camera by detecting the AprilTags mounted below the handle. We executed each step of the automated pick-and-place interface successfully, with no manual control input. The system grasped the t-handle based on the detected pose, and the planner found and executed a manipulation path to the sample location marker, which was placed at a non-trivial angle, touching a rock in the scene. The rock was placed on its end in a delicately balanced position, and the manipulator was able to bring the tool into contact with the rock with enough precision that the rock remained standing. The tool was then returned to the position from where it was grasped. This full experiment was repeated multiple times, though not without some grasp failures, due to noise in the visually estimated pose of the t-handle. However, the interface made it easy to recover from any failed step of the state-machine without ever requiring the operator take manual control of the manipulator.

Real-Time Scene Reconstruction and Data Collection at the Costa Rican Pacific Shelf Margin
Demonstrations at the Pacific continental margin were conducted during a two-week research cruise aboard the R/V Falkor using the SuBastian ROV. The automated manipulation component of this expedition focused on a demonstration of the vision system and data collection to aid the development of visual methods. The integration time of our system took approximately two days during cruise mobilization, highlighting the relative ease and flexibility with which the system can be implemented on a variety vehicles and manipulators. Camera image data was streamed over a GigE interface at 3 Hz to a topside workstation, which handled all processing and visualization. Joint encoder feedback from the manipulator was obtained by passively monitoring the serial communication between the manipulator and the ship's control computer. We visualized the real-time configuration of the manipulator in the 3D planning environment with the stereo point clouds projected into the scene. The point clouds were generated from the stereo imagery using the standard semi-global matching (SGM) method built into the ROS image processing pipeline, and the parameters were hand-tuned to achieve the best results. Figure 10 shows a frame from the real-time visualization captured on the seafloor during one of the dives. A good camera calibration combined with high water clarity, rich seafloor texture, and evenly distributed scene lighting resulted in high quality point clouds. These early results demonstrated the effectiveness of the vision system to capture the 3D structure of the workspace and the ability to fuse the information into a real-time scene representation that is useful for both manipulation planning and 3D visualization of the ROV configuration and planning environment.
During this expedition we collected an extensive dataset (Billings and Johnson-Roberson, 2020) of synchronized stereo and wrist mounted fisheye images along with the manipulator joint feedback from a diverse set of seafloor environments (Fig. 11). AprilTags mounted on plates were dispersed into the scenes to provide ground truth for the camera poses, and three different types of graspable handle objects were also randomly placed into the scenes. This dataset supported the development of our visual methods and is also intended to serve the underwater research community for the development of scene reconstruction, object detection, and pose estimation methods that work robustly in real seafloor environments.
The fisheye images were processed into a standalone dataset with annotated 2D bounding boxes and 6D poses for the handle objects visible in each frame. This dataset was released with the SilhoNet-Fisheye publication (Billings and Johnson-Roberson, 2020). Figure 12 shows sample images from this dataset. The combined dataset of stereo and fisheye imagery with synchronized manipulator joint feedback supported our development of visual methods for scene reconstruction.

Automated Sample Collection and Return within Active Submarine Volcanoes
For exploration of the Kolumbo and Santorini calderas, NUI 's manipulator was mounted to the starboard door and the stereo cameras were mounted to the port door (Fig. 14). Having the manipulator and stereo camera on opposite articulating doors allowed for flexibility in configuring the position of the arm according to the specific manipulation task and enabled on-the-fly adjustment of the manipulator and stereo positions separately. Unfortunately, the doors are actuated using hydraulic rams that lack position feedback. For safe motion planning, it was necessary to estimate the door positions in real-time. The estimated door positions were used to update the kinematic configuration of the vehicle in the planning scene. However, we observed that the doors could flex, introducing some error in the kinematic estimates that negatively impacted the accuracy of the stereo point cloud projection into the planning scene. To minimize accumulated error in the transform between the stereo camera frame and the manipulator base frame, the stereo frame was referenced directly to the base frame in the ROS transform tree. The base frame was accurately localised directly from the stereo camera through detection of tags fixed to the manipulator base.
To estimate the door positions in real-time, we affixed AprilTags to the starboard door and to the bow of the vehicle's payload bay (Fig. 15 (left)). The tags on the vehicle bow were mounted at a measured location relative to the vehicle reference frame, with the reference tag's Z-axis aligned with the Z-axis of the vehicle reference frame. The door joint axes of rotation were also aligned to the Z-axis of the vehicle frame, enabling a simple trigonometric calculation of the door angles based on the relative tag locations in the X-Y plane. We used the left stereo camera to track the relative pose of the AprilTags and used these estimates as observations in AprilTag-based visual SLAM (Pfrommer and Daniilidis, 2019) (Fig. 15). Figure 16 shows a schematic of the vehicle and visual SLAM system with the relevant transforms in the X-Y plane used to calculate the door angles. The visual SLAM provided the relative translations between the vehicle tag frame and the starboard tag frame, T vs , and between the vehicle tag frame and the stereo camera frame, T vp . Given that the translations between the vehicle tag and the door joint frames, T os and T op , were measured and known, the angle of the starboard and port doors, θ s and θ p respectively, were recovered as where T s = T vs − T os (2a) where the x and y subscripts indicate the corresponding component of the translation vector, and θ s0 and θ p0 are the measured angle offsets. Figure 13: Map of automated sample collection locations, with regional bathymetry adapted from Nomikou et al. (2012Nomikou et al. ( , 2013. The sea level contour is indicated in black. The dashed line indicates the Christiana-Santorini-Kolumbo tectonic line (Nomikou et al., 2012). Locations marked A, B, and D indicate automated sample collection and return sites, and location C indicates the site where a natural language proof-of-concept demonstration was conducted. Sampling depths ranged from 240 m to 501 m

Planner Controlled Biological Sample Collection
The Kolumbo-Santorini expedition resulted in several scientific achievements, including verification of the persistence of Kalliste Limnes (Camilli et al., 2015), 3D reconstruction of extremophile habitats within the calderas' craters, and sampling of benthic fluids, seafloor sediments, and biological materials. One of the most useful subsea tools for sample collection and return is a "slurp gun" vacuum sampler. For these operations, the slurp nozzle is in close proximity to the sample of interest and a vacuum pump sucks the sample through the hose into a collection chamber. To demonstrate automated slurp collection, we attached the slurp hose to the side of the manipulator, so that the end-effector could be commanded to the desired location to collect the slurp sample. We completed multiple successful sample collections, including that shown in Figure 17, where a slurp sample of a sediment microbial mat was collected using the planner interface to command the manipulator to the desired sample location, after which the manipulator was returned to the home position. Figure 14: The NUI vehicle is outfitted with clam shell doors that can be closed to reduce drag when cruising and opened to perform manipulation tasks. The manipulator is mounted to the starboard door and the stereo cameras are mounted to the port door. Figure 15: Fiducial-based visual SLAM from the left stereo camera was used to estimate the door angles in real-time using (left) tags mounted to the front of the vehicle frame and at the base of the manipulator on the starboard door. SLAM provided estimates of (middle) the relative transformations between the camera and the tag frames that were used (right) to estimate the door angles and update the vehicle model in the planning scene. The left stereo camera was also used in conjunction with the wrist mounted fisheye for (left) fiducial-based localization of tools.

Natural Language Control
Subsea ROV missions require close collaboration between the ROV pilots and scientists. The primary means by which pilots and scientists communicate is through spoken language-scientists use natural language to convey specific mission objectives to ROV pilots (e.g., requesting that a sample be taken from a particular location), while the pilots engage in dialogue to coordinate their efforts. Natural language provides a flexible, efficient, and intuitive means for people to interact with our automated manipulation framework. The inclusion of a natural language interface would support our goal to realize a framework that can be integrated seamlessly with standard ROV operating practices and may eventually mitigate the need for a second pilot.
Using the NUI HROV, we performed a proof-of-concept demonstration of an architecture that allows user control of an ROV manipulator using natural language provided as text or speech using a cloud-based speech recognizer. We frame natural language understanding as a symbol grounding problem (Harnad, 1990), whereby the objective is to map words in the utterance to their corresponding referents in a symbolic representation of the robot's state and action spaces. Consistent with contemporary approaches to language understanding, we formulate grounding as probabilistic inference over a learned distribution that models this mapping. In particular, given the syntactic parse of a natural language command Λ, we employ maximum a posteriori inference over the power set of referent symbols P(Γ) where S is a variable that denotes the robot's model of the environment (e.g., the type and location of different tools). We model this distribution using the Distributed Correspondence Graph (DCG) Howard et al. (2014), a factor graph (Fig. 18) that approximates the conditional probabilities of a Boolean correspondence variable φ ij that indicates the association between a specific symbol γ ij ∈ Γ, which may correspond to an object, action, or location, and each word λ i ∈ Λ. Critically, the composition of the DCG factor graph follows the hierarchical structure of language. The model is trained on corpora of annotated examples (i.e., words from natural language utterances paired with their corresponding groundings), whereby we independently learn the conditional probabilities for the different language elements, such as nouns (e.g., "the pushcore", "tool", and "tool tray"), verbs (e.g., "retrieve", "release", and "stow"), and prepositions (e.g., "inside" and "towards"). Together with the fact that the factor graph exploits the compositional nature of language, the DCG model is able to generalize beyond the specific utterances present in the training data.
For our initial implementation, the space of symbols Γ included the tools that the arm was able to grasp and the different steps that comprised the state machine underlying the pick-and-place pipeline. Figure 19 presents an example from a deployment at the Kolumbo caldera in which natural language was used to initiate path planning to the sample location and then to command the manipulator to execute the planned path. Several tests were conducted in which the manipulator was commanded through natural language input to move to a location specified by the sample marker in the planning interface and then return to the home position. These tests demonstrated the flexibility of our system to incorporate different operational Figure 18: A visualization of (top) the DCG factor graph for the expression "get the pushcore from the tooltray" aligned with (bottom) the associated parse tree. Shaded nodes denote observed random variables, while those rendered in white are latent. modalities through high level abstraction.

Performance Analysis
We evaluated the overall accuracy of the calibrated kinematic and visual system on the testbed. For this evaluation, we placed an AprilTag grid in the scene and activated every joint of the manipulator while keeping the tag grid in view of the fisheye camera. We used TagSLAM Pfrommer and Daniilidis (2019) to generate a visual SLAM estimated trajectory of the fisheye camera, and we used the manipulator joint feedback to also generate a kinematic based trajectory. These trajectories are plotted against each other in figure 20. The overall mean error between the kinematic and visual based trajectories is 1.16cm, the maximum trajectory error is 3.27cm, and the standard deviation is 0.65cm. These results are a conservative estimate of the system calibration accuracy as there are several sources of error: the visual SLAM accuracy degrades when the tags are near the edge of the fisheye image; the agreement between the kinematic and visual based pose of the fisheye camera depends on the accuracy of the hand-eye calibration; the joint feedback and fisheye images are not synchronized; and the SLAM and kinematic reference frames were mapped to each other through a single fisheye frame estimate of the tag grid pose, projected from the fisheye frame through the kinematic chain to the manipulator base frame. However, we have demonstrated in our experimental trials that the system accuracy is good enough to perform high level automation tasks.
The first five joints of the KRAFT manipulator are controlled through joint position set-point commands. We analyzed the manipulator control response for bias or hysteresis, as these are known issues with hydraulic actuation. Figure 21 shows plots of the commanded versus feedback positions during actuation of each joint of the testbed manipulator. The figure also shows a histogram of errors, binned at 0.5 • , between the commanded and followed joint trajectories. All of the joints except the wrist pitch exhibit small bias and no major hysteresis is evident. The wrist pitch exhibits a bias of approximately 1.5 • , which is significant, (a) Language commands the system to plan a path to the sample location (b) Language command to execute planned path Figure 19: Demonstration of a proof-of-concept framework that enabled operators to interact with our autonomous manipulation architecture using natural language. Given input in the form of free-form text, either entered by the operator or output by a cloud-based speech recognizer, we (left) infer the meaning of the command using a probabilistic language model. (a) In the case of the command to "go to the sample location", our system (top-right) determines the goal configuration and solves for a collision-free path in configuration space. (b) Given the command to "execute now", the manipulator then (bottom-right) executes the planned path to the goal. but did not prevent completion of high level automation tasks. Figure 22 shows the same plots for the NUI HROV manipulator made from data recorded during the field trials in Greece. The elbow and wrist joints exhibit little bias or hysteresis. However, both of the shoulder joints exhibited high bias, particularly the shoulder yaw joint, which had a bias of approximately 8 • . This high bias prevented the completion of a pick-and-place manipulation task during the field trial. Our current control system relies on the manipulator valve controller to move to the desired set-point and does not account for bias in the control response. It will be critical in the future to incorporate an adaptive controller into the system that can account for bias and hysteresis in the hydraulic actuators. Figure 20: Plot of the TagSLAM estimated trajectory (visual) of the fisheye camera versus the trajectory estimated from the manipulator joint feedback (kinematic). The trajectory is plotted separately for each coordinate axis with respect to the manipulator base frame.

Discussion and Future Work
During the course of our field trials, we identified operational challenges and failure modes for both the manipulator and vision systems. Addressing these issues is necessary to improve the robustness of the system and is the objective of ongoing research.
Underwater hydraulic manipulators have inherent characteristics which make them especially challenging to automate and can lead to mission failure if they are not accommodated by the planning and control systems. We identify three particular challenges. The first challenge is the senors that provide joint position feedback (e.g. potentiometers or encoders) can be noisy and prone to drift, resulting in an inaccurate estimate of the manipulator configuration, which can lead to self-collisions or collision with the vehicle or obstacles in the environment. This issue could be mitigated by continuously calibrating the arm using the vision system to detect and compensate for proprioceptive sensor drift. Such a fully automated kinematic calibration procedure is also a practical necessity for a system to be deployed on a space flight mission and would improve calibration accuracy over the manual procedure used in this report. Our ongoing work seeks to apply a feature-based mapping/structure-from-motion framework that jointly performs scene reconstruction and kinematic calibration of the manipulator using features from the fisheye camera. The second challenge is that hydraulic actuators can be imprecise. Typical hydraulic actuator characteristics include a bias between the commanded and reached joint positions, which we observed in the KRAFT manipulator, and hysteresis, where the offset between commanded and reached positions is variable with the direction of joint actuation and the position of the joint. These actuator effects could be mitigated through an adaptive control strategy that adjusts the joint commands to account for detected anomalies or offsets between the commanded and reached configurations. Sivčev et al. (2018b) reported hysteresis as high as 1.5 • in a Schilling Titan 2 manipulator and subsequently learned joint command offsets in a calibration procedure to compensate for it. The third challenge is that complete joint failure is common for underwater manipulators, reducing the degrees-of-freedom by at least one. Mitigation of this failure would require planning level adaptation to determine what manipulation tasks are still feasible. In this under-actuated operational state, the vehicle mobility might be considered within the kinematic planning to compensate for the loss of manipulator dexterity, drawing from the prior work on free-floating intervention.
Existing visual reconstruction methods are typically sensitive to lighting, image contrast, and the presence of texture, all of which are highly variable in underwater environments. Figure 23 compares point clouds generated using a standard SGM method from the same stereo camera under two different visual conditions. Under near-ideal conditions that include clear water, uniform illumination, and a richly textured seafloor as was the case during our Costa Rica expedition, the point cloud is highly detailed and exhibits a low amount of noise, resulting in a reconstruction that captures fine details of the scene. During the Kolumbo caldera operations, however, fine-grain unconsolidated sediments and amorphous microbial mats blanketed the seafloor, providing little texture for stereo matching. The illumination was uneven and particulates suspended in the water column caused turbidity and light scattering effects that degraded the quality of the images. Under these conditions, stereo matching is only able to recover the well defined edges of the vehicle, while very little of the seafloor is reconstructed. While these examples represent different extremes in underwater visual conditions, it is critical to develop scene reconstruction algorithms that can operate reliably across this range of conditions to achieve robust autonomy. We are currently investigating information-theoretic ways to exploit our ability to control the pose of the wrist-mounted fisheye camera and an adjacent light source to acquire targeted views and actively illuminate the scene in order to improve and extend reconstruction under both good and degraded visual conditions. Our system implementation currently assumes that the scene is semi-static, i.e., that the ROV position and scene state remain constant during the execution of a manipulator command. For example, if a command is given to grasp a detected tool, the pose of the tool in the scene is assumed to remain fixed during the execution of the grasp. If the tool were to move due to some disturbance before the grasp was completed, the grasp action would likely fail. Future work may integrate an obstacle-aware visual servoing controller to complete grasps or perform precise tool placement, which would reject disturbances to either the scene or the manipulator during task execution. Because our system relies on visual sensing, any disturbance to the scene that results in degraded water clarity, such as stirring up bottom sediment, can necessitate waiting for the water column to clear before the manipulation task can continue. While the KRAFT manipulator used in our field trials is particularly low power when idle, minimizing the energy cost of waiting for visual conditions to improve, future research may improve robustness of the system to degraded visual conditions by fusing acoustic imaging sonar data into the scene mapping framework. Compared to visual sensors, acoustic signals are not dependent on lighting conditions and are not degraded by haze in the water column or a sparsely textured seafloor.
The technology presented in this report can be directly integrated onto terrestrial-based underwater manipulation platforms in order to decrease operational risk, reduce system complexity, and increase overall (e) Good image quality and dense point cloud (f) Poor image quality and sparse point cloud Figure 23: The quality of stereo reconstruction is highly dependent on underwater conditions. Here, we compare stereo point clouds generated using the same camera system and stereo matching method, but with images captured within very different seafloor environments. The left images show the view from the left stereo camera, and the right images show the generated point clouds using a SGM-based stereo method. (e) The top row was captured in the clear waters off Costa Rica, with even scene lighting and highly textured seafloor. (f) The bottom row was captured in the Kolumbo caldera, with high backscatter and low texture microbial mats on the seafloor. efficiency. The current standard for ROV manipulation requires one or more pilots to operate the UVMS based on image feeds from an array of cameras on the vehicle that are displayed on a set of monitors in a ship-side control van. Existing systems do not provide pilots with an estimate of the 3D scene structure, putting the system at risk of collision between the arm and the vehicle or workspace objects. This, together with the cognitive load imposed by having to interpret multiple sensor streams makes it extremely challenging for pilots to establish and maintain situational awareness. The technology presented in this report can be integrated at three different levels with existing ROV systems. At the first and most basic level, the system can act as a decision support tool that provides a detailed real-time 3D visualization of the scene, including the vehicle and manipulator configuration and a reconstruction of the workspace, enabling a pilot to position the manipulator with greater accuracy, speed, and safety. At the second level, the system can be integrated into the manipulator control system for execution monitoring to limit the motion of the manipulator based on scene structure, preventing the pilot from moving the manipulator into collision or a risky configuration (Sivčev et al., 2018c). At the third and highest level, manipulation tasks may be fully automated so that a pilot simply selects a desired function or indicates an intent through some mode of communication such as natural language, whereupon the system plans and executes the task while providing visual feedback to the pilot. In this case, it is critical that the pilot be able to override the automated process and take over control of the arm at will. Single image frames need only be sent when a scene change is detected or on request. Future work on the vision system will develop methods for semantic-level scene understanding, which will further reduce the need for direct image streams back to the pilot. For a semantic aware system, natural language is well suited for human-machine interaction and can drastically reduce the data communication load between the vehicle platform and a remote operator by on-boarding data heavy computation (e.g., image processing) onto the vehicle's local compute system and interfacing with the remote operator through small bandwidth language packets. For our system to operate with pilot oversight, high level commands and sensory feedback need only be streamed at rates which match the dynamics of the scene. In the scenario where the vehicle is set down on the seafloor to collect samples, the relevant scene dynamics can be on the order of seconds, minutes or longer, enabling significant reduction of the communication bandwidth which is vital for remote operations over bandwidth limited connections, such as satellite links. Table 2 shows estimated bandwidth range requirements for the manipulator coms and image streams necessary to support direct teleoperation of an ROV manipulator system compared to the bandwidth requirements for natural language communication with the vehicle and only the necessary scene state feedback to inform the high level commands. In the case of direct teleoperation, the manipulator coms can range from 15 Hz to 200 Hz two-way communication with a typical packet size of 18 B. We estimate the image bandwidth for a single SD or HD camera with compressed data streamed at 10 Hz-30 Hz, though generally multiple camera views are streamed simultaneously back to the pilot for safe manipulator control. In the case of our high-level automation system, the natural language data rates are based on approximate estimates for the average letter count per word and the speech rate. This data rate represents the expected maximum bandwidth load when transmitted in real-time, as language based communication is intermittent and can be compressed. The scene state feedback includes the vehicle state such as the manipulator joint states and semantic information, such as the type and pose of detected tools. However, the visual scene state feedback takes up the bulk of the bandwidth and is assumed to be encoded as a compressed camera frame or view of the 3D scene reconstruction. As demonstrated in the table, communication requirements to support our high-level system reduce the necessary bandwidth load by at least an order of magnitude compared to the requirements of the most limited direct teleoperation modality.
Despite the technological challenges in reaching extraterrestrial worlds, the NASA Science Mission Directorate (SMD) sets its first priority "to discover the secrets of the universe, to search for life, and to protect and improve life on Earth" (NASA Science Mission Directorate, 2020) and "is undertaking a flagship mission to Jupiter's moon Europa, as its subsurface ocean has great potential to harbor extraterrestrial life." A Europa mission concept for a surface lander has reached relative maturity, having passed its delta Mission Concept Review (Hand et al., 2021). The sampling system is recognized as being critical to the success of the mission and relies on a robotic arm for "excavation, collection, and presentation (or transfer) of samples to scientific instruments for observation and analysis" (Hand et al., 2017). Due to the anticipated communication limitations, it is likely that the lander will be required to self-select sampling sites, in which "the sampling system would be capable of conducting a sampling cycle in a fully autonomous fashion with no input from ground operators, from target selection to sample delivery. This autonomous capability is to guard against a prolonged telecommunications fault during the short mission lifetime, and will be in place to provide added assurance that the mission threshold science would be met" (Hand et al., 2017). Challenges to the sampling system will be exacerbated by "poorly-characterized terrain at small scales", and "the terrain immediately in front of the landing spot must suffice for sampling locations; there is no mobility system that can be used to search for a better site" (Hand et al., 2017). The methods we demonstrated in this report for automated manipulator control and sample collection are directly applicable to operations focused on a Ladder of Life detection mission scenario (Neveu et al., 2018). With the exception of the wrist mounted camera, the manipulator and imaging system used in our demonstrations are very similar to the hardware for the Europa lander concept, consisting of a multi-DoF manipulator and vehicle-mounted stereo pair. A primary limiting factor on the integration of our autonomy methods with the lander would be the available computational power. However, for a stationary lander, the visual processing, which is the primary computational bottleneck, could operate at low-frame rates suitable for extraterrestrial exploration, assuming the environment dynamics are sufficiently slow. The methods we describe are also suitable to run on embedded systems and may be optimized accordingly.

Conclusions
An exobiology search mission to distant ocean worlds will require a highly automated exploratory vehicle, capable of operating in extreme conditions for an extended period of time. Such a platform will likely be outfitted with a manipulator to maximize the types of samples that could be collected. In this report we describe a vision system and control framework for automating an ROV manipulator. This architecture is readily integrated onto a wide array of vehicle platforms, and we have demonstrated the viability of the system in the field on two ROVs with different manipulators, including the NUI HROV which is dynamically reconfigurable. In November of 2019, we demonstrated planner-controlled sample collection and return within active submarine volcanoes that host diverse assemblages of extremophile organisms. These operation locations served as analogs to environments that may exist within other ocean worlds in our solar system and beyond.
A current limitation of our approach is a semi-static vehicle and scene assumption, where the ROV is held stationary and the scene does not change during execution of a manipulator motion, though the vehicle and scene state may change between motions. The vehicle is typically kept stationary by setting it down on the seafloor before manipulation is initiated. This assumption limits the type of sampling tasks that may be performed with the described system. For example collecting samples from a vertical wall, the underside of an ice shelf, or other moving objects would require free-floating control. Free-floating manipulation is an open problem in robotics, and a promising research direction that directly builds on our demonstrated system is obstacle aware disturbance rejection control of the manipulator. This method is similar to obstacle aware visual servoing, using feature based SLAM with the vision system to compensate for vehicle motions and stabilize the end-effector. A disturbance rejection approach would enhance the flexibility of the system to be easily integrated on different vehicles and manipulators without requiring the generation of complex vehicle and manipulator dynamic models. While the demonstrated system represents a significant step towards autonomous sample collection and return from seafloor environments, more advancements are required before the system can be deployed reliably in a fully automated fashion. In particular, visual methods must be developed that are robust to the optical challenges of the underwater environment in order to enable safe and targeted sample collection and precision tool handling. These methods must be robust to dynamic scenes, insensitive to the intensity inconsistency of underwater lighting and perform well in sparsely featured and low-textured environments. Fusion of sparse feature based methods for SLAM with learning-based methods for dense scene reconstruction and high-level semantic scene understanding, such as segmentation, object detection and tool pose estimation may provide an appropriate path forward to overcome this challenge.
In summary, automated exploration of unstructured seafloor environments is within reach of current underwater robotic technology. More development is needed, particularly in methods for scene reconstruction and understanding, to make this technology sufficiently reliable for fully automated deployment, but results from our oceanographic expeditions described in this report demonstrate that a wide range of existing ROVs and manipulator systems can be adapted, with moderate effort, for high level automation capabilities.