Toward Force Estimation in Robot-Assisted Surgery using Deep Learning
  with Vision and Robot State
  Knowledge of interaction forces during teleoperated robot-assisted surgery
could be used to enable force feedback to human operators and evaluate tissue
handling skill. However, direct force sensing at the end-effector is
challenging because it requires biocompatible, sterilizable, and cost-effective
sensors. Vision-based deep learning using convolutional neural networks is a
promising approach for providing useful force estimates, though questions
remain about generalization to new scenarios and real-time inference. We
present a force estimation neural network that uses RGB images and robot state
as inputs. Using a self-collected dataset, we compared the network to variants
that included only a single input type, and evaluated how they generalized to
new viewpoints, workspace positions, materials, and tools. We found that
vision-based networks were sensitive to shifts in viewpoints, while state-only
networks were robust to changes in workspace. The network with both state and
vision inputs had the highest accuracy for an unseen tool, and was moderately
robust to changes in viewpoints. Through feature removal studies, we found that
using only position features produced better accuracy than using only force
features as input. The network with both state and vision inputs outperformed a
physics-based baseline model in accuracy. It showed comparable accuracy but
faster computation times than a baseline recurrent neural network, making it
better suited for real-time applications.
