Robust deep learning-based semantic organ segmentation in hyperspectral
  images
  Semantic image segmentation is an important prerequisite for
context-awareness and autonomous robotics in surgery. The state of the art has
focused on conventional RGB video data acquired during minimally invasive
surgery, but full-scene semantic segmentation based on spectral imaging data
and obtained during open surgery has received almost no attention to date. To
address this gap in the literature, we are investigating the following research
questions based on hyperspectral imaging (HSI) data of pigs acquired in an open
surgery setting: (1) What is an adequate representation of HSI data for neural
network-based fully automated organ segmentation, especially with respect to
the spatial granularity of the data (pixels vs. superpixels vs. patches vs.
full images)? (2) Is there a benefit of using HSI data compared to other
modalities, namely RGB data and processed HSI data (e.g. tissue parameters like
oxygenation), when performing semantic organ segmentation? According to a
comprehensive validation study based on 506 HSI images from 20 pigs, annotated
with a total of 19 classes, deep learning-based segmentation performance
increases - consistently across modalities - with the spatial context of the
input data. Unprocessed HSI data offers an advantage over RGB data or processed
data from the camera provider, with the advantage increasing with decreasing
size of the input to the neural network. Maximum performance (HSI applied to
whole images) yielded a mean dice similarity coefficient (DSC) of 0.89
(standard deviation (SD) 0.04), which is in the range of the inter-rater
variability (DSC of 0.89 (SD 0.07)). We conclude that HSI could become a
powerful image modality for fully-automatic surgical scene understanding with
many advantages over traditional imaging, including the ability to recover
additional functional tissue information.
