Comparative Validation of Machine Learning Algorithms for Surgical
  Workflow and Skill Analysis with the HeiChole Benchmark
  PURPOSE: Surgical workflow and skill analysis are key technologies for the
next generation of cognitive surgical assistance systems. These systems could
increase the safety of the operation through context-sensitive warnings and
semi-autonomous robotic assistance or improve training of surgeons via
data-driven feedback. In surgical workflow analysis up to 91% average precision
has been reported for phase recognition on an open data single-center dataset.
In this work we investigated the generalizability of phase recognition
algorithms in a multi-center setting including more difficult recognition tasks
such as surgical action and surgical skill. METHODS: To achieve this goal, a
dataset with 33 laparoscopic cholecystectomy videos from three surgical centers
with a total operation time of 22 hours was created. Labels included annotation
of seven surgical phases with 250 phase transitions, 5514 occurences of four
surgical actions, 6980 occurences of 21 surgical instruments from seven
instrument categories and 495 skill classifications in five skill dimensions.
The dataset was used in the 2019 Endoscopic Vision challenge, sub-challenge for
surgical workflow and skill analysis. Here, 12 teams submitted their machine
learning algorithms for recognition of phase, action, instrument and/or skill
assessment. RESULTS: F1-scores were achieved for phase recognition between
23.9% and 67.7% (n=9 teams), for instrument presence detection between 38.5%
and 63.8% (n=8 teams), but for action recognition only between 21.8% and 23.3%
(n=5 teams). The average absolute error for skill assessment was 0.78 (n=1
team). CONCLUSION: Surgical workflow and skill analysis are promising
technologies to support the surgical team, but are not solved yet, as shown by
our comparison of algorithms. This novel benchmark can be used for comparable
evaluation and validation of future work.
