PEg TRAnsfer Workflow recognition challenge report: Does multi-modal
  data improve recognition?
  This paper presents the design and results of the "PEg TRAnsfert Workflow
recognition" (PETRAW) challenge whose objective was to develop surgical
workflow recognition methods based on one or several modalities, among video,
kinematic, and segmentation data, in order to study their added value. The
PETRAW challenge provided a data set of 150 peg transfer sequences performed on
a virtual simulator. This data set was composed of videos, kinematics, semantic
segmentation, and workflow annotations which described the sequences at three
different granularity levels: phase, step, and activity. Five tasks were
proposed to the participants: three of them were related to the recognition of
all granularities with one of the available modalities, while the others
addressed the recognition with a combination of modalities. Average
application-dependent balanced accuracy (AD-Accuracy) was used as evaluation
metric to take unbalanced classes into account and because it is more
clinically relevant than a frame-by-frame score. Seven teams participated in at
least one task and four of them in all tasks. Best results are obtained with
the use of the video and the kinematics data with an AD-Accuracy between 93%
and 90% for the four teams who participated in all tasks. The improvement
between video/kinematic-based methods and the uni-modality ones was significant
for all of the teams. However, the difference in testing execution time between
the video/kinematic-based and the kinematic-based methods has to be taken into
consideration. Is it relevant to spend 20 to 200 times more computing time for
less than 3% of improvement? The PETRAW data set is publicly available at
www.synapse.org/PETRAW to encourage further research in surgical workflow
recognition.
