Critical Evaluation of Deep Neural Networks for Wrist Fracture Detection
  Wrist Fracture is the most common type of fracture with a high incidence
rate. Conventional radiography (i.e. X-ray imaging) is used for wrist fracture
detection routinely, but occasionally fracture delineation poses issues and an
additional confirmation by computed tomography (CT) is needed for diagnosis.
Recent advances in the field of Deep Learning (DL), a subfield of Artificial
Intelligence (AI), have shown that wrist fracture detection can be automated
using Convolutional Neural Networks. However, previous studies did not pay
close attention to the difficult cases which can only be confirmed via CT
imaging. In this study, we have developed and analyzed a state-of-the-art
DL-based pipeline for wrist (distal radius) fracture detection -- DeepWrist,
and evaluated it against one general population test set, and one challenging
test set comprising only cases requiring confirmation by CT. Our results reveal
that a typical state-of-the-art approach, such as DeepWrist, while having a
near-perfect performance on the general independent test set, has a
substantially lower performance on the challenging test set -- average
precision of 0.99 (0.99-0.99) vs 0.64 (0.46-0.83), respectively. Similarly, the
area under the ROC curve was of 0.99 (0.98-0.99) vs 0.84 (0.72-0.93),
respectively. Our findings highlight the importance of a meticulous analysis of
DL-based models before clinical use, and unearth the need for more challenging
settings for testing medical AI systems.
