A Multisite, Report-Based, Centralized Infrastructure for Feedback and
  Monitoring of Radiology AI/ML Development and Clinical Deployment
  An infrastructure for multisite, geographically-distributed creation and
collection of diverse, high-quality, curated and labeled radiology image data
is crucial for the successful automated development, deployment, monitoring and
continuous improvement of Artificial Intelligence (AI)/Machine Learning (ML)
solutions in the real world. An interactive radiology reporting approach that
integrates image viewing, dictation, natural language processing (NLP) and
creation of hyperlinks between image findings and the report, provides
localized labels during routine interpretation. These images and labels can be
captured and centralized in a cloud-based system. This method provides a
practical and efficient mechanism with which to monitor algorithm performance.
It also supplies feedback for iterative development and quality improvement of
new and existing algorithmic models. Both feedback and monitoring are achieved
without burdening the radiologist. The method addresses proposed regulatory
requirements for post-marketing surveillance and external data. Comprehensive
multi-site data collection assists in reducing bias. Resource requirements are
greatly reduced compared to dedicated retrospective expert labeling.
