MedPerf: Open Benchmarking Platform for Medical Artificial Intelligence
  using Federated Evaluation
  Medical AI has tremendous potential to advance healthcare by supporting the
evidence-based practice of medicine, personalizing patient treatment, reducing
costs, and improving provider and patient experience. We argue that unlocking
this potential requires a systematic way to measure the performance of medical
AI models on large-scale heterogeneous data. To meet this need, we are building
MedPerf, an open framework for benchmarking machine learning in the medical
domain. MedPerf will enable federated evaluation in which models are securely
distributed to different facilities for evaluation, thereby empowering
healthcare organizations to assess and verify the performance of AI models in
an efficient and human-supervised process, while prioritizing privacy. We
describe the current challenges healthcare and AI communities face, the need
for an open platform, the design philosophy of MedPerf, its current
implementation status, and our roadmap. We call for researchers and
organizations to join us in creating the MedPerf open benchmarking platform.
