Utilizing a digital swarm intelligence platform to improve consensus
  among radiologists and exploring its applications
  Radiologists today play a key role in making diagnostic decisions and
labeling images for training A.I. algorithms. Low inter-reader reliability
(IRR) can be seen between experts when interpreting challenging cases. While
teams-based decisions are known to outperform individual decisions,
inter-personal biases often creep up in group interactions which limit
non-dominant participants from expressing true opinions. To overcome the dual
problems of low consensus and inter-personal bias, we explored a solution
modeled on biological swarms of bees. Two separate cohorts; three radiologists
and five radiology residents collaborated on a digital swarm platform in real
time and in a blinded fashion, grading meniscal lesions on knee MR exams. These
consensus votes were benchmarked against clinical (arthroscopy) and
radiological (senior-most radiologist) observations. The IRR of the consensus
votes was compared to the IRR of the majority and most confident votes of the
two cohorts.The radiologist cohort saw an improvement of 23% in IRR of swarm
votes over majority vote. Similar improvement of 23% in IRR in 3-resident swarm
votes over majority vote, was observed. The 5-resident swarm had an even higher
improvement of 32% in IRR over majority vote. Swarm consensus votes also
improved specificity by up to 50%. The swarm consensus votes outperformed
individual and majority vote decisions in both the radiologists and resident
cohorts. The 5-resident swarm had higher IRR than 3-resident swarm indicating
positive effect of increased swarm size. The attending and resident swarms also
outperformed predictions from a state-of-the-art A.I. algorithm. Utilizing a
digital swarm platform improved agreement and allows participants to express
judgement free intent, resulting in superior clinical performance and robust
A.I. training labels.
