Ten years of image analysis and machine learning competitions in
  dementia
  Machine learning methods exploiting multi-parametric biomarkers, especially
based on neuroimaging, have huge potential to improve early diagnosis of
dementia and to predict which individuals are at-risk of developing dementia.
To benchmark algorithms in the field of machine learning and neuroimaging in
dementia and assess their potential for use in clinical practice and clinical
trials, seven grand challenges have been organized in the last decade.
  The seven grand challenges addressed questions related to screening, clinical
status estimation, prediction and monitoring in (pre-clinical) dementia. There
was little overlap in clinical questions, tasks and performance metrics.
Whereas this aids providing insight on a broad range of questions, it also
limits the validation of results across challenges. The validation process
itself was mostly comparable between challenges, using similar methods for
ensuring objective comparison, uncertainty estimation and statistical testing.
In general, winning algorithms performed rigorous data preprocessing and
combined a wide range of input features.
  Despite high state-of-the-art performances, most of the methods evaluated by
the challenges are not clinically used. To increase impact, future challenges
could pay more attention to statistical analysis of which factors relate to
higher performance, to clinical questions beyond Alzheimer's disease, and to
using testing data beyond the Alzheimer's Disease Neuroimaging Initiative.
Grand challenges would be an ideal venue for assessing the generalizability of
algorithm performance to unseen data of other cohorts. Key for increasing
impact in this way are larger testing data sizes, which could be reached by
sharing algorithms rather than data to exploit data that cannot be shared.
