Quantifying the Reproducibility of Graph Neural Networks using
  Multigraph Brain Data
  Graph neural networks (GNNs) have witnessed an unprecedented proliferation in
tackling several problems in computer vision, computer-aided diagnosis, and
related fields. While prior studies have focused on boosting the model
accuracy, quantifying the reproducibility of the most discriminative features
identified by GNNs is still an intact problem that yields concerns about their
reliability in clinical applications in particular. Specifically, the
reproducibility of biological markers across clinical datasets and distribution
shifts across classes (e.g., healthy and disordered brains) is of paramount
importance in revealing the underpinning mechanisms of diseases as well as
propelling the development of personalized treatment. Motivated by these
issues, we propose, for the first time, reproducibility-based GNN selection
(RG-Select), a framework for GNN reproducibility assessment via the
quantification of the most discriminative features (i.e., biomarkers) shared
between different models. To ascertain the soundness of our framework, the
reproducibility assessment embraces variations of different factors such as
training strategies and data perturbations. Despite these challenges, our
framework successfully yielded replicable conclusions across different training
strategies and various clinical datasets. Our findings could thus pave the way
for the development of biomarker trustworthiness and reliability assessment
methods for computer-aided diagnosis and prognosis tasks. RG-Select code is
available on GitHub at https://github.com/basiralab/RG-Select.
