Codabench: Flexible, Easy-to-Use and Reproducible Benchmarking Platform
  Obtaining standardized crowdsourced benchmark of computational methods is a
major issue in data science communities. Dedicated frameworks enabling fair
benchmarking in a unified environment are yet to be developed. Here we
introduce Codabench, an open-source, community-driven platform for benchmarking
algorithms or software agents versus datasets or tasks. A public instance of
Codabench (https://www.codabench.org/) is open to everyone, free of charge, and
allows benchmark organizers to compare fairly submissions, under the same
setting (software, hardware, data, algorithms), with custom protocols and data
formats. Codabench has unique features facilitating the organization of
benchmarks flexibly, easily and reproducibly, such as the possibility of
re-using templates of benchmarks, and supplying compute resources on-demand.
Codabench has been used internally and externally on various applications,
receiving more than 130 users and 2500 submissions. As illustrative use cases,
we introduce 4 diverse benchmarks covering Graph Machine Learning, Cancer
Heterogeneity, Clinical Diagnosis and Reinforcement Learning.
