Benchpress: A Scalable and Versatile Workflow for Benchmarking Structure
  Learning Algorithms
  Describing the relationship between the variables in a study domain and
modelling the data generating mechanism is a fundamental problem in many
empirical sciences. Probabilistic graphical models are one common approach to
tackle the problem. Learning the graphical structure for such models is
computationally challenging and a fervent area of current research with a
plethora of algorithms being developed. To facilitate the benchmarking of
different methods, we present a novel Snakemake workflow, called Benchpress for
producing scalable, reproducible, and platform-independent benchmarks of
structure learning algorithms for probabilistic graphical models. Benchpress is
interfaced via a simple JSON-file, which makes it accessible for all users,
while the code is designed in a fully modular fashion to enable researchers to
contribute additional methodologies. Benchpress currently provides an interface
to a large number of state-of-the-art algorithms from libraries such as
BDgraph, BiDAG, bnlearn, gCastle, GOBNILP, pcalg, r.blip, scikit-learn, TETRAD,
and trilearn as well as a variety of methods for data generating models and
performance evaluation. Alongside user-defined models and randomly generated
datasets, the workflow also includes a number of standard datasets and
graphical models from the literature, which may be included in a benchmarking
study. We demonstrate the applicability of this workflow for learning Bayesian
networks in five typical data scenarios. The source code and documentation is
publicly available from http://github.com/felixleopoldo/benchpress.
