Evaluating structure learning algorithms with a balanced scoring
  function
  Several structure learning algorithms have been proposed towards discovering
causal or Bayesian Network (BN) graphs. The validity of these algorithms tends
to be evaluated by assessing the relationship between the learnt and the ground
truth graph. However, there is no agreed scoring metric to determine this
relationship. Moreover, this paper shows that some of the commonly used metrics
tend to be biased in favour of graphs that minimise edges. While graphs that
are less complex are desirable, some of the metrics favour underfitted graphs,
thereby encouraging limited propagation of evidence. This paper proposes the
Balanced Scoring Function (BSF) that eliminates this bias by adjusting the
reward function based on the difficulty of discovering an edge, or no edge,
proportional to their occurrence rate in the ground truth graph. The BSF score
can be used in conjunction with other traditional metrics to provide an
alternative and unbiased assessment about the capability of a structure
learning algorithm in discovering causal or BN graphs.
