Efficient and Scalable Structure Learning for Bayesian Networks:
  Algorithms and Applications
  Structure Learning for Bayesian network (BN) is an important problem with
extensive research. It plays central roles in a wide variety of applications in
Alibaba Group. However, existing structure learning algorithms suffer from
considerable limitations in real world applications due to their low efficiency
and poor scalability. To resolve this, we propose a new structure learning
algorithm LEAST, which comprehensively fulfills our business requirements as it
attains high accuracy, efficiency and scalability at the same time. The core
idea of LEAST is to formulate the structure learning into a continuous
constrained optimization problem, with a novel differentiable constraint
function measuring the acyclicity of the resulting graph. Unlike with existing
work, our constraint function is built on the spectral radius of the graph and
could be evaluated in near linear time w.r.t. the graph node size. Based on it,
LEAST can be efficiently implemented with low storage overhead. According to
our benchmark evaluation, LEAST runs 1 to 2 orders of magnitude faster than
state of the art method with comparable accuracy, and it is able to scale on
BNs with up to hundreds of thousands of variables. In our production
environment, LEAST is deployed and serves for more than 20 applications with
thousands of executions per day. We describe a concrete scenario in a ticket
booking service in Alibaba, where LEAST is applied to build a near real-time
automatic anomaly detection and root error cause analysis system. We also show
that LEAST unlocks the possibility of applying BN structure learning in new
areas, such as large-scale gene expression data analysis and explainable
recommendation system.
