Adaptive Bayesian Reticulum
  Neural Networks and Decision Trees: two popular techniques for supervised
learning that are seemingly disconnected in their formulation and optimization
method, have recently been combined in a single construct. The connection
pivots on assembling an artificial Neural Network with nodes that allow for a
gate-like function to mimic a tree split, optimized using the standard approach
of recursively applying the chain rule to update its parameters. Yet two main
challenges have impeded wide use of this hybrid approach: (a) the inability of
global gradient ascent techniques to optimize hierarchical parameters (as
introduced by the gate function); and (b) the construction of the tree
structure, which has relied on standard decision tree algorithms to learn the
network topology or incrementally (and heuristically) searching the space at
random. Here we propose a probabilistic construct that exploits the idea of a
node's unexplained potential (the total error channeled through the node) in
order to decide where to expand further, mimicking the standard tree
construction in a Neural Network setting, alongside a modified gradient ascent
that first locally optimizes an expanded node before a global optimization. The
probabilistic approach allows us to evaluate each new split as a ratio of
likelihoods that balances the statistical improvement in explaining the
evidence against the additional model complexity --- thus providing a natural
stopping condition. The result is a novel classification and regression
technique that leverages the strength of both: a tree-structure that grows
naturally and is simple to interpret with the plasticity of Neural Networks
that allow for soft margins and slanted boundaries.
