Growing Deep Forests Efficiently with Soft Routing and Learned
  Connectivity
  Despite the latest prevailing success of deep neural networks (DNNs), several
concerns have been raised against their usage, including the lack of
intepretability the gap between DNNs and other well-established machine
learning models, and the growingly expensive computational costs. A number of
recent works [1], [2], [3] explored the alternative to sequentially stacking
decision tree/random forest building blocks in a purely feed-forward way, with
no need of back propagation. Since decision trees enjoy inherent reasoning
transparency, such deep forest models can also facilitate the understanding of
the internaldecision making process. This paper further extends the deep forest
idea in several important aspects. Firstly, we employ a probabilistic tree
whose nodes make probabilistic routing decisions, a.k.a., soft routing, rather
than hard binary decisions.Besides enhancing the flexibility, it also enables
non-greedy optimization for each tree. Second, we propose an innovative
topology learning strategy: every node in the ree now maintains a new learnable
hyperparameter indicating the probability that it will be a leaf node. In that
way, the tree will jointly optimize both its parameters and the tree topology
during training. Experiments on the MNIST dataset demonstrate that our
empowered deep forests can achieve better or comparable performance than
[1],[3] , with dramatically reduced model complexity. For example,our model
with only 1 layer of 15 trees can perform comparably with the model in [3] with
2 layers of 2000 trees each.
