Deep differentiable forest with sparse attention for the tabular data
  We present a general architecture of deep differentiable forest and its
sparse attention mechanism. The differentiable forest has the advantages of
both trees and neural networks. Its structure is a simple binary tree, easy to
use and understand. It has full differentiability and all variables are
learnable parameters. We would train it by the gradient-based optimization
method, which shows great power in the training of deep CNN. We find and
analyze the attention mechanism in the differentiable forest. That is, each
decision depends on only a few important features, and others are irrelevant.
The attention is always sparse. Based on this observation, we improve its
sparsity by data-aware initialization. We use the attribute importance to
initialize the attention weight. Then the learned weight is much sparse than
that from random initialization. Our experiment on some large tabular dataset
shows differentiable forest has higher accuracy than GBDT, which is the state
of art algorithm for tabular datasets. The source codes are available at
https://github.com/closest-git/QuantumForest
