Learning Representations for Axis-Aligned Decision Forests through Input
  Perturbation
  Axis-aligned decision forests have long been the leading class of machine
learning algorithms for modeling tabular data. In many applications of machine
learning such as learning-to-rank, decision forests deliver remarkable
performance. They also possess other coveted characteristics such as
interpretability. Despite their widespread use and rich history, decision
forests to date fail to consume raw structured data such as text, or learn
effective representations for them, a factor behind the success of deep neural
networks in recent years. While there exist methods that construct smoothed
decision forests to achieve representation learning, the resulting models are
decision forests in name only: They are no longer axis-aligned, use stochastic
decisions, or are not interpretable. Furthermore, none of the existing methods
are appropriate for problems that require a Transfer Learning treatment. In
this work, we present a novel but intuitive proposal to achieve representation
learning for decision forests without imposing new restrictions or
necessitating structural changes. Our model is simply a decision forest,
possibly trained using any forest learning algorithm, atop a deep neural
network. By approximating the gradients of the decision forest through input
perturbation, a purely analytical procedure, the decision forest directs the
neural network to learn or fine-tune representations. Our framework has the
advantage that it is applicable to any arbitrary decision forest and that it
allows the use of arbitrary deep neural networks for representation learning.
We demonstrate the feasibility and effectiveness of our proposal through
experiments on synthetic and benchmark classification datasets.
