Modeling Text with Decision Forests using Categorical-Set Splits
  Decision forest algorithms typically model data by learning a binary tree
structure recursively where every node splits the feature space into two
sub-regions, sending examples into the left or right branch as a result. In
axis-aligned decision forests, the "decision" to route an input example is the
result of the evaluation of a condition on a single dimension in the feature
space. Such conditions are learned using efficient, often greedy algorithms
that optimize a local loss function. For example, a node's condition may be a
threshold function applied to a numerical feature, and its parameter may be
learned by sweeping over the set of values available at that node and choosing
a threshold that maximizes some measure of purity. Crucially, whether an
algorithm exists to learn and evaluate conditions for a feature type determines
whether a decision forest algorithm can model that feature type at all. For
example, decision forests today cannot consume textual features directly --
such features must be transformed to summary statistics instead. In this work,
we set out to bridge that gap. We define a condition that is specific to
categorical-set features -- defined as an unordered set of categorical
variables -- and present an algorithm to learn it, thereby equipping decision
forests with the ability to directly model text, albeit without preserving
sequential order. Our algorithm is efficient during training and the resulting
conditions are fast to evaluate with our extension of the QuickScorer inference
algorithm. Experiments on benchmark text classification datasets demonstrate
the utility and effectiveness of our proposal.
