MP-Boost: Minipatch Boosting via Adaptive Feature and Observation
  Sampling
  Boosting methods are among the best general-purpose and off-the-shelf machine
learning approaches, gaining widespread popularity. In this paper, we seek to
develop a boosting method that yields comparable accuracy to popular AdaBoost
and gradient boosting methods, yet is faster computationally and whose solution
is more interpretable. We achieve this by developing MP-Boost, an algorithm
loosely based on AdaBoost that learns by adaptively selecting small subsets of
instances and features, or what we term minipatches (MP), at each iteration. By
sequentially learning on tiny subsets of the data, our approach is
computationally faster than other classic boosting algorithms. Also as it
progresses, MP-Boost adaptively learns a probability distribution on the
features and instances that upweight the most important features and
challenging instances, hence adaptively selecting the most relevant minipatches
for learning. These learned probability distributions also aid in
interpretation of our method. We empirically demonstrate the interpretability,
comparative accuracy, and computational time of our approach on a variety of
binary classification tasks.
