Probabilistic Gradient Boosting Machines for Large-Scale Probabilistic
  Regression
  Gradient Boosting Machines (GBM) are hugely popular for solving tabular data
problems. However, practitioners are not only interested in point predictions,
but also in probabilistic predictions in order to quantify the uncertainty of
the predictions. Creating such probabilistic predictions is difficult with
existing GBM-based solutions: they either require training multiple models or
they become too computationally expensive to be useful for large-scale
settings. We propose Probabilistic Gradient Boosting Machines (PGBM), a method
to create probabilistic predictions with a single ensemble of decision trees in
a computationally efficient manner. PGBM approximates the leaf weights in a
decision tree as a random variable, and approximates the mean and variance of
each sample in a dataset via stochastic tree ensemble update equations. These
learned moments allow us to subsequently sample from a specified distribution
after training. We empirically demonstrate the advantages of PGBM compared to
existing state-of-the-art methods: (i) PGBM enables probabilistic estimates
without compromising on point performance in a single model, (ii) PGBM learns
probabilistic estimates via a single model only (and without requiring
multi-parameter boosting), and thereby offers a speedup of up to several orders
of magnitude over existing state-of-the-art methods on large datasets, and
(iii) PGBM achieves accurate probabilistic estimates in tasks with complex
differentiable loss functions, such as hierarchical time series problems, where
we observed up to 10% improvement in point forecasting performance and up to
300% improvement in probabilistic forecasting performance.
