Deep Probability Estimation
  Reliable probability estimation is of crucial importance in many real-world
applications where there is inherent uncertainty, such as weather forecasting,
medical prognosis, or collision avoidance in autonomous vehicles.
Probability-estimation models are trained on observed outcomes (e.g. whether it
has rained or not, or whether a patient has died or not), because the
ground-truth probabilities of the events of interest are typically unknown. The
problem is therefore analogous to binary classification, with the important
difference that the objective is to estimate probabilities rather than
predicting the specific outcome. The goal of this work is to investigate
probability estimation from high-dimensional data using deep neural networks.
There exist several methods to improve the probabilities generated by these
models but they mostly focus on classification problems where the probabilities
are related to model uncertainty. In the case of problems with inherent
uncertainty, it is challenging to evaluate performance without access to
ground-truth probabilities. To address this, we build a synthetic dataset to
study and compare different computable metrics. We evaluate existing methods on
the synthetic data as well as on three real-world probability estimation tasks,
all of which involve inherent uncertainty. We also give a theoretical analysis
of a model for high-dimensional probability estimation which reproduces several
of the phenomena evinced in our experiments. Finally, we propose a new method
for probability estimation using neural networks, which modifies the training
process to promote output probabilities that are consistent with empirical
probabilities computed from the data. The method outperforms existing
approaches on most metrics on the simulated as well as real-world data.
