A New Smooth Approximation to the Zero One Loss with a Probabilistic
  Interpretation
  We examine a new form of smooth approximation to the zero one loss in which
learning is performed using a reformulation of the widely used logistic
function. Our approach is based on using the posterior mean of a novel
generalized Beta-Bernoulli formulation. This leads to a generalized logistic
function that approximates the zero one loss, but retains a probabilistic
formulation conferring a number of useful properties. The approach is easily
generalized to kernel logistic regression and easily integrated into methods
for structured prediction. We present experiments in which we learn such models
using an optimization method consisting of a combination of gradient descent
and coordinate descent using localized grid search so as to escape from local
minima. Our experiments indicate that optimization quality is improved when
learning meta-parameters are themselves optimized using a validation set. Our
experiments show improved performance relative to widely used logistic and
hinge loss methods on a wide variety of problems ranging from standard UC
Irvine and libSVM evaluation datasets to product review predictions and a
visual information extraction task. We observe that the approach: 1) is more
robust to outliers compared to the logistic and hinge losses; 2) outperforms
comparable logistic and max margin models on larger scale benchmark problems;
3) when combined with Gaussian- Laplacian mixture prior on parameters the
kernelized version of our formulation yields sparser solutions than Support
Vector Machine classifiers; and 4) when integrated into a probabilistic
structured prediction technique our approach provides more accurate
probabilities yielding improved inference and increasing information extraction
performance.
