Beyond L2-Loss Functions for Learning Sparse Models
  Incorporating sparsity priors in learning tasks can give rise to simple, and
interpretable models for complex high dimensional data. Sparse models have
found widespread use in structure discovery, recovering data from corruptions,
and a variety of large scale unsupervised and supervised learning problems.
Assuming the availability of sufficient data, these methods infer dictionaries
for sparse representations by optimizing for high-fidelity reconstruction. In
most scenarios, the reconstruction quality is measured using the squared
Euclidean distance, and efficient algorithms have been developed for both batch
and online learning cases. However, new application domains motivate looking
beyond conventional loss functions. For example, robust loss functions such as
$\ell_1$ and Huber are useful in learning outlier-resilient models, and the
quantile loss is beneficial in discovering structures that are the
representative of a particular quantile. These new applications motivate our
work in generalizing sparse learning to a broad class of convex loss functions.
In particular, we consider the class of piecewise linear quadratic (PLQ) cost
functions that includes Huber, as well as $\ell_1$, quantile, Vapnik, hinge
loss, and smoothed variants of these penalties. We propose an algorithm to
learn dictionaries and obtain sparse codes when the data reconstruction
fidelity is measured using any smooth PLQ cost function. We provide convergence
guarantees for the proposed algorithm, and demonstrate the convergence behavior
using empirical experiments. Furthermore, we present three case studies that
require the use of PLQ cost functions: (i) robust image modeling, (ii) tag
refinement for image annotation and retrieval and (iii) computing empirical
confidence limits for subspace clustering.
