Spectral Algorithm for Low-rank Multitask Regression
  Multitask learning, i.e. taking advantage of the relatedness of individual
tasks in order to improve performance on all of them, is a core challenge in
the field of machine learning. We focus on matrix regression tasks where the
rank of the weight matrix is constrained to reduce sample complexity. We
introduce the common mechanism regression (CMR) model which assumes a shared
left low-rank component across all tasks, but allows an individual per-task
right low-rank component. This dramatically reduces the number of samples
needed for accurate estimation. The problem of jointly recovering the common
and the local components has a non-convex bi-linear structure. We overcome this
hurdle and provide a provably beneficial non-iterative spectral algorithm.
Appealingly, the solution has favorable behavior as a function of the number of
related tasks and the small number of samples available for each one. We
demonstrate the efficacy of our approach for the challenging task of remote
river discharge estimation across multiple river sites, where data for each
task is naturally scarce. In this scenario sharing a low-rank component between
the tasks translates to a shared spectral reflection of the water, which is a
true underlying physical model. We also show the benefit of the approach on the
markedly different setting of image classification where the common component
can be interpreted as the shared convolution filters.
