Fast Line Search for Multi-Task Learning
  Multi-task learning is a powerful method for solving several tasks jointly by
learning robust representation. Optimization of the multi-task learning model
is a more complex task than a single-task due to task conflict. Based on
theoretical results, convergence to the optimal point is guaranteed when step
size is chosen through line search. But, usually, line search for the step size
is not the best choice due to the large computational time overhead. We propose
a novel idea for line search algorithms in multi-task learning. The idea is to
use latent representation space instead of parameter space for finding step
size. We examined this idea with backtracking line search. We compare this fast
backtracking algorithm with classical backtracking and gradient methods with a
constant learning rate on MNIST, CIFAR-10, Cityscapes tasks. The systematic
empirical study showed that the proposed method leads to more accurate and fast
solution, than the traditional backtracking approach and keep competitive
computational time and performance compared to the constant learning rate
method.
