Exact Pareto Optimal Search for Multi-Task Learning: Touring the Pareto
  Front
  Multi-Task Learning (MTL) is a well-established paradigm for training deep
neural network models for multiple correlated tasks. Often the task objectives
conflict, requiring trade-offs between them during model building. In such
cases, MTL models can use gradient-based multi-objective optimization (MOO) to
find one or more Pareto optimal solutions. A common requirement in MTL
applications is to find an {\it Exact} Pareto optimal (EPO) solution, which
satisfies user preferences with respect to task-specific objective functions.
Further, to improve model generalization, various constraints on the weights
may need to be enforced during training. Addressing these requirements is
challenging because it requires a search direction that allows descent not only
towards the Pareto front but also towards the input preference, within the
constraints imposed and in a manner that scales to high-dimensional gradients.
We design and theoretically analyze such search directions and develop the
first scalable algorithm, with theoretical guarantees of convergence, to find
an EPO solution, including when box and equality constraints are imposed. Our
unique method combines multiple gradient descent with carefully controlled
ascent to traverse the Pareto front in a principled manner, making it robust to
initialization. This also facilitates systematic exploration of the Pareto
front, that we utilize to approximate the Pareto front for multi-criteria
decision-making. Empirical results show that our algorithm outperforms
competing methods on benchmark MTL datasets and MOO problems.
