Emerging Relation Network and Task Embedding for Multi-Task Regression
  Problems
  Multi-task learning (mtl) provides state-of-the-art results in many
applications of computer vision and natural language processing. In contrast to
single-task learning (stl), mtl allows for leveraging knowledge between related
tasks improving prediction results on the main task (in contrast to an
auxiliary task) or all tasks. However, there is a limited number of comparative
studies on applying mtl architectures for regression and time series problems
taking recent advances of mtl into account. An interesting, non-linear problem
is the forecast of the expected power generation for renewable power plants.
Therefore, this article provides a comparative study of the following recent
and important mtl architectures: Hard parameter sharing, cross-stitch network,
sluice network (sn). They are compared to a multi-layer perceptron model of
similar size in an stl setting. Additionally, we provide a simple, yet
effective approach to model task specific information through an embedding
layer in an multi-layer perceptron, referred to as task embedding. Further, we
introduce a new mtl architecture named emerging relation network (ern), which
can be considered as an extension of the sluice network. For a solar power
dataset, the task embedding achieves the best mean improvement with 14.9%. The
mean improvement of the ern and the sn on the solar dataset is of similar
magnitude with 14.7% and 14.8%. On a wind power dataset, only the ern achieves
a significant improvement of up to 7.7%. Results suggest that the ern is
beneficial when tasks are only loosely related and the prediction problem is
more non-linear. Contrary, the proposed task embedding is advantageous when
tasks are strongly correlated. Further, the task embedding provides an
effective approach with reduced computational effort compared to other mtl
architectures.
