Leveraging in-domain supervision for unsupervised image-to-image
  translation tasks via multi-stream generators
  Supervision for image-to-image translation (I2I) tasks is hard to come by,
but bears significant effect on the resulting quality. In this paper, we
observe that for many Unsupervised I2I (UI2I) scenarios, one domain is more
familiar than the other, and offers in-domain prior knowledge, such as semantic
segmentation. We argue that for complex scenes, figuring out the semantic
structure of the domain is hard, especially with no supervision, but is an
important part of a successful I2I operation. We hence introduce two techniques
to incorporate this invaluable in-domain prior knowledge for the benefit of
translation quality: through a novel Multi-Stream generator architecture, and
through a semantic segmentation-based regularization loss term. In essence, we
propose splitting the input data according to semantic masks, explicitly
guiding the network to different behavior for the different regions of the
image. In addition, we propose training a semantic segmentation network along
with the translation task, and to leverage this output as a loss term that
improves robustness. We validate our approach on urban data, demonstrating
superior quality in the challenging UI2I tasks of converting day images to
night ones. In addition, we also demonstrate how reinforcing the target dataset
with our augmented images improves the training of downstream tasks such as the
classical detection one.
