Mining Fashion Outfit Composition Using An End-to-End Deep Learning
  Approach on Set Data
  Composing fashion outfits involves deep understanding of fashion standards
while incorporating creativity for choosing multiple fashion items (e.g.,
Jewelry, Bag, Pants, Dress). In fashion websites, popular or high-quality
fashion outfits are usually designed by fashion experts and followed by large
audiences. In this paper, we propose a machine learning system to compose
fashion outfits automatically. The core of the proposed automatic composition
system is to score fashion outfit candidates based on the appearances and
meta-data. We propose to leverage outfit popularity on fashion oriented
websites to supervise the scoring component. The scoring component is a
multi-modal multi-instance deep learning system that evaluates instance
aesthetics and set compatibility simultaneously. In order to train and evaluate
the proposed composition system, we have collected a large scale fashion outfit
dataset with 195K outfits and 368K fashion items from Polyvore. Although the
fashion outfit scoring and composition is rather challenging, we have achieved
an AUC of 85% for the scoring component, and an accuracy of 77% for a
constrained composition task.
