Multi-Modal Attribute Extraction for E-Commerce
  To improve users' experience as they navigate the myriad of options offered
by online marketplaces, it is essential to have well-organized product
catalogs. One key ingredient to that is the availability of product attributes
such as color or material. However, on some marketplaces such as
Rakuten-Ichiba, which we focus on, attribute information is often incomplete or
even missing. One promising solution to this problem is to rely on deep models
pre-trained on large corpora to predict attributes from unstructured data, such
as product descriptive texts and images (referred to as modalities in this
paper). However, we find that achieving satisfactory performance with this
approach is not straightforward but rather the result of several refinements,
which we discuss in this paper. We provide a detailed description of our
approach to attribute extraction, from investigating strong single-modality
methods, to building a solid multimodal model combining textual and visual
information. One key component of our multimodal architecture is a novel
approach to seamlessly combine modalities, which is inspired by our
single-modality investigations. In practice, we notice that this new
modality-merging method may suffer from a modality collapse issue, i.e., it
neglects one modality. Hence, we further propose a mitigation to this problem
based on a principled regularization scheme. Experiments on Rakuten-Ichiba data
provide empirical evidence for the benefits of our approach, which has been
also successfully deployed to Rakuten-Ichiba. We also report results on
publicly available datasets showing that our model is competitive compared to
several recent multimodal and unimodal baselines.
