PAM: Understanding Product Images in Cross Product Category Attribute
  Extraction
  Understanding product attributes plays an important role in improving online
shopping experience for customers and serves as an integral part for
constructing a product knowledge graph. Most existing methods focus on
attribute extraction from text description or utilize visual information from
product images such as shape and color. Compared to the inputs considered in
prior works, a product image in fact contains more information, represented by
a rich mixture of words and visual clues with a layout carefully designed to
impress customers. This work proposes a more inclusive framework that fully
utilizes these different modalities for attribute extraction. Inspired by
recent works in visual question answering, we use a transformer based sequence
to sequence model to fuse representations of product text, Optical Character
Recognition (OCR) tokens and visual objects detected in the product image. The
framework is further extended with the capability to extract attribute value
across multiple product categories with a single model, by training the decoder
to predict both product category and attribute value and conditioning its
output on product category. The model provides a unified attribute extraction
solution desirable at an e-commerce platform that offers numerous product
categories with a diverse body of product attributes. We evaluated the model on
two product attributes, one with many possible values and one with a small set
of possible values, over 14 product categories and found the model could
achieve 15% gain on the Recall and 10% gain on the F1 score compared to
existing methods using text-only features.
