Extending CLIP for Category-to-image Retrieval in E-commerce
  E-commerce provides rich multimodal data that is barely leveraged in
practice. One aspect of this data is a category tree that is being used in
search and recommendation. However, in practice, during a user's session there
is often a mismatch between a textual and a visual representation of a given
category. Motivated by the problem, we introduce the task of category-to-image
retrieval in e-commerce and propose a model for the task, CLIP-ITA. The model
leverages information from multiple modalities (textual, visual, and attribute
modality) to create product representations. We explore how adding information
from multiple modalities (textual, visual, and attribute modality) impacts the
model's performance. In particular, we observe that CLIP-ITA significantly
outperforms a comparable model that leverages only the visual modality and a
comparable model that leverages the visual and attribute modality.
