{'fraction_non_alphanumeric': 0.028779376632035114, 'fraction_numerical': 0.011167287071503973, 'mean_word_length': 4.905511811023622, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Multimodal E-Commerce Product Classification Using Hierarchical Fusion

In this work, we present a multi-modal model for commercial product classification, that combines features extracted by multiple neural network models from textual (CamemBERT and FlauBERT) and visual data (SE-ResNeXt-50), using simple fusion techniques. The proposed method significantly outperformed the unimodal models' performance and the reported performance of similar models on our specific task. We did experiments with multiple fusing techniques and found, that the best performing technique to combine the individual embedding of the unimodal network is based on combining concatenation and averaging the feature vectors. Each modality complemented the shortcomings of the other modalities, demonstrating that increasing the number of modalities can be an effective method for improving the performance of multi-label and multimodal classification problems.


I. INTRODUCTION
Product profiles on e-commerce platforms typically contain unstructured text, product titles, and images, and provide details about the product that is important to e-commerce. The profiles help customers make purchasing decisions and, on the other hand, provide input for automated services of e-commerce sites, such as question-answering and product recommendation systems. Since the number of products and the number of product categories in an e-commerce platform can be so high that it is unmanageable for humans, it is important to have methods that are able to accurately assign these products to their categories [3].
Most research on product classification has focused on the use of text-based cues, even though products on e-commerce platforms contain more than textual information and ignore the valuable information that associated images contain. Therefore, it might be useful to experiment with multimodal models that can combine features extracted from different modalities.
In this work, we proposed a multimodal product classification model based on several pre-trained transformer-based architectures for language modeling and image classification. The transformer-based language models were used to learn a representation of text features from the textual information, and pre-trained visual approaches are used to obtain a representation from images. We used hierarchical fusion to combine the representations obtained from the language models and the visual model, and a fully connected layer to perform the final classification.
We empirically evaluated the proposed method in the Multimodal Product Classification Task of the SIGIR 1 2020 E-Commerce Workshop Data Challenge. The results of the experiments show superior performance compared to unimodal and other multimodal methods. Our experiments have also shown that text-based classification models generally perform better than visual models. On the other hand, the inclusion of both modalities allows us to benefit from complimentary feature information and outperformed baseline approaches.
The remainder of the paper is organized into six sections. Section II describes related work relevant to our research. Section III presents our proposed multimodal hierarchical fusion model architecture, section IV presents the general experimental settings and implementation, while in section V we summarize the experimental results and discussion based on the experiments. Finally, section VI presents our conclusions.

II. RELATED WORK
In 2020, a large-scale competition to classify multimodal product data was organized as part of the SIGIR 2020 ecommerce workshop. The dataset was provided by Rakuten France, and each product included a title and a detailed description in French, as well as a product image. The task was to predict 27 category labels from four major genres: books, children, households, and entertainment. Several teams and researchers participated [2], [4], [9], [11]. The most common solution proposed by most authors was to fine-tune pre-trained text and image models as feature extractors and then use a bimodal fusion mechanism to combine predictions. Most teams used the Transformer-based model BERT [5] for text feature extraction and ResNet [6] for image feature extraction. For bimodal fusion, the methods used were even more diverse. Roughly in order of increasing complexity, the methods included simple late decision-level fusion [9] and coattention [2].
Bidirectional Encoder Representations from Transformers (BERT) is a state-of-the-art open-source language model [1]. BERT Models are fine-tuned for specific goals. Sun et al. [12] demonstrated improved text classification performance by using various strategies, such as layer selection, layer-wise learning rate, etc. BERT has been used in a variety of ways to improve language processing tasks. Variations of BERT, trained on large French corpora, include FlauBERT [8] and CamemBERT [10].
In this study, we will fine-tune the pre-trained French language models for each textual modality separately and the SE -ResNeXt-50 [7] for image products to extract features for each modality. The extracted features are hierarchically fused using different fusion strategies to make the prediction.

III. HIERARCHICAL FUSION MODEL
In this section, we describe our proposed model architecture for classifying product types.
The core of our method is to project the unimodal representation sequences generated by the transformers P , T f , T c and D f , D c into a common representation space and then learn a classifier in this common space: where Z is a fusion function that combines the extracted features, and f is a neural network classifier that maps the features of the resulting joint space to a class prediction y. P is the reduced image representation in SE -ResNeXt-50 [7], T f , T c , D f , D c are the representation sequences extracted from the titles and the descriptions by FlauBERT [8] and CamemBERT [10], respectively. The proposed hierarchical model uses state-of-the-art pretrained architectures in the base models. In developing the fusion model, we have tried to keep it as simple as possible, since a simple architecture can provide good performance with less complexity and cost. The fusion architecture is a simple trainable module.

A. Text Encoding Models
We fine-tuned the two French-language models, FlauBERT and CamemBERT, for encoding textual information of the product title and description into the textual sequence representation. FlauBERT was trained with unsupervised masked language modeling (MLM, similar to BERT), while Camem-BERT was pre-trained on French text based on Facebook's RoBERTa model.

B. Image Encoding Model
For the purpose of obtaining representations of product images, we use the SE-ResNeXt-50 [7] model. The last fully-connected (FC) layer of the pre-trained model has been replaced, and the model has been fine-tuned for our categorization task (that will be removed for the feature extraction). Based on the work of [13]- [15], an input image I is resized to 224 × 224 and divided into 16 × 16 regions. A raw image vector, a.k.a. a regional feature representation, is then produced by applying the SE-ResNeXt-50 model to each region I i (i = 1, 2, . . . , 256). The final image feature vector (P ) is then obtained by averaging all regional image vectors.
where ResNeXt(I i ) is the raw image vector extracted via SE-ResNeXt-50 from region i, N r (256 in this work) is the number of regions. The final visual representation P full is the average of all regional image vectors.
C. Multi-modal Joint Representation Learning P represents the visual features extracted by running the input images through the SE-ResNeXt-50 model. We used a 1D convolution layer with max-pooling (P ∈ R d ) to fit the dimension of the image representation to the textual representations. Similarly, textual features extracted by passing titles to CamemBERT and FlauBERT models are represented as T _c, T _f ∈ (R d ) and descriptions extracted by passing descriptions through CamemBERT and FlauBERT models are represented as D f , D c ∈ (R d ). To combine the text and image features, we implemented the following fusing functions (Z): • Addition Fusion: The sum of input representations is defined as the addition operation. Given two inputs (x 1 , x 2 ) of any modalities with the same dimension d, the addition operation yields the output X add = The concatenation operation is defined as the concatenation of representations across a dimension. In terms of the base models: X con =< x 1 , x 2 > , where X con ∈ R d×2 and <> is the concatenation operation. • Average Fusion: The mean of the input representations is used to define the average operation.

D. Architecture
Our suggested strategy involves training a fully connected neural network model that uses the hierarchically fused embedding vectors generated by the three unimodal networks to construct a new probability vector over the target categories. Our research aims to discover the most effective strategies to combine the representations discovered by several distinct  Figure 1 consists of the following layers: • Input layer: The input layer receives and preprocesses visual and textual modalities . We feed the textual base models with the product titles and descriptions individually, resulting in four different textual embeddings, and split the input images into regions before passing them to SE-ResReXt-50.
• Representation layer: Using fine-tuned unimodal networks, the Representation layer extracts feature vectors from preprocessed input data. It also combines the SE-ResReXt-50 outputs by region.
• Intermediate fusion layer: The embeddings of the text modalities (product title and description) learned from each language model (CamemBERT, FlauBERT) were first fused using one of our fusion methods in the Intermediate fusion layer, then the fused results from CamemBERT and FlauBERT were further fused to obtain the final vector representation for text modalities.
• Feed forward layer. Finally, the fused vector is passed through a three-layer fully connected neural network to produce a classification result. The classifications are generated using the softmax activation function.
When training the fusion model, the uni-modal network layers are frozen, and the refined product description and title representations are combined to produce a single refined representation, which, along with the reduced image representation, serves as input to our fully connected neural network. The fusion layers' operations produced various variants of the proposed hierarchical architecture.

A. Dataset
We used a dataset provided by the Rakuten Institute of Technology for the Rakuten Multi-modal Product Data Classification and Retrieval Challenge to test the effectiveness of our proposed model. This task's data set contains 99K products, nearly 84K of which were included in the training dataset. There are 27 different product type codes in the text and image dataset, totaling 55K unique products. The four parent domains that group the entire product catalog are Children, Household, Entertainment, and Books. The text data is in French and consists of the title and description of the product.
The description field has a mean length of 35 words and a maximum length of 2068 words, while the title field has a mean length of 11 words and a maximum length of 56 words. The product images are all square, with white or black borders and a 500 × 500 pixel resolution. Each product is associated with an image. We preprocessed the data and performed the following steps: • Text Pre-processing: We tokenize the product title and description using the NLTK 2 tokenizer, lowercase the text and remove HTML tags. • Product images: To fine tune the pretrained mage models were augmented by randomly rotating, flipping, and extracting random crops from the original images. • Product images were resized to a uniform size of 224 × 224 and normalized per channel using ImageNet mean and standard deviation.

B. Implementation Details
PyTorch was used to create all of our models. We employed the 138M parameter F lauBERT base architecture with encoded vocabulary, as well as the 110M parameter CamemBERT base architecture. With a batch size of 32 and The pre-trained weights of the trained uni-modal base models were used to fine-tune the learning module for the multimodal representation. The uni-modal network layers were frozen, and we used feature fusion methods to train our suggested multimodal design, followed by a fully connected. network with a Softmax output layer to perform product classification. As a loss function, we employed the categorical cross-entropy minimization objective, the AdamW optimizer with a learning rate value of 2 × 10 −5 for text models, and the Adam optimizer with 10 −3 for image models.

V. RESULTS AND DISCUSSION
Multi-modal models were created with the goal of combining the representations given by uni-modal models to produce more accurate, reliable, and robust predictions. Several versions have been developed, with varying levels of overall performance. In the following sections, we present our findings. We split the original training set and utilized it for training and testing because the labels for the test set were not available.
We employed two alternative splits to train and test our proposed methods in order to assess the generalization power of our model. We used 90% for training and 10% for testing in the first phase. 10% from the training was also used as a validation set. We used 80% for training and 20% for testing in the second phase. A validation set made up of 10% of the 80 percent training was also utilized. The 90/10 split was utilized to create a test set that was around the same size as the challenge.

A. 90/10, Train/test Split
In the first experiment, we used a 90/10 split between training and testing, using 90% for training and 10% for testing the model. Table I shows the performance of each uni-modal model, namely CamemBERT-title, CamemBERT-description, FlauBERT-title, FlauBERT-description, and SE -ResNeXt individually. TableII shows the multimodal hierarchical fusion models on the test set using the concatenation/average fusion method in the first fusion layer (fusion of text embeddings) and different combination methods in the last fusion layer. In the proposed hierarchical fusion architecture, it is noticeable that the model using the average combination in both fusion layers performs well.

B. Introducing More Layers
After running different experiments, the model using the average fusion method in both fusion layers showed the best performance compared to other models trained with a 90/10 split of the dataset. We started incrementally experimenting with Decision Layer by adding additional layers to further improve the performance of this model as follows: • Adding a dropout layer after the last fusion layer. • Stacking a dropout layer followed by a fully connected (FC) layer and a non-linearity before the final projection layer. Table III presents the performance of the basic proposed architecture and the modified architecture having stacked dropout out, FC and non-linearity after the final fusion layer. As we can see in Table III, the performance of the models decreased when we added more additional layers. Therefore, the modified model with more layers did not seem to improve the results, while the simplest architecture ( Figure 1) showed better overall performance and reduced the training time, giving room for further experiments.
It is noticeable in the simplest hierarchical architecture that the model that used average combination in both fusion layers has a good performance on the test set. However, to obtain a robust model, further investigation is needed to verify that the model is not overfitted. To address the issue, we split the dataset in a way that allows us to obtain more test set, which is the second split strategy mentioned in section V.

C. 80/20, Train/Test Split
This spilt was performed to see how the models performs on less training set and a bit more test set. Table IV and  table V shows the performance of uni-modal models and the proposed hierarchical model variations in terms of accuracy and F1 Score on the test set of the given dataset split, using Concatenation/Average fusion method in the first fusion layer (textual embeddings' fusion) and different combination methods in the final fusion layer. We had the best performance when we used concatenation in the first and then averaging in the second fusion layer in the 80/20 split as shown in table V, but full average fusion outperformed the concatenation based models in the validation and test sets as evidenced in table II. The best model has achieved an overall F1 Score of 92.67%, which indicates that merging the two modalities boosts the performance compared to uni-modal networks.

VI. CONCLUSION
In this work, multiple Deep Learning based pre-trained and fine-tuned models were used with different fusion strategies with the goal of producing a simple, easily trainable and good quality model for multimodal e-commerce product classification. We used a dataset provided by Rakuten Institute of Technology for the Rakuten Multi-modal Product Data Classification and Retrieval Challenge that contains 27 different product type codes and a total of 55K unique products.
The best performance was obtained using the average fusion operation in all fusion layers, where the model reached a F1-Score of 92.67% on the test set given 90/10 dataset split. When it came to 80/20 dataset split, the best achieved model was concatenation-based in fusing textual features, and average-based fusion in the final fusion layer with F1-Score of 92.46%. Our experimental results in general showed that fine-tuning pre-trained models for feature vector representation with simple hierarchical fusion approaches can yield good results in multimodal e-commerce product classification.