{'fraction_non_alphanumeric': 0.04530589998637417, 'fraction_numerical': 0.00640414225371304, 'mean_word_length': 5.241071428571429, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
X-modaler: A Versatile and High-performance Codebase for Cross-modal Analytics

With the rise and development of deep learning over the past decade, there has been a steady momentum of innovation and breakthroughs that convincingly push the state-of-the-art of cross-modal analytics between vision and language in multimedia field. Nevertheless, there has not been an open-source codebase in support of training and deploying numerous neural network models for cross-modal analytics in a unified and modular fashion. In this work, we propose X-modaler -- a versatile and high-performance codebase that encapsulates the state-of-the-art cross-modal analytics into several general-purpose stages (e.g., pre-processing, encoder, cross-modal interaction, decoder, and decode strategy). Each stage is empowered with the functionality that covers a series of modules widely adopted in state-of-the-arts and allows seamless switching in between. This way naturally enables a flexible implementation of state-of-the-art algorithms for image captioning, video captioning, and vision-language pre-training, aiming to facilitate the rapid development of research community. Meanwhile, since the effective modular designs in several stages (e.g., cross-modal interaction) are shared across different vision-language tasks, X-modaler can be simply extended to power startup prototypes for other tasks in cross-modal analytics, including visual question answering, visual commonsense reasoning, and cross-modal retrieval. X-modaler is an Apache-licensed codebase, and its source codes, sample projects and pre-trained models are available on-line: https://github.com/YehLi/xmodaler.


INTRODUCTION
Vision and language are two fundamental capabilities of human intelligence. Humans routinely perform cross-modal analytics through the interactions between vision and language, supporting the uniquely human capacity, such as describing what they see with a natural sentence (image captioning [1,26,27] and video captioning [11,14,24]) and answering open-ended questions w.r.t the given image (visual question answering [3,9]). The valid question of how language interacts with vision motivates researchers to expand the horizons of multimedia area by exploiting cross-modal analytics in different scenarios. In the past five years, vision-to-language has been one of the "hottest" and fast-developing topics for cross-modal analytics, with a significant growth in both volume of publications and extensive applications, e.g., image/video captioning and the emerging research task of vision-language pre-training. Although numerous existing vision-to-language works have released the open-source implementations, the source codes are implemented in different deep learning platforms (e.g., Caffe, TensorFlow, and PyTorch) and most of them are not organized in a standardized and user-friendly manner. Thus, researchers and engineers have to make intensive efforts to deploy their own ideas/applications for vision-to-language based on existing open-source implementations, which severely hinders the rapid development of cross-modal analytics.
To alleviate this issue, we propose the X-modaler codebase, a versatile, user-friendly and high-performance PyTorch-based library that enables a flexible implementation of state-of-the-art vision-tolanguage techniques by organizing all components in a modular fashion. To our best knowledge, X-modaler is the first open-source codebase for cross-modal analytics that accommodates numerous kinds of vision-to-language tasks.
Specifically, taking the inspiration from neural machine translation in NLP field, the typical architecture of vision-to-language models is essentially an encoder-decoder structure. An image/video is first represented as a set of visual tokens (regions or frames), CNN representation, or high-level attributes via pre-processing, which are further transformed into intermediate states via encoder (e.g., LSTM, Convolution, or Transformer-based encoder). Next, conditioned the intermediate states, the decoder is utilized to decode each word at each time step, followed by decode strategy module (e.g., greedy decoding or beam search) to compose the final output sentence. More importantly, recent progress on cross-modal analytics has featured visual attention mechanisms, that trigger the cross-modal interaction between the visual content (transformed by encoder) and the textual sentence (generated by decoder) to boost vision-to-language. Therefore, an additional stage of cross-modal interaction is commonly adopted in state-of-the-art vision-tolanguage techniques. The whole encoder-decoder structure can  Figure 1: An overview of the architecture in X-modaler, which is composed of seven major stages: pre-processing, encoder, cross-modal interaction, decoder, decode strategy, training strategy, and pre-training. Each stage is empowered with the functionality that covers a series of widely adopted modules in state-of-the-arts.
be optimized with different training strategies (e.g., cross entropy loss, or reinforcement learning). In addition, vision-language pre-training approaches (e.g., [10,13,30]) go beyond the typical encoder-decoder structure by including additional pre-training objectives (e.g., masked language modeling and masked sentence generation). In this way, the state-of-the-art cross-modal analytics techniques can be encapsulated into seven general-purpose stages: pre-processing, encoder, cross-modal interaction, decoder, decode strategy, training strategy, and pre-training. Following this philosophy, X-modaler is composed of these seven general-purpose stages, and each stage is empowered with the functionality that covers a series of commonly adopted modules in state-of-the-arts. Such modular design in X-modaler enables a flexible implementation of state-of-the-art algorithms for vision-tolanguage, and meanwhile allows the easy plug-ins of user-defined modules to facilitate the deployment of novel ideas/applications for cross-modal analytics.
In summary, we have made the following contributions: (I). To our best knowledge, X-modaler is the first open-source codebase that unifies comprehensive high-quality modules for cross-modal analytics. (II). X-modaler provides the easy implementations of state-of-the-art models for image captioning, video captioning, and vision-language pre-training, in a standardized and user-friendly manner. Moreover, X-modaler can be simply extended to support other vision-language tasks, e.g., visual question answering, visual commonsense reasoning, and cross-modal retrieval. (III). In X-modaler, we release all the reference codes, pre-trained models, and tools for each vision-language task, which will offer a fertile ground for deploying cross-modal analytics in industry and designing novel architectures in academia.

ARCHITECTURE
In this section, we present the detailed architecture in our X-modaler consisting of seven major stages, as shown in Figure 1. Figure 2 depicts the detailed class diagram of our X-modaler codebase.

Pre-processing
The pre-processing stage is utilized to transform the primary inputs of image/video and textual sentence into visual and textual tokens. Besides the typical tokenizing, we include numerous modules to represent each input image/video in different ways: (1) directly taking the output CNN Representation of fully-connected layers in 2D/3D CNNs as image/video features; (2) detecting a set of regions as bottom-up signals via Faster R-CNN as in [1]; (3) recognizing objects from visual content via pre-trained object classifier [25]; (4) extracting high-level semantic attributes through Multiple Instance Learning [15,28]; (5) exploring the semantic or spatial object relations between every two regions [26].

Encoder
The encoder stage is to take the visual/textual tokens as input and produce intermediate states to encode the semantic content. After transforming each visual/textual token via visual/textual embedding, the most typical way to construct encoder is to directly adopt LSTM to sequentially encode the sequence of tokens. The module of Graph Convolutional Networks (GCN) [26] can be further utilized to strengthen each region-level encoded features by exploiting the graph structure among regions. Instead of the sequential modeling in LSTM module, Convolution module [2,6] fully employs convolutions in encoder to enable parallelization within a sequence during training. Inspired by recent success of Transform-style encoder-decoder in NLP field [20], we include the self-attention module that leverages self-attention mechanism to enhance each local (region/frame) feature by exploring the intramodal feature interactions.

Cross-modal Interaction
The cross-modal interaction stage aims to boost vision-language tasks by encouraging more interactions between the two different modalities. Attention module denotes the conventional attention mechanism that dynamically measures the contribution of each local image region [23] or frame [24] based on the hidden state of decoder. Top-down attention module [1] exploits visual attention at object level. Co-attention module [12] enables bi-directional interaction between visual and textual tokens. Meshed memory attention module [8] utilizes memory-augmented attention that boosts inter-modal interaction with a priori knowledge. X-Linear attention module [16] models the higher order interactions with both spatial and channel-wise bilinear attention.

Decoder
The decoder stage targets for decoding each word at each time step conditioned on the intermediate states of the inputs induced by the encoder. Similar to the modules of encoder stage, the decoder can also be constructed in different forms: LSTM/GRU that autoregressively produces each word, Convolution which fully leverages convolutions for decoding sentence, or Transformer that first exploits the word dependency via self-attention mechanism and further captures the co-attention across vision & language via cross-attention mechanism to facilitate word generation.

Decode Strategy
The decode strategy stage includes two common modules to generate the final output sentence at inference: (1) greedy decoding that samples the word with the maximum probability at each time step until we select the special end-of-sentence token or reach the maximum length; (2) beam search, i.e., a heuristic search algorithm that maintains a beam including several most likely partial sentences at each decoding time step.

Training Strategy
In the training strategy stage, we assemble a series of practical training strategies adopted in state-of-the-art vision-to-language models: (1) cross entropy module that penalizes the prediction of each word with cross entropy loss; (2) label smoothing module [19] further regularizes the classifier for word prediction by estimating the marginalized effect of label-dropout; (3) schedule sampling module [4] capitalizes on a curriculum learning strategy to gently change the input token at each time step from the ground-truth word token to the estimated one from previous step; (4) reinforcement learning module [17] enables the direct optimization of the whole encoder-decoder structure with expected sentence-level reward loss.

Pre-training
To endow the base encoder-decoder structure with the capabilities of multi-modal reasoning for vision-language pre-training, we involve the pre-training stage that pre-trains the base structure  with several vision-language proxy tasks, e.g., masked language modeling, masked sentence generation, and visual-sentence matching as in [10,30].

APPLICATIONS AND EVALUATIONS
This section details the exemplary implementations of each visionlanguage task (as shown in Figure 3) in our X-modaler, coupled with the experimental results over several vision-language benchmarks (e.g., COCO [7], MSVD [5], and MSR-VTT [22]) for image/video captioning task. Image/Video Captioning. This task aims to auto-regressively generate the natural sentence that depicts the visual content of input image/video. We re-implement several state-of-the-art image captioning approaches (e.g., 2 Transformer [8] and X-LAN [16]) and video captioning methods (e.g., TDConvED [6] and Transformer [18]) by allocating different modules in the unified encoder-decoder paradigm (see Figure 3 (a)). Table 1 and 2 show the performance comparison of our re-implemented methods through X-modaler over COCO for image captioning and MSVD & MSR-VTT for video captioning, which manage to achieve state-of-the-art performances on each benchmark. Vision-language Pre-training (VLP). VLP is to pre-train a unified encoder-decoder structure over large-scale vision-language benchmarks, which can be easily adapted to vision-language downstream tasks. The state-of-the-art VLP models (e.g., TDEN [10] and ViLBERT [12]) can be implemented as a two-stream Transformer structure (see Figure 3 (b)): object and sentence encoders first separately learns the representations of each modality, and the cross-modal interaction module further performs multi-modal reasoning, followed with the decoder for sentence generation. Visual Question Answering (VQA). In VQA, the model predicts an answer to the given natural language question with regard to an image. Here we show the implementation of a base model for this task in Figure 3 (c). This base model first encodes the input question and image via object and sentence encoders, and further utilizes cross-modal interaction module (e.g., the attention mechanism in [29]) to achieve the holistic image-question representation. Finally, a single-layer MLP is leveraged as a classifier to predict answer based on the holistic image-question representation.
Cross-modal Retrieval. It aims to search an image/caption from a pool given its caption/image. It is natural to formulate this task as a ranking problem that sorts images/captions according to the learnt image-sentence matching scores. Here the image-sentence matching score can be directly measured as the dot product between the encoded features of image and caption (see Figure 3 (d)). Visual Commonsense Reasoning (VCR). VCR tackles two problems: visual question answering and answer justification, that requires the model to predict an answer or judge the correctness of the chosen rationale respectively. Each problem is framed as multiple choice task. Similar to VQA, we can measure the holistic imagesentence feature via cross-modal interaction module based on the multi-modal outputs of object and sentence encoders, which will be further leveraged to predict the score for each possible response via a classifier (see Figure 3 (e)).