{'fraction_non_alphanumeric': 0.032922011412963956, 'fraction_numerical': 0.011266643905769888, 'mean_word_length': 4.805209513023782, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
CaMEL: Mean Teacher Learning for Image Captioning

Describing images in natural language is a fundamental step towards the automatic modeling of connections between the visual and textual modalities. In this paper we present CaMEL, a novel Transformer-based architecture for image captioning. Our proposed approach leverages the interaction of two interconnected language models that learn from each other during the training phase. The interplay between the two language models follows a mean teacher learning paradigm with knowledge distillation. Experimentally, we assess the effectiveness of the proposed solution on the COCO dataset and in conjunction with different visual feature extractors. When comparing with existing proposals, we demonstrate that our model provides state-of-the-art caption quality with a significantly reduced number of parameters. According to the CIDEr metric, we obtain a new state of the art on COCO when training without using external data. The source code and trained models are publicly available at: https://github.com/aimagelab/camel.

Regardless of these architectural and structural improvements, the training methodology has remained almost unaltered. Indeed, most of the existing approaches for image captioning are based on the usage of a single language model, trained to reproduce the ground-truth caption through a crossentropy loss and later, in a fine-tuning stage, through the REINFORCE algorithm [9], [23]. In this paper, we take a different path and investigate the development of a training strategy that is based on the interplay between two distinct language models. In particular, we draw inspiration from the Mean Teacher Learning approach [24] -which has been successfully employed to learn visual representation in a selfsupervised manner [25] -and propose a schema in which two language models learn and interact together at training time.
One of the two language models is employed as teacher, while the other is employed as a student in a knowledge distillation relationship [26], [27], [28]. Parameters update, on the teacher model, is carried out by averaging successive states of the student, through an exponential moving average. In this way, the teacher slowly "follows" the student state through time. We devise and compare strategies to apply this interplay paradigm in both the cross-entropy training stage and during the finetuning with reinforcement learning. Noticeably, at test time one of the two language models can be discarded, so that the number of parameters is kept on pair with traditional models that employ a single language model at training time. We name our model CaMEL -short for Captioner with Mean tEacher Learning. As shown in Fig. 1, our model outperforms existing approaches in terms of caption quality, while being significantly less demanding in terms of number of parameters. We assess the performances of the proposed training strategy on the COCO dataset [29], employing different knowledge distillation strategies, and in comparison with other state-of-the-art approaches that have been trained on the same dataset. We also compare our results on the COCO online test server. Results demonstrate the appropriateness of the proposed solution, which attains a new state of the art on COCO when training without using external data.

II. RELATED WORK
Early deep learning-based image captioning approaches entail encoding the visual information by using a CNN-based encoder, and then generating the textual output by using RNNbased language models conditioned on the encoding [2], [3], [9], [30].
Visual encoding. To better capture spatial information and structure, the encoding component has been modified to introduce attention mechanisms, which rely on grids of CNN features in early works [4], [31], [32], and on image regions containing visual entities in following works [6], [33]. An alternative strategy to model spatial and semantic relations between the objects in the image is employing graph convolution neural networks [34], [35], [36]. Finally, a more recent trend entails applying Transformer-like architectures as visual encoders [15], [37], being those also applicable to image patches directly [38], [39]. In our case, we follow the recent paradigm of employing features extracted from largescale multi-modal architectures [16], [17] like CLIP [18].
Training strategies. The training strategy for image captioning architectures usually follows the time-wise crossentropy paradigm. This was later combined with a finetuning phase based on the application of the REINFORCE algorithm, to allow using as optimization objectives captioning metrics directly [9], [23], overcoming the issue of their nondifferentiability and boosting the final performance. As a strategy to improve both training phases, in [51] it is proposed to exploit a teacher model trained on image attributes to generate additional supervision signals for the captioning model. These are in the form of soft-labels, which the captioning model has to align with in the cross-entropy phase, and reweighting of the caption words to guide the fine-tuning phase. Additional improvement to the performance of recent selfattention-based image captioning approaches is due to the use of large-scale vision-and-language pre-training [17], [19], [20], [22], [50], which can be done on noisy image-text pairs, also exploiting pre-training losses different from cross-entropy, such as the masked token loss [19], [20]. Different from previous methods, our approach is based on the interplay of two different language models that are trained with the mean teacher learning paradigm and knowledge distillation, without relying on large-scale pre-training.

A. Preliminaries
Most captioning approaches rely on a single language model, which is conditioned on input images and is trained to reproduce ground-truth sentences. Formally, given a dataset of image-caption pairs D = {(v i , t i )} i , the language model aims at learning the probability distribution of the next word in a sequence, conditioned on the input image, i.e.
where v is an input image, τ indicates time, and {w τ } τ is the sequence of words comprising the generated caption. The model is trained according to a time-wise cross-entropy (XE) loss over the entire dataset, as follows: where θ indicates the set of parameters of the model. After a training stage with cross-entropy, sequence generation is usually fine-tuned using reinforcement learning. When training with XE, indeed, the model is trained to predict the next token given previous ground-truth words; in reinforcement learning the model is asked to generate an entire sequence and receives a reward that is proportional to the similarity of the generated caption with respect to the groundtruth. A standard practice is to employ a variant of the selfcritical sequence training (SCST) approach [9] on sequences sampled using beam search [6]: to decode, the top-k words are sampled from the language model probability distribution at each timestep, and a beam with the top-k sequences with the highest probability is maintained during the generation.
Following previous works [6], the usual practice is to use the CIDEr-D score as reward, as it well correlates with human judgment [52]. In our case, we baseline the reward using the mean of the rewards [13]. The final gradient expression for the SCST training is, therefore where w i is the i-th sentence in the beam, r(·) is the reward function, and b = i r(w i ) /k is the baseline, computed as the mean of the rewards obtained by the sampled sequences.

B. CaMEL
In CaMEL, instead of training a single language model, we rely on the interplay of two different language models -an online and a target language model, that interact and learn from each other during the training phase, both during the XE pre-training and during the SCST fine-tuning. At test time, each of the two language models can be used, alone, for captioning input images. The interaction between the online and target language models at training time is two-fold. The online language model is trained, either via XE or SCST, with respect to groundtruth captions. In addition, it performs knowledge distillation with the target language model. The target language model, in turn, updates its weights according to an exponential moving average of the online model weights. An overview of our approach is given in Fig. 2.
Knowledge distillation. The target network provides regression targets to train the online network. This is done through knowledge distillation -treating the online language model as a student network and the target model as a teacher.
Given a visual input v and a conditioning partial sentence w, at each timestep τ both networks provide output logits over a vocabulary of N tokens, denoted as p t,τ and p o,τ for the target model and the online model, respectively. Given the teacher, which is kept fixed, we learn to match these distributions by minimizing a mean squared error loss with respect to the parameters of the online network, i.e. min θo τ where θ o indicates the set of parameters of the online network.
Model averaging. The parameters of the target language model are updated as an exponential moving average [25] of the parameters of the online network θ o . Formally, the parameters of the target model are given by where θ t indicates the set of parameters of the target network and λ ∈ [0, 1] is a target decay rate. In practice, we keep λ fixed during the entire training process. This strategy results in the target network keeping a weighted average of successive states from the online network, thus performing a form of model ensembling.

detach() # stop gradient return (t -s).square().mean()
Objective. During XE pre-training, the final objective we employ to train the online network is a combination of the standard XE loss, which is computed with respect to groundtruth captions, and of the knowledge distillation loss with respect to the target logits. After each SGD update of the online network, the target network is updated through the model averaging. Algorithm 1 provides the PyTorch pseudocode of the training loop during the XE stage.
Extension to the SCST stage. The same training methodology is also applied during SCST fine-tuning. In this case, given that both language models generate a beam of k captions, there are k 2 pairs of sequences on which the MSE loss can be potentially applied. In the following, we experiment by matching the top-1 caption in each beam, according to the probability assigned by the models themselves or to the score assigned by an external image-text model. Further, we also experiment when matching each caption in the online beam with a caption in the target beam, and then applying the MSE loss on each pair.
Network architecture. CaMEL follows an encoder-decoder Transformer [45] architecture, where the encoder processes visual features via bi-directional attention and the decoder generates captions in an auto-regressive manner. Following [13], [49], our encoder incorporates additional memory slots, en- hancing its ability to encode knowledge and relations learned from visual data. Specifically, we expand the set of keys and values in self-attention layers with extra and independent learnable vectors, which can encode a priori knowledge retrieved through attention. Our decoder is composed of a stack of decoder layers, each performing a right-masked selfattention and a cross-attention followed by a position-wise feed-forward network. We also test the usage of a mesh-like connectivity between the encoder and the decoder, following [13]. In this case, the mesh mechanism further connects each encoder and decoder layer in a mesh-like structure, augmenting its ability to deal with low-or high-level features.
The architecture is the same for both online and target models, while embracing independent parameters updated with different strategies during training.

A. Dataset
Following the dominant paradigm in literature, we train and evaluate our model on the COCO dataset [29]. As such, we do not rely on large-scale image-text datasets [17]. COCO is composed of more than 120, 000 images, each of them associated with 5 human-collected captions. We follow the Karpathy et al. [2] splits, using 5, 000 images for both validation and testing and the rest for training. We also evaluate our model on the COCO online test server, which includes 40, 775 images for which annotated captions are not publicly available.
Implementation details. To represent words, we use Byte Pair Encoding (BPE) [58] with a vocabulary size of 49, 408, which is then linearly projected to the input dimensionality of the model. We use standard sinusoidal positional encodings [45] to represent word positions. All models comprise three layers in the visual encoder and three layers in the decoder, each with a dimensionality of 512, a feed-forward dimensionality of 2048, and a number of heads equal to 8. We apply dropout at the output of each sub-layer, with a dropout probability equal to 0.1. The number of memory slots is set to 40. In all experiments, we employ Adam [59] as optimizer and a beam size equal to 5. During pre-training with XE loss, we use a batch size of 50, following the typical Transformer learning rate scheduling strategy [45] with a warmup equal to 10, 000 iterations. During the SCST finetuning stage, we use a batch size of 30 and a fixed learning rate equal to 5 × 10 −6 . The MSE loss is computed by considering only valid tokens and masking the rest. The target network momentum λ is set to 0.999.

C. Ablation Study
Visual features assessment. We firstly discuss the role of visual features, by comparing traditional detection-based features with grid-based features extracted from modern multimodal models. In particular, we consider object detection features extracted from a Faster R-CNN model [6] pre-trained   on the Visual Genome dataset [60], and grid-based features extracted from CLIP [18], which has been trained with language supervision. Since CLIP visual encoders can be either based on ViT-like or CNN-like architectures, we either employ the output of the last Transformer layer, removing the CLS token, or extract the grid of features produced immediately after the last convolutional layer.
As it can be seen from Table I, Faster R-CNN features can be surpassed by modern multi-modal architectures, which brings a significant advantage in terms of all captioning metrics. While CLIP-RN50 performs worse than Faster R-CNN features when training a single language model without CaMEL and mesh-like connectivity, ViT-based and larger CNN-based models achieve better performance. The best performance is reached by the CLIP-RN50×16 variant, which employs an EfficientNet-style architecture scaling. Overall, this brings an improvement of 7.1 CIDEr points with respect to traditional detection-based features (from 113.9 to 121.0) when training a single language model without the CaMEL technique and without mesh-like connectivity.
Role of CaMEL. We then assess the impact of the proposed training technique during the XE pre-training stage, by also testing with different weights for the knowledge distillation loss, which we indicate with λ KD . Results are reported in Table I. As it can be seen, CaMEL improves the performance when used with all the previously mentioned features, by a considerable margin. This confirms the appropriateness of using a mean teacher learning paradigm in image captioning. When using the RN50×16 encoder, for instance, applying CaMEL with a distillation weight of 0.1 brings an improvement of 4.7 CIDEr points (121.0 vs 125.7). Finally, the CaMEL training strategy improves the performance also when using a mesh-like connectivity, from 122.6 to 125.0 CIDEr points when using λ KD equal to 0.01. Overall, during XE pre-training CLIP features bring a 6.2% relative improvement with respect to traditional detectionbased features, and CaMEL introduces, in the best feature configuration, a relative advancement of 3.9% with respect to a typical single model training.
Evaluation of different SCST strategies. Turning to the evaluation of the SCST fine-tuning stage, in Table II we compare the performance of different knowledge distillation strategies. In particular, we experiment the CaMEL best version, in which we pair the two logits sequences with the highest probability from both models to compute the KD loss, and the CaMEL all version where we use all the sequences generated by the beam search algorithm, pairing them in sorted order of log probability. Further, we explore the use of CLIP text embeddings in the MSE loss. In CaMEL best with CLIP text embeddings, we select the most probable caption in each of the two beams, and then apply the MSE loss between the CLS tokens given by the CLIP text encoder.
Moreover, we investigate the usage of the Hungarian matching algorithm [61] to couple captions in the online and target beams, using their CLIP embedding similarity as distance function. We then compute the MSE loss on the original logits between all pairs -in CaMEL all version -or only the most probable caption from the target model and its most similar one from the online model -in CaMEL best version. The best version, based on all metrics except BLEU scores, is CaMEL best without additional algorithms, while for the BLEU metrics the best one is CaMEL all always without the use of additional algorithms to pair captions.
Performance on COCO. As it can be observed from Table III,  our proposal reaches 138.9 CIDEr points, beating all the compared approaches. Adding the mesh-like connectivity to the decoder further improves the results to 140.6 CIDEr points. This represents an increase of 5.0 CIDEr points with respect to the current state of the art when training on the COCO dataset exclusively [44]. Further, in Fig. 1 we compare the aforementioned approaches in terms of both CIDEr and number of parameters. As it can be noticed, not only CaMEL reports state-of-the-art results in terms of caption quality, but it also features a significant reduction in terms of number of trainable weights. As most of previous literature has increased caption quality by increasing the model capacity, our approach represents an outlier in this trend, and demonstrates that stateof-the-art CIDEr levels can be obtained even with a very lightweight model.
Finally, in Table III and Fig. 1 we also report the performance obtained by a recent approach which employs large-scale pre-training on external data, i.e. VinVL [20]. While this approach is not directly comparable with CaMEL considering that it employs more data, we notice that our proposal is on pair with the Base version of VinVL, and only 0.3 CIDEr points below its Large version. In terms of model size and number of parameters, VinVL is also extremely more demanding than our proposal (cfr. Fig. 1). This further confirms that the usage of proper visual features and of mean teacher learning strategy can achieve a good caption quality with a reduced model size.
Online evaluation. Finally, we also report the performance of our method on the online COCO test server 1 . In this case, we also employ an ensemble of four models trained with the mesh-like connectivity. The evaluation is done on the COCO test split, for which ground-truth annotations are not publicly available. Results are reported in Table IV, in comparison with the top-performing approaches of the leaderboard. As it can be seen, our method surpasses the current state of the art on all metrics, achieving an advancement of 4.2 CIDEr points with respect to the best performer.

V. CONCLUSION
In this work, we investigated the use of the mean teacher learning paradigm applied to the image captioning task. We presented CaMEL, Captioner with Mean tEacher Learning, a novel Transformer-based network that is trained with the interaction of two different language models that learn from each other through knowledge distillation and model averaging. Experimentally, we validated our method with different knowledge distillation strategies and visual feature extractors, surpassing the current state of the art on the COCO dataset without using external data and pre-training strategies. Furthermore, CaMEL achieves similar performance to other recent proposals that make use of large-scale pre-training, while being much smaller in terms of number of parameters.