{'fraction_non_alphanumeric': 0.030298017809389906, 'fraction_numerical': 0.007912221211369337, 'mean_word_length': 4.903010577705452, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
ZoDIAC: Zoneout Dropout Injection Attention Calculation

Recently the use of self-attention has yielded to state-of-the-art results in vision-language tasks such as image captioning as well as natural language understanding and generation (NLU and NLG) tasks and computer vision tasks such as image classification. This is since self-attention maps the internal interactions among the elements of input source and target sequences. Although self-attention successfully calculates the attention values and maps the relationships among the elements of input source and target sequence, yet there is no mechanism to control the intensity of attention. In real world, when communicating with each other face to face or vocally, we tend to express different visual and linguistic context with various amounts of intensity. Some words might carry (be spoken with) more stress and weight indicating the importance of that word in the context of the whole sentence. Based on this intuition, we propose Zoneout Dropout Injection Attention Calculation (ZoDIAC) in which the intensities of attention values in the elements of the input sequence are calculated with respect to the context of the elements of input sequence. The results of our experiments reveal that employing ZoDIAC leads to better performance in comparison with the self-attention module in the Transformer model. The ultimate goal is to find out if we could modify self-attention module in the Transformer model with a method that is potentially extensible to other models that leverage on self-attention at their core. Our findings suggest that this particular goal deserves further attention and investigation by the research community. The code for ZoDIAC is available on www.github.com/zanyarz/zodiac .


Introduction
From the early days of the development of the first generation of neural networks [1,2] developed for perception dating back to at least half a century ago to the current decade, the era of modern generations of these models, the subject of visual attention [3,4] has been a pinpoint of interest for researchers. However, for many years it remained a mystery how to tap into the power of the attention mechanism for various modalities of information.
This was until the attention mechanism was utilized for neural machine translation [5]. With the advent of Transformer model [6], the true power of the attention mechanism was realized by everyone. Without the need for recurrence or convolutions and fully relying on the attention mechanism, the Transformer model was originally designed for mapping internal relationships among the elements of paired sequences of word embeddings for neural machine translation. Transformer employs a self-attention module (depicted in Figure  1(a)) inside a multi-head attention module, creating a wide parallelized model that can be trained faster than models with recurrences [5,7,8] or convolutions [9,10] for neural machine translation.
Looking at the whole story, the first notable moment in the evolution path of neural networks since over a half century ago until today was around a decade ago when AlexNet [11] the first deep convolutional neural network practically capable of solving the image classification problem end-to-end at a large scale was introduced. This sparked curiosity among other researchers to investigate the usefulness of deep neural network models for neural machine translation, an effort that led to the creation of encoder-decoder architectures [8]. Very soon, the encoder-decoder architecture became the defacto architecture in neural networks employed for neural machine translation. After investigating the features of encoder-decoder architecture, the idea of using attention mechanism for neural machine translation in the context of encoderdecoder architecture was born, which led to the creation of Transformer model.
The use of self-attention has yielded state-of-the-art results in natural language understanding (NLU) [12] and natural language generation (NLG) [13] tasks as well as image classification [14][15][16][17][18] and other computer vision tasks [19,20]. In vision-language, the utilization of self-attention has led to state-ofthe-art results in vision-language understanding and generation tasks [21][22][23]. Specifically, recent advancements in the utilization of self-attention and multihead attention for image captioning (which we discuss briefly in Section 2) has yielded state-of-the-art results under all image captioning metrics.
It has become evident that the self-attention mechanism is a useful and interesting tool that deserves more attention and investigation for further  Fig. 1 (a) Conventional self-attention used in Transformer [6]. (b) Our proposed Zoneout Dropout Injection Attention Calculation module. The GELU and dropout augmented selfattention in ZoDIAC calculate the attention values with refined intensities. Using GELU, the intensity refinement is performed element-wise over the whole input sequence. Then using a particular dropout rate, we lower the intensity of attention values once again, this time to eliminate the least important context. For intensity calculation, we create an attention map from the secondary and same transformation of queries and values respectively. The utilization of GELU creates an intensity refinement effect over the new attention map. After the intensity calculation is performed, the scalar intensity value is injected (multiplied) element-wise to all attention values with refined intensities using dropout and GELU. improvement. Self-attention successfully models the internal interactions and relationships among the elements of source and target sequence via modeling these relationships inside the source and target sequences first, and then modeling the relationships between source and target sequence.
As we mentioned earlier, the use of attention was first explored in the context of visual attention inspired by how it works in biological organizations. Like how deep learning models are inspired by the way the human brain works in many ways, we focus on the fact that when communicating with each other face to face or vocally, we tend to express different words in the sentence with various levels of intensity, indicating the importance of those words in the context of the whole sentence. These words can be referred to as memory pinpoints with high intensities, when trying to remember a conversation we had with another person for thought or sentence generation, we tend to review the most important words in the conversation we had to obtain the most useful information for generating a rich description of the conversion. Similarly, when trying to remember and describe a visual event (or object) we saw earlier to someone else, we tend to describe the most important features (with the highest intensities) of that event to give a rich and informative description of the event. Based on the whole intuition that words in sentences should be weighted to reflect their intensities while being memorized (learned) and later used for ZoDIAC sentence generation describing some memorized context, we introduce Zoneout Dropout Injection Attention Calculation (ZoDIAC), which is displayed in Figure 1(b) and that leverages on intensity calculation over the words in the sentence based on the context of each word in the sentence. Our contributions to this work can be summarized as the following: • ZoDIAC outperforms the conventional self-attention employed in the Transformer model under all image captioning metrics. • ZoDIAC leverages on intensity calculation over on attention map generated from values and secondary projection of queries and outperforms the selfattention. • Our proposed ZoDIAC method could potentially be considered an extension to other models that use self-attention at their core, in our experiments namely Transformer and AoA model.

Related Work
A wide variety of methods have been used for automatic image captioning.
Template-based methods [24][25][26] required prior knowledge of visual features and relied on visual feature engineering. The first deep learning models used for image captioning, such as the Show and Tell introduced by Vinyals et al. [27], and the first attentive deep model, Show, Attend and Tell, introduced by Xu et al. [28], performed better than template-based methods by employing CNNs. End-to-end image captioning systems rely on CNN backbones for feature extraction. The CNN backbones that are commonly used for visionlanguage tasks are ResNet101 [29] and ResNext152 [30]. Early deep learning methods for image captioning used conventional convolutional architectures that operate upon the entire image to extract the visual features in the encoder part of the model [28,[31][32][33]. Anderson et al. [34] introduced the Bottomup and Top-down attention model for image captioning and visual question answering. Top-down attention refers to the application of attention mechanisms in the decoder part of the model, whereas Bottom-up attention refers to the application of visual attention in the encoder part of the model. In Bottom-up attention, the input image is passed through an object detector, usually Faster-RCNN [35]. From the RoI Align layer in the object detector, we get the region proposals or the coordinates of the objects in the input image. Visual features are extracted from these regions using a CNN backbone, and these features concatenated with the word embedding at each time step are sent to an attention LSTM and then to an attention network that performs visual attention. The attention values and the hypothesis vector of the attention LSTM are then sent to a language LSTM for generating token embeddings at word level [34].
In the past, researchers have employed the Transformer model and variations of it with bottom-up attention features for image captioning. Pan et al. [36] introduced a bilinear pooling mechanism in the conventional self-attention block, resulting in the x-linear attention block [36], which exploits the spatial and channel-wise bilinear attention values to reveal the second and infinity order interactions between the multi(or single)-modal input features [36]. Guo et al. [37] created the normalized and the geometry-aware self-attention block that exploits geometrical information presented in visual features. Using a 4dimensional vector that contains the relative positions and coordinates of the bounding boxes for the objects in the image, the relative geometry features among the objects are calculated.
Cornia et al. [38] employed linear transformation and sigmoidal gating over the memory states of encoders in the encoder stack and decoders in the decoder stack of the Transformer, resulting in a Meshed-Memory Transformer network that performs significantly better than the conventional Transformer. Herade et al. [39] used object labels as attributes to be concatenated with visual features extracted by the CNN backbone as input information for the encoder in the Transformer. Yu et al. [40] introduced a Multi-modal Transformer that used multiple views of the object proposal sets with different orders to provide the encoder in the Transformer with different sets of Bottom-up features. Liu et al. [41] similarly employed visual attention values alongside the context attention values and attributes attention values as cross-modal information. Li et al. [42] introduced the Entangled Transformer by applying weighting on a meshed network of linear transformations of the queries and values in the visual encoder and the attributes encoder.
Recently, we have witnessed the effectiveness of pre-training transformers for image captioning. Specifically, Unified Vision-Language Pretraining [22] opened the door to pre-training a transformer on vision-language tasks. More recently, OFA [43] leveraged vision-language pre-training to obtain state-ofthe-art results on image captioning and other vision and language tasks. In particular, they pre-train a transformer on object detection (bottom-up feature extraction) and image reconstruction as vision tasks and text infilling as language task. Another recently published method, namely LEMON [44], leverages expanding the parameters of a transformer model used for vision-language pre-training.
Although the self-attention and multi-head attention have been modified in various ways and tailored for image captioning, nevertheless none of these modifications address the issue of refining the final output sequence considering the intensities of elements in the source and target sequence.

Methodology
To understand how ZoDIAC (Firgure 1(b)) resembles leveraging on information gathered from the past for thought or sentence generation in our brain, we need to have a detailed understanding of how self-attention (Figure 1(a)) works.
Attention (in self-attention) is defined as mapping queries and a set of key-value pairs to an output. A multi-head attention module that includes multiple (usually 8 or 12) self-attention heads is used inside the encoder and ZoDIAC decoder parts of the Transformer model. The multi-head attention is used inside the encoder once and the decoder twice. In both encoder and decoder, the self-attention is used over the input sequences to capture the internal relationships among the element of the sequences (source sequence for the encoder and target sequence for the decoder). Specifically, in the decoder, when multi-head attention is used inside the decoder for the second time, it maps the relationships among the elements of source (as key and value) and target sequence (as query). The multi-head attention includes linear layers that transform the input sequences into query, key and value. The query (Q), key (K) and value (V ) are sent to self-attention after transformation.
There is a 2-step process for calculating final attention values in selfattention. The first step entails creating an attention map via application of the softmax function over the result of matrix multiplication (MatMul) operation between query and key, and scaling (and masking while training) over the results before application of the softmax function. In the second step, the query-key attention map and the value are used inside a MatMul operation for final attention values generation. The attention values are later used inside a feed-forward log-softmax layer for word generation at the token level. In biological organizations, visual attention is applied to the visual input to create visual features with refined intensities that help the brain understand the context of the scene. Self-attention resembles how attention works in biological organizations via creating an attention map from the input and then applying the attention map to the input.
We redefine attention as mapping queries and a set of past-current value pairs to generate the scalar intensity value, and then mapping queries and a set of key-value pairs to generate the attention values. After injection of scalar intensity value into the attention values, we achieve attention values with refined intensities, which are going to be used in a feed-forward log-softmax layer for word generation at the token level. Via generating an intensity map and an intensity value from it and applying the intensity value to the input, ZoDIAC (Figure 1(b)) resembles how current information should be processed based on the intensity of information gathered (learned) from the past in biological organizations.

GELU For Intensity Refinement
Gaussian Error Linear Units (GELU) [45] non-linear function combines the properties of Zoneout [46], Dropout [47] and ReLu activation, all in one place. ReLu and Dropout are like each other because they both block some incoming signals from the previous layer. Their difference is that ReLu drops the signals that have values below zero. On the other hand, Dropout drops (sets to zero) some portions of the signal randomly. Zoneout [46] is the complete opposite of dropout because some portion of the signals are kept identical (multiplied by one) and are selected to be identical to the previous layer.
Combining all these features, GELU activation function is defined (in a simplified form) as the following: Looking at Eq.(1), we realize that the GELU function first cuts the signal in half, then it searches for the right amount of signal to be added to the previously cut signal. Euler's number (e) is used in the form of e −t 2 to create a curve for Area Under Curve (AUC) calculation, which correlates with the correct amount of signal that should be added to the previously cut signal at each time step t. Therefore, Eq.(1) shows how the intensity of input signals (element-wise) is being refined by the GELU activation in Figure 2. GELU can also be approximated with xσ(1.702x). The sigmoid function (σ) generates a value between 0 and 1. Therefore, multiplying the result of the sigmoid function by the incoming signal (x) always guarantees a refinement (lowering) effect over the input signal.

Refined Current Attention (RCA)
In order to create the past-current value pairs (current attention value and past intensity value before injection in Figure 2(a)) we leverage on parallel attention and intensity map calculation. Specifically, as shown in Figure 2 (b), query is transformed into past (Q 2 ) and current (Q 1 ) queries. Via the application of MatMul operation over the current query (Q 1 ) and key we calculate the current attention map. The utilization of GELU over the current query and key and the result of MatMul creates an intensity refinement effect over the attention map before the application of Softmax over the attention map. The current attention map and the value are used inside MatMul operation for current attention value generation. The attention values are then sent to a dropout layer with a particular dropout rate that is different than the dropout rate used in other parts of the model. The Transformer model employs a dropout rate of 0.1 for Residual Dropout [6]. We refer to the dropout rate used in Residual Dropout as the system dropout rate and the dropout rate used in ZoDIAC for current attention value refinement as ZoDIAC dropout rate. Using a different dropout rate inside our model forces it to randomly learn and ZoDIAC eliminate the least important parts of the sequence. This is shown on the left side of Figure 2(a). With masking operation identical to how it is performed in self-attention [6] (Figure 1(a)), dimension of key denoted as d k , the dropout rate for RCA denoted as δ Z and GELU denoted as G and dropout as D, and MatMul operation not shown for conciseness, the refined current attention (RCA) depicted in left side of Figure 2(a) is put together as the following in Eq.(2-3):

Past Intensity Value (PIV)
Via performing MatMul operation over past query (Q 2 ) and value, we create an intensity map. The computation process of this intensity map is identical to how the refined current attention map is calculated in Eq. (2), expect that instead of using the current query and key for intensity map generation, here we use past query and value without using softmax activation for this purpose.
After the past intensity map is computed, the boolean mask (broadcastable to intensity map with same dimensions) is multiplied element-wise to the past intensity map. After the injection of the mask into the past intensity map, via performing an average calculation operation over all the remaining attention values in the past intensity map, we achieve a Regional Attention Pooling (RAP) scalar value. Via applying a scalar (element-wise) activation function of choice over the RAP scalar value we achieve the past intensity value. By injecting a zoneout factor into the past intensity value we achieve a refined past intensity value, this value is then injected (element-wise) into the refined current attention values as we explain in the following section. Considering the past intensity value calculation function as PIV, regional attention pooling function as RAP, mask as the same boolean mask used for masking operation in RCA, the zoneout factor as ζ, the scalar activation gate as φ, dimension of value as d v , total number of values in past intensity map (or mask) as N , element-wise addition and multiplication as ⊕ and , and GELU denoted as G, the past intensity value calculation in the right side of Figure 2(a)) is shown in the following equations Eq.(4-8): P IV (Q, V ) = ζ ⊕ φ RAP (P IV map , mask) .
When PIV is used inside the decoder, Eq. (7) is infused into Eq. (8). When used inside the encoder, where there is no masking, Eq.(6) is injected into Eq. (8). The scalar activation gate could be sigmoid or tanh (φ = σ or tanh).
Note the difference between masking in RCA and the injection of the mask here in RAP, which eliminates worthless connections (setting to 0), and leaves the remaining connections for use in RAP in PIV. When RCA and self-attention [6] are used inside the decoder, to prevent leftward tokens to attend over the future (rightward) tokens in the target sequence, masking is performed. This masking includes masking out all values in the attention map (setting to −∞), which are considered as least important connections, before the application of softmax. This is the reason behind naming RAP in PIV.

ZoDIAC
With refined current attention calculation function denoted as RCA, past intensity value calculation function as PIV, input sequences (with positional encoding information added [6]) transformed via linear layers into current query denoted as Q 1 , past query as Q 2 , key as K and value as V , and element-wise multiplication as , we define ZoDIAC (Figure 2(a)) as the following: As briefly mentioned earlier, ZoDIAC is used inside a modified version of multi-head attention that can support ZoDIAC with necessary sources of information. This is further explained in detail in the following section.

ZoDIAC Multi-Head Attention (ZMHA)
In the Transformer model, the input (source and target) sequence is augmented with positional encoding information [6]. After the positional information is added to the sequence, the sequence is fed to linear layers that transform the sequence into query, key and value. The query, key and value are used inside self-attention modules as each head inside the multi-head attention. We redefined attention earlier and constructed the ZoDIAC module, therefore, we need to modify (redefine) multi-head attention in a way that enables this module to support the employment of ZoDIAC modules as each attention heads.
Our first modification involves adding a linear layer inside our modified multi-head attention that transforms the query into past query. The second modification involves adding the GELU activation for intensity refinement. This is since we need to refine the intensities of values in key, value and past and current queries before the transformation.
Considering concatenation defined as brackets, total number of heads denoted as H, each ZoDIAC head az Z, GELU denoted as G, the first transformation of query as current query denoted as Q 1 , the second transformation of query as past query denoted as Q 2 , last linear layer that transforms the ZoDIAC attention values denoted as W o (as shown in Eq. (12)), X as a variable that could be replaced with past query, current query, key and value, and W X (as shown in Eq.(10)) as a stereotype for other linear layers that transform the key, value and past and current queries, the ZoDIAC-Multi-Head Attention (ZMHA) shown in Figure 2(b) can be put together as in the following equations Eq. (10)(11)(12):

Implementation Details
Hyper Parameters: We found a system dropout rate of 0.1, a ZoDIAC dropout rate (δ Z ) of 0.2, and a zoneout factor (ζ) of 1 as the best configuration for ZoDIAC. We use sigmoid (σ) and tanh as scalar activation gate (φ).

MS-COCO:
This is the most commonly used dataset among researchers for image captioning and object detection. Due to the huge size of this dataset, standard splits have been defined for training, testing and validation purposes. In our experiments, we use the Karpathy's split [48] for MS-COCO [49] for offline evaluation. We also report our results using MS-COCO's online evaluation server.
Word Embeddings: In our experiments, we use learned embeddings from scratch instead of employing Word2Vec [55], Glove [56] or BERT [12] embeddings. Our model has to learn representations from scratch, instead of using an external source of knowledge, resulting in a fair and robust comparison with other methods.
Loss, Optimizer & Learning Rate: We use the ADAM optimizer [57] with a learning rate of 5e-4 and we decay the learning rate starting at the first epoch every three epochs by a factor of 0.8 and maximum of 30 epochs is used for training all model variants. We also train our models using REIN-FORCE algorithm (Self-Critical Sequence Training loss) [58]. In particular, we use a better variant of this algorithm that benefits from faster convergence and slightly better performance [59]. For Self-Critical (SC) training (and finetuning before training using SC loss starts), we employ constant learning of 1e-5. When tanh is used as a scalar activation gate (φ = tanh), self-critical training starts without fine-tuning. When sigmoid is used as scalar activation gate (φ = σ), For self-critical training, we train all models for 30 epochs.
ZoDIAC Encoder-Decoder Configs: In the majority of experiments, we initially utilize a stack of 6 encoders and 6 decoders, and we use 8 ZoDIAC heads in each encoder and decoder. Similarly, other model parameters such as the dimensions of query, key and value and feed-forward layers are identical to those used in the Transformer-Base model. Since ZMHA utilizes an extra linear layer for the transformation of query into past query, this naturally leads to a parameter increase. Therefore, in our ablation studies, we use a stack of 5 encoders and 5 decoders alongside investigating the effectiveness of other parts of ZoDIAC.
Feature Extraction: For object detections and region proposals, we use Faster-RCNN [35] pre-trained on Visual Genome [60] and for visual feature extraction from the detected regions we use ResNet101 pre-trained on ImageNet [61] as CNN backbones.
Batch & Beam Sizes: In the majority of our experiments we use a batch size of 20 and for evaluation, we use beam search with a beam size of 3. For evaluation of AoA and ZoDIAC+AoA models, we used a beam size of 2.

Discussion and Results
We train our proposed ZoDIAC method employed in Transformer and the conventional Multi-Head attention with Bottom-up attention mechanism added to the encoders and decoders on MS-COCO [49] from scratch and without fine-tuning.
Employing metrics that focus on the textual features of the generated captions and ground truth captions, such as BLEU [50], ROUGE [52] or METEOR [51], ensures the quality of generated captions from a neural machine translation vantage. On the other hand, employing metrics that consider the quality of vision-language relationships such as CIDER [53] and SPICE [54] ensures the quality of generated captions from both neural machine translation and computer vision vantages.
The CIDER metric [53], reflects the quality of relationships between textual and visual properties of input image and input ground truth captions. This is achieved by computing Term Frequency and Inverse Document Frequency (TF-IDF) and considering the level of relevancy between a set of captions and the input image.
The SPICE metric [54] considers spatial relationships among objects in the image using scene graphs and then evaluates the quality of captions based on the existence of the discovered spatial relationships in the captions. This metric reflects the quality of the vision-language properties of the captions, as well as CIDER [53].
The results of our experiments on the MS-COCO trained with crossentropy loss are reported in Table 1. These results are reported from best of 5 runs for each model. Table 2 shows the results of ensemble evaluation using 5 runs. We show the results for experiments using self-critical sequence training (SCST) in Table 3. We consider the CIDER score as the most important metric for all our experiments, although other metrics are indicators of performance as well, however CIDER is the only metric that focuses on the quality of generated captions given an image and a set of ground truth captions. This is also the main reason why we optimize the model on CIDER metric at the self-critical training stage. ZoDIAC Looking at the results from Table 1, we observe that the results of ZoDIAC are superior to the self-attention module used inside the conventional Transformer model. For the sake of comparison, we also report the results of the Transformer model on MS-COCO reported by Sharma et al. [62]. It is worthy of being mentioned that when tanh is used as a scalar activation gate instead of the sigmoid (σ), the results are further improved. We believe this is since the tanh activation gate creates a value between negative one and one, whereas the sigmoid function creates a value between 0 and 1, By adding the resulting values to zoneout factor (set as 1), ZoDIAC is creating different refinement effects. When tanh is used, the past intensity value can be less than 1 or greater than 1. When sigmoid is used, considering the zoneout factor, the past intensity value is always between 1 and 2. In other words, when tanh is used, the model can decrease or increase the intensity of attention values, whereas when sigmoid is used the model only has to learn how much it should increase the intensity of attention values.
Furthermore, we observe that when ZoDIAC is used as an extension inside another model that leverages self-attention, namely the Attention-on-Attention (AOA) model [63], the results are also improved, but the gains are marginal compared to when ZoDIAC is used inside the Transformer model. Our investigations reveal that this is because AoA model employs an LSTM alongside multi-head attention and their proposed refinement modules. In other words, there are fewer self-attention modules used inside AoA model, in comparison with the transformer model. Likewise, our experiments reveal that ZoDIAC could be a potentially good choice as an extension for models that highly leverage on a self-attention module with minimal changes inside the Transformer model and more changes in parameter size and number of layers. Also in Table 1 we observe that AoA achieves slightly lower results than the ones reported by Sharma et al. [62]. For the reasons discussed above, we did not perform further experiments with ZoDIAC the inside AoA model. We also found that when ZoDIAC is extended to AoA model, we have to use a ZoDIAC dropout rate of 0.3 (in order to match the properties of AoA) and also that we have to remove the GELU pre-activations before linear transformations are performed inside the multi-head attention module in order to achieve better results than AoA. By looking at the results from Table 2, we can conclude that when trained with cross-entropy loss, ZoDIAC is a superior model for image captioning in comparison with the conventional self-attention module inside the Transformer model. These results are reported from the ensemble evaluation of five models trained with different random parameters. It is also worthy to mention that we could not find results reported in published literature from ensemble evaluation of the Transformer model on Karpathy's test set (MS-COCO dataset).
For SCST loss training the results are reported in Table 3. We can conclude that when trained with SCST loss, ZoDIAC achieves slightly better results in comparison with the conventional self-attention module inside the Transformer model. Because performance gains for ZoDIAC+AoA model were marginal at the cross-entropy loss stage, we decided not to run further experiments with ZoDIAC+AoA at the self-critical training stage. We conclude that ZoDIAC can potentially be a good choice for achieving better results for models that leverage on self-attention when trained with SCST loss, however these improvements are marginal compared to the improvements we achieve when the ZoDIAC model is trained with cross-entropy loss. Therefore, we encourage the readers to investigate the effectiveness of ZoDIAC for tasks that do not require SCST loss training, such as image classification with transformers [17]. ZoDIAC At the current time, the state-of-the-art results in image captioning are generated by models such as OFA [43], LEMON [44] and SimVLM [64] that utilize huge parameters and vision-language pre-training methods. For the OFA model, although the official code has been released by authors, the full pre-training data has not been released yet. Also for LEMON and SimVLM, the official codes have not been released yet, therefore we could not conclude that ZoDIAC is a good choice for these models, but we encourage the readers to investigate the effectiveness of ZoDIAC on these models or similar models that employ the self-attention module.

Ablations
To reveal the effectiveness of different parts of ZoDIAC we perform careful ablation studies. First, we want to study the effect of the dropout rate on the performance of the model. Second, we want to reveal the effect of the zoneout factor on the performance of the model. We also investigate the effect of the employment of GELU in our model. The effect of choosing tanh as scalar gate over sigmoid was shown in tables 1, 2 and 3. By looking at Table 4, we can see that when there is no scalar activation gate used inside ZoDIAC, a system dropout rate of 0.2 and a ZoDIAC dropout rate of 0.3 is the best combination. However, when the scalar activation gate is used, the system dropout rate of 0.1 and the ZoDIAC dropout rate of 0.2 is the best combination. Therefore in our final experiments, we chose the combination of 0.1 and 0.2 when sigmoid or tanh are sued as scalar activation gates.

Conclusions
In this work, we proposed Zoneout Dropout Injection Attention Calculation (ZoDIAC), a novel attention module that is a successor of the self-attention module in the Transformer model. Self-attention is refined inside ZoDIAC and affected by the intensity injection of attention map generated from values and second linear projection of queries that can be considered as past queries. Inspired by how the intensities of past events cast an effect on decisions or actions in the current time, ZoDIAC improves the performance of the Transformer model, which relies on the self-attention module that exploits the intra-relationships in the input sequence. ZoDIAC improves this feature via refining the intra-relationships in the input sequence and injecting the intensities of the attention map generated from the input sequence and another projection of it that can be considered as the past form of the input sequence. We believe that ZoDIAC has opened the door to a new way of thinking about the self-attention module in the Transformer model and how it could potentially improve other models that utilize self-attention and we hope that other members of the research community will benefit from this work.

Declarations
• Funding: This work has not received any funding. N/A • Conflict of interest/Competing interests: The authors declare that they have no conflict of interest or competing interest in this work. N/A. • Consent to participate: The authors have consented to participate in this work. • Consent for publication: Authors have consented to be published in the journal. • Availability of data and materials: The data-set used in all experiments and results reported in this work is publicly available for download. COCO data-set is available at "cocodataset.org". • Code availability: The code for our experiments is publicly available github. com/zanyarz/zodiac. • Authors' contributions: Zanyar Zohourianshahzadi designed the ZoDIAC method and performed all the experiments and wrote the original text of the paper. Jugal K. Kalita provided suggestions regarding the text and structure of the paper and helped with article preparation phase.