{'fraction_non_alphanumeric': 0.03396980461811723, 'fraction_numerical': 0.0174955595026643, 'mean_word_length': 4.786485097636176, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Aggregating Global Features into Local Vision Transformer

Local Transformer-based classification models have recently achieved promising results with relatively low computational costs. However, the effect of aggregating spatial global information of local Transformer-based architecture is not clear. This work investigates the outcome of applying a global attention-based module named multi-resolution overlapped attention (MOA) in the local window-based transformer after each stage. The proposed MOA employs slightly larger and overlapped patches in the key to enable neighborhood pixel information transmission, which leads to significant performance gain. In addition, we thoroughly investigate the effect of the dimension of essential architecture components through extensive experiments and discover an optimum architecture design. Extensive experimental results CIFAR-10, CIFAR-100, and ImageNet-1K datasets demonstrate that the proposed approach outperforms previous vision Transformers with a comparatively fewer number of parameters.


I. INTRODUCTION
Transformer-based architecture has achieved tremendous success in the field of natural language processing (NLP) [37] [7]. Inspired by the great success of transformer in the language domain, vision transformer [8] has been proposed and achieved superior performance on the ImageNet dataset. The vision transformer splits the image into patches and feeds into the transformer, the same way as words token in NLP, and passes through several multi-head self-attention layers of the transformer to establish the long-range dependencies.
Unlike the word token, a high-resolution image contains more pixels compared to words in the passage. This leads to an increase in the computation cost as self-attention in the transformer has quadratic complexity. To alleviate this problem, various local attention-based transformers [24] [36] [48] have been proposed with a linear computation complexity. However, all the proposed approaches could not establish longrange dependencies and some of them are very complicated.
To overcome these issues in the local transformers, we develop a very simple module, named multi-resolution overlapped attention(MOA), to generate global features. The proposed module only consists of multiplication and addition operations and is embedded after each stage in the transformer before the downsampling operation. As the module is added only after each stage instead of each transformer layer, it does not add much computation cost and the number of parameters. Our experiments show that aggregating the resultant features of this module to the local transformer establish the long-range dependencies and hence significantly increases the accuracy in contrast to the total number of parameters as shown in Figure 1 Our proposed MOA module takes the output generated by the group of local window-based attention as an input. It first converts it to a 2D feature map, and projects it to a new low-dimension feature map. Similar to ViT [8], the projected feature map is divided into a fixed number of patches except for a few modifications. In contrast to ViT [8], the patch sizes of query and key-value are different. The resolution of the patches in the query is the same as the window size used in the local transformer layer. In contrast, the resolution of patches in key-value is slightly larger than the query patch and overlapped. The hidden dimension of the MOA global attention module is kept the same as the previous transformer layer. Therefore, the resultant features are directly aggregated to the output of the previous transformer layer.
Extensive experiments show that keeping the key-value patches slightly larger with overlap to each other leads to significant performance gain due to small information exchange between two neighborhood windows. In short, our method exploits the neighborhood information along with global information exchange between all non-local windows by embedding the proposed MOA mechanism in the local Fig. 2. The architecture of the MOA-T is composed of three stages. Each stage consists of a patch embedding/merging layer and local transformer block along with a global multi-resolution overlapped attention module after each stage except the last stage. In the beginning patch, a partition layer is included to divide the image into a fixed number of patches. transformer.
The contributions of the proposed approach are summarized as below: 1) We propose a multi-resolution overlapped attention (MOA) module that can be plugged in after each stage in the local Transformer to promote information communication along with nearby windows and all non-local windows. 2) We thoroughly study the impact of global information in local transformer using the proposed MOA module. 3) We investigate the effect of the dimension of essential architecture components through extensive experiments and discover the optimum architecture for image classification. 4) We train the proposed model from scratch on CIFAR-10/CIFAR-100 [19] and ImageNet-1K [6] datasets and achieve state-of-the-art accuracy using a local transformer.

II. RELATED WORK A. Convolutional Neural Networks
After the revolutionary invention of AlexNet [20], convolutional neural network (CNN) has become a standard network for all computer vision tasks, such as image classification [25] [29], object detection [22], tracking [47], segmentation [13] [28], counting [31], and image generation [41]. Various versions of CNNs have been proposed to improve the performance by making it deeper and/or broader, such as VGG network [33], ResNet [12], Wide-ResNet [45], DenseNet [17], etc. There are also several works proposed to make it more efficient by modifying the individual convolutional layer, such as dilated convolution [42], depth-wise separable convolution [4], group convolution [20], etc. In our work, we employ the convolutional layer along with the transformer layer to reduce the overall dimension of the feature map. Our experiments show that the combination of convolutions and multi-head attention increases the performance.  [26] have been proposed that utilize either channel-based or position based self-attention layers to augment the convolution network. Non-local network [39] and PSANet [49] model the spatial relationship between all the pixels in the feature map and are embedded the attention module after each block in CNN, whereas SENet [16] establishes a channel relationship in the convolution network by squeezing the features using global average pooling. CBAM [40], BAM [27] and dual attention network [9] employ both channel and position based attention mechanisms separately, then combine the resultant features from both attention modules using either element-wise addition or concatenation and uses the resultant features into convolution output after each stage, whereas GCNet [2] combines SENet [16] and nonlocal network [39] together and propose the hybrid attention mechanism that aggregates the information of both channel and spatial relationships in the same attention module.

C. Vision Transformers
Similar to AlexNet, vision Transformer (ViT) [8] has changed the perspective of researchers towards solving computer vision problems. Since then, many vision transformerbased networks have been proposed to improve accuracy or efficiency. The ViT needs to be pre-trained on large datasets such as JFT300M [34] to achieve high performance. DeiT [35] solves this problem by student-teacher setup, substantial augmentation, and regularization techniques. To train the transformer on the mid-sized dataset like ImageNet-1K from scratch, the token-to-token vision transformer [43] recursively aggregate neighboring tokens (patches) into one token (patch) to reduce the number of tokens. A Cross-ViT [3] comes up with a dual branch approach with multi-scale patch size to produce robust image features and pyramid vision Transformer (PVT) [38] introduces a multi-scale-based spatial dimension design similar to FPN [23] in CNN and demonstrated good performance. Furthermore, PVT introduced a spatial reduction in key to reduce the computation cost in multi-head attention.
Various local attention-based transformers have been introduced to alleviate the quadratic complexity issues [36] [24] [48]. The HaloNet [36] introduces the idea of a slightly larger window of key than the query in a local attention mechanism and proves its effectiveness through various experiments. In our model, the key is also calculated using a slightly larger Fig. 3. Patch creation for query embedding is shown in the blue, and key/value is shown in the red for feature map size 9 × 9 and window size 3 × 3. Blue patches have the same size as the window and are non-overlapped to each other. In contrast, red patches are larger and slightly overlapped with each other. Appropriate padding is applied while creating the key-value patches.
patch, but in the context of global attention, the idea of a larger key is different from the HaloNet. A swin Transformer [24] proposes a non-overlapping window-based local self-attention mechanism to avoid quadratic complexity and achieve comparable performance and aggregated nested Transformer [48] come with the multi-scale approach with block-aggregation mechanism after each stage.
Some Transformer-based works have been proposed to utilize both local and global features [11] [5]. A Transformer in Transformer (TNT) [11] further divides the local patches (visual sentences) into smaller patches (visual words). The MHA on visual words embedding is calculated and aggregated to the sentence embedding to establish the global relationship. The twin Transformer [5] is quite the same as ours. However, global attention is applied after each local Transformer layer, increasing the computation cost significantly. In contrast, we apply it after each stage, and we have slightly larger and overlapped patches in key in multi-head attention. The proposed network efficiently utilizes global information in the local transformer and achieves higher accuracy than the abovementioned transformer-based models.

III. PROPOSED METHOD
We aim to provide global information exchange across all windows in the local transformer by increasing the minimal computation cost and a number of parameters. An overview of our proposed model is shown in Figure 2, which shows MOA module after each stage. All stages have a similar architecture design, including patch merging layer and local transformer block except the first stage. The first stage consists of patch partition, linear embedding layer, and local transformer block.
Our global MOA module is applied between each stage before the patch merging layer. Specifically, the model takes an RGB image as an input and splits it into fix number of patches. Here each patch is treated as a token. In our experiment on the ImageNet dataset, we set the patch size to 4 × 4, which leads to 4 × 4 × 3 = 48 feature dimensions for each patch. These row features are projected to a specific dimension C using the patch embedding layer in the first stage. The resultant features are then passed through consecutive stages consisting of patch merging layer, local transformer block, and MOA module in-between each stage. Unlike swin Transformer [24], our Transformer block employs the same self-attention mechanism as ViT [8] without any shifted window approach. Similar to swin Transformer, the number of tokens is reduced, and the output dimension is doubled in the patch merging layer after each stage. For example, the resolution after the first, second and third stage is , and H 8 × W 8 , respectively. The average pooling layer is inserted at the end of the last stage, followed by a linear layer to generate a classification score. The detailed explanation of each element of architecture are as follows:

A. Patch embedding layer
It is a basic linear embedding layer applied to the row features of patches to project it to a specific dimension C.

B. Patch merging layer
Patch merging layer reduces the number of tokens by concatenating the features of 2 × 2 neighboring patches and doubles the number of hidden dimensions by applying a linear layer on the concatenated 4C -dimensional features.

C. Local Transformer Block
The local transformer block consists of a local windowbased standard multi-head attention module, followed by a two-layer MLP with GELU non-linearity. A layer norm is used before each multi-head attention module and each MLP with residual connection after each module.

D. Multi-resolution Overlapped Attention Block
To utilize the advantage of global information in local transformer, we apply a global attention module named multiresolution overlapped attention (MOA) in-between each stage. The architecture of the MOA mechanism is the same as the standard multi-head attention except for a few modifications. Similar to standard MHA, it first divides the feature map into the fixed size of patches. However, unlike the standard MHA, patches for generating key and value embeddings are slightly larger and overlapped, while the patches for query embedding are non-overlapped as shown in Figure 3.
As shown in the Figure 3, the input to MOA block is of size W × H× hidden dim, Where W = W 2 , W 4 or W 8 , H = H 2 , H 4 or H 8 , and hidden dim = 96, 192, or 384. Calculating query, key, and value embeddings directly from the input is quite expensive in computation. For example, in context to the ImageNet dataset, the feature map size of the input to MOA block after the first stage is 56 × 56 × 96. Deriving query embedding directly from the input feature with a patch size 14 will lead to the resultant feature of dimension 14 × 14 × 96 = 18816. Therefore, we first reduce the hidden dimension with factor R by applying 1 × 1 convolution, which reduces the computation cost. The resultant feature dimension after applying the convolution is H × W × hiddendim R . This leads to feature size in one query patch is 14×14× hiddendim R , which is projected to the one-dimensional vector of size: 1 × 1 × hiddendim. The total number of the query is H 14 × W 14 . Similarly, the key and value vector are projected, but the patch size is slightly larger than the query as shown in Figure 3. In our model, we set the key-value patch size to 16. Therefore, the number of key-value will be according to the equation: ( H−16+(2×padding) stride + 1, W −16+(2×padding) stride + 1). Multi-head attention is applied to this query, key, and value embedding, followed by two-layer MLP with GELU non-linearity in between. Similar to the Transformer block, layer norm is applied along with residual connection after each MOA module. At last, on the resultant features, 1 × 1 convolution is applied, followed by broadcast addition of resultant features with the output of the previous transformer block, which contains the local information.

E. Relative Position Index
We use relative position bias B ∈ R M 2 ×N 2 , as used by [1] [15] [14] [30] , in the heads of both local and global attentions during similarity computation: where Q ∈ R M 2 ×d is a query matrix, K, V ∈ R N 2 ×d are the key and value matrices; d is the hidden dimension, M 2 is the total number of patches in the queue and N 2 is total the number of patches in the key.

F. Architecture Detail
By following the previous works [24] [5], we build three versions of the model: MOA-T, MOA-S, and MOA-B for the ImageNet dataset and only two versions of the model: MOA-T and MOA-B for the CIFAR -10/100 dataset as it is quite smaller. Table I shows the architecture configurations for the CIFAR and the ImageNet datasets. In the CIFAR based models, both MOA-T and MOA-B contain the same number of Transformer layers: 12, but have a different number of hidden dimensions. In context to the ImageNet based models, the total number of layers for MOA-T and MOA-S is 12 and 24 respectively, but the hidden dimension is kept the same, whereas MOA-S and MOA-B have the same number of Transformer layers: 24, with contrast hidden dimensions 96 and 124 respectively.

IV. EXPERIMENTAL EVALUATIONS
We verify our model through extensive experiments on CIFAR-10/CIFAR-100 and ImageNet-1K for image classifica-tion. We design three architecture versions: MOA-T, MOA-S, and MOA-B, for the classification tasks.
A. CIFAR-10/100 Results CIFAR-10 and CIFAR-100 datasets consist of 50,000 training and 10,000 test images of resolution 32 × 32 with the total number of classes 10 and 100, respectively. We train the network for 300 epochs using AdamW [18] optimizer with an initial learning rate of 0.009 and weight decay of 0.05. We utilize a cosine decay learning rate schedular along with 20 warm-up epochs. We implemented two models: MOA-T and MOA-B for the CIFAR dataset with total batch-size 128 and stochastic drop-rate 0.2 [21]. Table II shows the performance of our model on the CIFAR-10 and CIFAR-100 datasets. We presented only two models with the same number of layers but with different hidden dimensions for this dataset. As shown in the table, it can be seen that both models outperform all the previous Transformer-based models by a significant amount. It improves the performance by 0.59% and 0.98% on CIFAR-10 and 0.56% and 0.23% on CIFAR-100 for the Tiny and Base models, respectively, compared to Swin Transformer. For the Base model, our model achieves state-of-the-art accuracy on local vision Transformer with a comparatively fewer number of parameters and GFLOPs. The accuracy of other models is reported by training the models from scratch with the same training setting reported in the papers [24] [35] [38].

B. ImageNet Results
ImageNet-1K dataset consists of around 1.28M training images and 50K validation images with 1000 classes. We resize all the images to the resolution 224 × 224 during training. We follow the same training technique, like Swin and Twin, and train the network for 300 epochs using AdamW [18] optimizer with a cosine learning rate schedular and 20 warmup epochs. We keep the batch-size 128 for MOA-T and 64 for MOA-S and MOA-B models per GPU. We employ a total of four GPUs together during training leads to a total batch-size 512 for MOA-T and 256 for MOA-S and MOA-B models. We utilize the same augmentation technique used by [24] such as a mixture of cutmix [44] and mixup [46] and regularization technique stochastic drop rate. We set the drop rate [21] of 0.2, 0.3, and 0.5 respectively for MOA-T, MOA-S, and MOA-B. Table  III shows our model's result and a similar Transformer-based model on the ImageNet-1K classification task. Our proposed models: MOA-T, MOA-S, and MOA-B, achieve higher accuracy than most of the Transformerbased models with significant parameter reduction. MOA-T outperforms Twin-S and Swin-T by 0.34% with around 22% fewer parameters. Our MOA-S improves the performance by 0.5% and 0.3% compared to Swin-S and Twin-M respectively, even with the lower batch size during training. Our MOA-B achieves the state-of-the-art accuracy of 83.7% on ImageNet-1K with comparatively fewer parameters with a smaller batch size than the remaining vision transformers. Our model increases the computation cost by a negligible amount, but the  performance improvement and parameter reduction are highly rewardable.

V. ABLATION STUDY
In this section, we conduct ablation experiments to understand the effect of the dimension of each component, such as window size, the overlapped area between the key-value patches, and the reduction factor in global attention, in our model. We employ the Tiny model to perform all ablation

A. Window-size
The sequence length of the local-Transformer is one of the essential factors on which computation cost relies. As the sequence length increases, computation cost in the selfattention mechanism increases as well. In a local vision Transformer, sequence length depends on the window size. There is always a trade-off between the accuracy and computation cost based on the sequence length. We perform experiments with various window sizes in our model and find that 4 × 4 and 14×14 window size works well on CIFAR-100 and ImageNet datasets, respectively, as shown in Table IV. Furthermore, we remove the stages where the window size is greater than the feature map size to significantly reduce the number of parameters.

B. Overlapped Portion
To initiate the neighborhood information transmission, we propose to use slightly larger and overlapped keys. To investigate the effect of the portion of the overlapped area, we perform experiments with different percentages of overlapped portions in keys as shown in Table V. It can be seen from the results that the performance is increased in terms of accuracy as the percentage decreases, which means only slight information exchange between the neighborhood windows are required to improve the performance. Furthermore, fewer overlapped portions decrease the sequence length, which reduces the number of parameters and GFLOPs.

C. Reduction
Before the MOA global attention, the hidden dimension is reduced to decrease the number of parameters and computation cost. Table VI shows the performance of our model with various values of R. From the result, it is evident that R = 32 achieves the best result with comparatively less number of parameters and computation cost than a smaller value of R.

D. Effect of Overlapped Key-Value
To verify the effect of overlapped and larger key-value patches, we train the model without overlapping patches and compare the results. Furthermore, we also conduct an experiment without applying global attention in-between each stage to verify the significance of global information exchange. From the result in Table VII, it can be seen that including global attention and overlapped key-value patches achieve the best performance.

VI. CONCLUSION
The paper has investigated the effect of aggregating global information in local Transformer after each stage and neighborhood pixel information transmission. We have also proposed a multi-resolution overlapped attention (MOA) module that can be plugged in after each stage in the local transformer to promote information communication along with nearby windows. Our results show that both types of features: global and local, are crucial for image classification. As a result, exploiting both features leads to significant performance gain on the standard classification datasets such as CIFAR10/100 and the ImageNet with comparatively fewer parameters.