{'fraction_non_alphanumeric': 0.029337836776372635, 'fraction_numerical': 0.01154661849029927, 'mean_word_length': 4.98566055477198, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Efficient Multi-Purpose Cross-Attention Based Image Alignment Block for Edge Devices

Image alignment, also known as image registration, is a critical block used in many computer vision problems. One of the key factors in alignment is efficiency, as inefficient aligners can cause significant overhead to the overall problem. In the literature, there are some blocks that appear to do the alignment operation, although most do not focus on efficiency. Therefore, an image alignment block which can both work in time and/or space and can work on edge devices would be beneficial for almost all networks dealing with multiple images. Given its wide usage and importance, we propose an efficient, cross-attention-based, multi-purpose image alignment block (XABA) suitable to work within edge devices. Using cross-attention, we exploit the relationships between features extracted from images. To make cross-attention feasible for real-time image alignment problems and handle large motions, we provide a pyramidal block based cross-attention scheme. This also captures local relationships besides reducing memory requirements and number of operations. Efficient XABA models achieve real-time requirements of running above 20 FPS performance on NVIDIA Jetson Xavier with 30W power consumption compared to other powerful computers. Used as a sub-block in a larger network, XABA also improves multi-image super-resolution network performance in comparison to other alignment methods.


Introduction
Image alignment (image registration) aims to align or match images to a chosen reference image. This task constitutes an important part of many computer vision problems dealing with multiple images either in space and/or time, such as restoration [41,47], segmentation [21], HDR imaging [45], stereo imaging [40], multi and single-image super-resolution and video super-resolution [24,43,44].
To construct a backbone used in many vision-related problems, there have been many image alignment methods/blocks developed to this day with different focuses. These methods can be divided into three main categories: feature-point extraction-based alignment, classical optical flow-based alignment and deep learning-based alignment. Feature-point extraction-based alignment algorithms utilize extracted feature points to create a global transform matrix between reference and target frames. Later, the resulting matrix is used to warp frames or images onto each other [34]. Classical optical flow-based alignment algorithms compute the flow vectors between reference frame pixels and target frame pixels, then use the flow vectors in the warping process. These methods can vary greatly, from computing affine transforms of image patches using flow vectors [27] to using a contrast constancy assumption and iteratively reducing misalignments between image pairs [10]. Deep learning-based alignment algorithms inherit ideas such as deformable convolutions [13] and deep-learning based optical flow [15], and attention mechanisms [47].
Despite having been implemented in many ways, most of the aforementioned image alignment methods suffer trading between accuracy and speed, especially on low-end devices. Feature-point extraction-based alignment algorithms require the planar scene assumption, which does not hold most of the time. Although having efficient implementations, classical optical-flow algorithms can fail with large motions. They can also cause bottlenecks when used in deep learning pipelines, as it avoids being able to provide an end-to-end deep learning-based solution. This result may result in sub-optimal solution, since optical flow part is not trained and is based on hand-crafted features. This is indeed the problem of integrating classical algorithms into deep learning pipelines.
On the other hand, having the entire solution blocks in deep learning framework helps using hardware at the hand to the full extent easily by using the already supplied deployment tools such as NVIDIA's TensorRT [4] and Intel's OpenVINO [6], and improves the performance of the overall trained pipeline. Besides, deep learning-based alignment methods are fairly more recent and have promising advancements. Deformable convolutions add 2D offsets to sampling locations in traditional convolutions, introducing more adaptation and easing the alignment [18]. Deep learning-based optical flow methods [36,38] yield accurate alignment; however, they can be quite expensive and hence may not be feasible to run on embedded or edge devices. Recent work on image alignment used attention mechanism because of its native feature matching and transforming properties [7,47], however direct application of attention operation for image alignment can be memory and computation hungry. The use of attention mechanism started with natural language processing domain with the seminal works [9,28,39] and later transferred to image domain [14] and it is known to improve the performances of various networks on image domain [42] as well. Nonetheless, using attention for image alignment is fairly recent and not much study is present, especially focusing on edge device interference efficiency.
Motivated by the recent performance & accuracy related studies on attention, the lack of efficient attention-based alignment studies, and the lack of a general-purpose deep learning-based image alignment block applicable to different problems on edge devices; we propose a cross-attention based image alignment block which can be integrated into many deep learning pipelines with ease. We name our pro-posed method as XABA (cross-attention based aligner). XABA is designed with efficiency and plug-and-play approach in mind that can run in real-time. We divide reference and target images into non-overlapping sub-images, extract features from each sub-image, and efficiently compute a cross-attention matrix to align sub-images in featurelevel. Dividing images into blocks allows us to process them in parallel, ease implementations on embedded systems, and force local information extraction. In addition, to boost the proposed methods performance in large motions, the baseline block is applied in a pyramidal fashion to the input image at different scales which effectively increases the attended area while keeping the computational cost requirements at minimum. The final output of the network is used by fusing the pyramidal network outputs with a pixel attention based fusion module. We also do not have handcrafted hyper-parameters for our cost function unlike [7], which further eases the training process.
For more robustness and adaptation to different applications, we also propose a sparsification scheme to calculate the attention matrix, which can be used to find sparse attention matrices. This can be further exploited for fast matrix multiplication during inference.
Furthermore, as an alternative to the classical softmax non-linearity used in attention matrices, we combined hardthresholded ReLU (clips above 1) with row-normalization as the activation function. This effectively normalizes the rows of the calculated attention matrix, promotes sparsity, and makes the implementation more suitable to be used in edge devices due to the lightness of the activation function.
The main contributions of this paper are: • We propose a block-based, pyramidal, multi-purpose, deep-learning based image alignment block using cross-attention.
• We propose an alternative to softmax activation, combining hard-thresholded-ReLU with Normalization.
• The proposed block is efficiency-focused and real-time applicable, proven by multiple tests on edge devices.

Related Works
Feature-point based alignment. This method aims to match feature points extracted from images via [20,34,46]. Using outlier methods like RANSAC [16] and their derivations [31,32], some extracted feature points are eliminated. Then, using rest of the matching feature points, a global transformation matrix is generated between images. The major downside of this method is that the motion is restricted to be globally uniform. In other words, every part of the image is being subjected to the same global motion, which is not correct most of the time.
Traditional & deep-learning-based optical flow alignment. Optical flow calculates flow vectors for each pixel between reference image pixels and target image pixels. Since a one-to-one correspondence is obtained between pixels (for occlusion-free regions), images can be aligned accordingly. Optical flow is an under-constrained problem, hence additional constraints like brightness consistency and confining to small movements are required. Lucas-Kanade [27] and Horn-Schunck [19] are two of the most well-known classical flow algorithms, with other variations also present [37]. Classical flow algorithms are still being developed, one of the recent ones being DISFlow [26] which focuses on time complexity. Deep-learning based optical-flow, on the contrary, are much more recent which gained momentum with FlowNet [15]. FlowNet proposed using a convolutional network in flow estimation for the first time. FlowNet2 [23] offered using correlation layers as an improvement. It also included a new stacked architecture including image warping, and sub-networks for small displacements. PWC-Net [36] refined flow in a coarse-tofine manner and utilized feature warping to reduce the network size. In addition to performing better than FlowNet2, PWC-Net also has a smaller network size. Recent algorithms like RAFT [38] started to utilize correlation volumes and recurrent structures to further increase the flow quality. RIFE [22] directly estimates intermediate flow estimations from a low-framerate video for frame interpolation purposes. Even though most deep-learning based flow methods have high accuracies, they are not applicable on embedded environments due to their high computational demand at practically meaningful image sizes.

Attention and attention-based alignment.
Bahdanau [9] and Luong [28] models, also known as additive and multiplicative/dot-product attentions respectively, are the first remarkable studies about attention. Later on, Google proposed their attention-based network architecture, the Transformer [39]. All these studies were done on natural language processing; however, attention-based solutions in computer vision problems were also starting to emerge. Inspired by non-local means denoising algorithm [11], Wang et al. [42] proposed a non-local building block which captures long-range dependencies within feature maps. This study prepared the ground for many vision applications, such as classification [33,48], object detection [12,17], image segmentation [21,30], image superresolution [40], and video super-resolution [24,43,44]. Regardless, the idea of attention is fairly new on image alignment algorithms. Only a handful of studies [7,43,47] are present, most of which do not focus on efficiency and deployment on embedded environments.

Background information
Attention mechanism. The attention mechanism proposed in [39] and [28] has three main concepts: key, query and value.
Query (Q) is the input vector for which the attention is desired to be calculated. Query is compared with all keys (K) by taking the dot products between Q and K, creating a key-query matrix (QK T ). Higher values in the resulting matrix indicate higher correlation between the relevant elements of Q and K. The matrix is then normalized with some constant ( √ d k ) and its softmax is calculated to make the sum of each row 1. At the end, multiplying with values (V ) gives the attention (1).
In self-attention, Q, K and V are equal to each other. Naturally, the equality does not hold for cross-attention. In our study, Q represents the feature matrix of the reference image, the image to be aligned. K is the feature matrix of the target image. V is the reference image itself.

Baseline Block -Interframe Aligner
Our baseline block structure which we refer as interframe aligner block is given in Fig. 3. Using cascaded convolutional layers with skip connections and ReLU activations, features of the input images (I 1 , I 2 ) are extracted. Note that the parameters of this residual feature extraction network is shared between images. These features are then convolved with 1x1 kernels to reduce the number of operations. Note that depending on the application, the layer with 1x1 kernel may or may not share the same parameters with its parallel branch. For instance, 1x1 convolutions can be shared in aligning two RGB images as done in our experiments, however; to match images in different domains (such as thermal and RGB images) further adaptations may be needed.
After the feature extraction, resulting features are sent to tensor-to-block (T2B) operator. T2B operator divides the feature images into non-overlapping patches and stacks these blocks in batch dimension. In this sense, it is somewhat similar to pixel unshuffling [35] where the unshuffled pixels are stacked on the channel dimension. This enables us to only match spatially closer features with each other while allowing parallel processing.
To find the correlation between two image features, T2B outputs are matrix multiplied to create a dot-product attention matrix as described in Eq. (1). After normalization along the rows of the result with a non-linearity (softmax, or hard-thresholded ReLU with Normalization), A, the attention matrix between two input image features is generated.  . Baseline block -interframe aligner. Reference (I1) and target frames (I2) are sent as inputs and aligned frame I21 is obtained at the output. fe and fm are modifiable dimension parameters for feature extraction. T2B is the operator transforming tensors into non-overlapping blocks and reorganize the tensor in the batch dimension, while B2T is the inverse operator. * is matrix multiplication operation. Activation is the function to normalize the attention matrix, A.
Note that this matrix A transfers images from K (key) domain to Q (query) domain via an adaptive linear combination and blending. This linear combination map is then used for aligning I 1 and I 2 . Block-to-tensor (B2T) operation is applied at the end to reverse the effects of T2B operator, which is similar to pixel shuffling [35] where it operates along the channel dimension.
As discussed above, the baseline block is good enough to capture and align small displacements between features. However, to be able to capture large motions, nonoverlapping block size should be increased which is a parameter of T2B operator. Unfortunately, increasing the block size increases the size of A, and hence it increases the computational load and memory requirements which is not suitable for edge devices. As an alternative to handle large motions, we propose a pyramidal processing scheme which is more efficient and suitable for edge devices.

Pyramidal Global Alignment Block
As described in Sec. 3.2, our baseline interframe aligner is suitable for capturing local feature matches. In other words, it is good at capturing small displacement of features between images. To effectively handle large motions and effectively increase the block size while being computationally light, we propose another block which we refer as Pyramidal Global Alignment Block. This block encapsulates different number of baseline blocks dedicated to work with different scales of the input images (Fig. 2).
An input image pair (I 1 , I 2 ) is sent to the alignment block. Each baseline interframe aligner takes down scaled input image pairs ( K I 1 , K I 2 ), where the downscaling factors are denoted by K = {1, 1 2 , 1 4 , ...

Pixel Attention Based Fusion Block
Individual outputs of the baseline blocks for different scales constitute candidate images and hence candidate pixels. Inherently, at unit scale level, the image resolution is high but only small displacements are handled. At 1 2 scale, level medium displacements are handled but the image resolution is lower. At 1 4 scale level, very large displacements are handled; however, the resolution is at its lowest. Given these, a selection mechanism is needed. Pixel Attention Based Fusion Block is used for this purpose in such a way that it tries to combine different outputs to form a single image.
The fusion block inherits a cascaded convolutional network which generates a mask M using all interframe aligner outputs, shown in Fig. 4. Concatenated aligned images (I C ) are passed through the CNN and the activation function, which performs normalization in the channel dimension. This ensures that the contribution of all interframe aligner output energies are unchanged. Each channel of the resulting mask M are then used as a multiplicative mask for the corresponding images in I C and all masked images are summed up to obtain the final image. Note that the mask in this case determines the combination ratio of all interframe aligner outputs from different scales.

Experiments
For the experiments, we used Kitti [29] dataset which is commonly used in image alignment, stereo and optical flow benchmarking. Kitti includes 200 training and 200 test stereo scene pairs, captured in rural and city traffic. Images are in RGB and lossless png format, with resolutions not the same among all images but all around 1250x375.
The experiments can be divided into three different sections. In the first experiment, we used XABA by itself for image alignment. In the second experiment, we used the image pairs of the Kitti dataset and posed a Multi Image Super-Resolution problem (MISR) and showed the performance of XABA in combination with a Single Image Super-Resolution (SISR) network to solve MISR problem. In the third experiment, for the different parameter settings and block sizes of XABA, we have taken measurements from NVIDIA Jetson Xavier and showed real-time capabilities of the proposed method.

Training Details
For the first and second experiments, we used Adam optimizer with β 1 = 0.9 and β 2 = 0.999 and used maximum learning rate = 2e-3 with Knee learning rate scheduling [25] and warm up for all of our experiments. Mini batch size of 8 is used and the models are trained for 170 epochs, where each epoch consumes the training images 10 times. Each mini-batch is composed of image patches cropped from random images from the training set and standard geometric transformations such as rotate & flip were used as a form of data augmentation. We used 320x320 as crop size and for the second experiment (Sec. 4.3) the low resolution cropped images were created by 4 times downscaling the original images. In both experiments, Charbonnier loss was used with = 0.1 as defined in (2). Charbonnier loss is the smoother version of L1-loss, which is known to have better convergence characteristics than L2-loss.

Alignment Performance
For this experiment, we used images from Kitti dataset paired in time as inputs to XABA and tried to align these images and warp the reference image to the target image. An example pair and alignment results of the different methods can be seen in Fig. 5 As shown in Fig. 5, optical flow based methods' performance drastically drops whenever there is a large motion. This is basically due to the fact that these methods constraint the change in between frames to planar geometric motion with 1-to-1 pixel correspondence. These constraints from the point of the view of attention mechanism are indeed equivalent to limiting the Attention matrix, A, to a permutation matrix where there is one and only one entry being 1 for all of its rows. However, in our case, we can "relax" the permutation matrix constraint by letting the sum of each row to 1 (by using softmax or hard-thresholded ReLU with Normalization), rather than forcing only a single element to be 1 in each row. This relaxation allows contribution of multiple pixels and blending, which warps the reference image to the target image with better performance, which can be seen in Tab

Super-Resolution Performance
To show the effectiveness of XABA as a sub-network of a greater network, we conducted a multi-image superresolution experiment. Two different images of the same scene is given as an input to a network to find x4 higher resolution image of the same scene. For this experiment, we selected XLSR [8] as the SISR baseline network. Then  only by changing the input filters to accept two input RGB images, we constructed so called XLSR MISR. By combining XLSR MISR with DISFlow+Warp and XABA, we constructed XLSR MISR + DIS + Warp and XLSR MISR + XABA, respectively. Here for XABA, we used two different activation functions. PSNR values are calculated using the original high resolution images with RGB outputs. As seen in Tab. 3, when an input tensor adjusted SISR method (XLSR MISR) is combined with our XABA (HT+N), the performance is improved by 0.1dB. Furthermore, the SISR method is chosen to be with low number of parameters to limit its receptive field to a limited region. This causes degradation on the performance on MISR problem, as shown in XLSR MISR without any alignment. The usage of correct alignment with XABA shows the effectiveness of our algorithm. Although DISFlow corrects and aligns the relevant data in the receptive field region, Kitti image pairs usually have large motion and large motions cannot be effectively compensated. An example output of the different alignment methods combined with the superresolution network can be seen in Fig. 6.

Embedded Benchmarks Performance
In this experiment, we have investigated the computational performance of our proposed method XABA on an embedded computation device. For this purpose, NVIDIA Jetson AGX Xavier GPU [2] was used in the benchmark

Timing Results and Power-Efficiency Analysis
The inference performances of different configurations of the proposed model are comparatively demonstrated in this section. Note that Tab. 4 indicates timing performances of global and local alignment models. It can be seen that using FP16 for inference is faster than using FP32. On the other hand, the global alignment inference performance can reach about 170 frame per seconds (FPS) on high performance computing device, which is included in the table as a reference to compare it with Jetson Xavier's performance.
As it can be seen from the Tab. 4, it is possible to run the proposed block in real-time on Jetson for some specific configuration such as XABA10fp16Soft which indicates the non-overlapping block size for T2B operator is 10 and FP16 is used for inference and activation in Attention module is softmax. Note that by using Hard-Thresholding we could increase the performance of the similar block using softmax in all of the cases and this simple change added almost +3FPS to most of the configurations. Note that local alignment section is also given in the table to get a better understanding of the effects of pyramidal processing and baseline block. Also note that local alignment block can be seen as a pyramid with 0 depth and can still have meaningful usages for aligning smaller displacements such as aligning frames of a video stream. Fig. 7 presents FPS performance results of global align-   Figure 7. The global alignment inference performance benchmarks in terms of FPS for different block configurations.
The computing performance of XABA10fp16Soft inference model meets real-time requirement in terms of FPS for image alignment processing according to the results in Fig. 7. Indeed, the bottleneck analysis was performed with NVIDIA Profiler for this model. According to bottleneck analysis, the computing time of softmax layer showed that this layer has dominant computation compared to other layers. This inspired our Hard-Thresholding activation proposal which replaces softmax function in activation function. The global alignment inference performance has been increased to 26 FPS from 23 FPS by using hard thresholding and FP16 precision in XABA10fp16HTN. This performance result provides 10% speedup compared to previous inference model (XABA10fp16Soft).
The power efficiency experiments proved that Jetson AGX GPU provides the most power efficient computation with respect to comparative results in Fig. 8. According to Fig. 8, the Jetson AGX GPU performance provides about 1.6x power efficiency compared to RTX3080 GPU under FP16 precision. From this point of view, this edge device meets real-time image alignment processing requirement in FP16 inference case with 30 Watts power consumption. Although RTX 3080 GPU has high computation performance, its power efficiency performance gave drastically lower result compared to Jetson AGX GPU.

Conclusion
In this study, we proposed our multi-purpose, crossattention based image alignment block, XABA. By processing the images in blocks inside a pyramidal block based alignment structure, we capture local relationships with minimal computational need. Focusing on efficiency, we further prove with tests that XABA can run in real-time on edge devices such as NVIDIA Jetson Xavier.
Our experiments reveal that XABA can outperform common optical-flow based alignment methods. We have also shown that XABA can be used as a sub-network aligner in larger deep-learning based scenarios like single and multiimage super-resolution with good performance. Embedded benchmarks and power analyses further prove that pyramidal structure of XABA allows us to realize a power-efficient image aligner.