{'fraction_non_alphanumeric': 0.035130970724191066, 'fraction_numerical': 0.011754347347567686, 'mean_word_length': 4.906396255850234, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
VidFace: A Full-Transformer Solver for Video FaceHallucination with Unaligned Tiny Snapshots

In this paper, we investigate the task of hallucinating an authentic high-resolution (HR) human face from multiple low-resolution (LR) video snapshots. We propose a pure transformer-based model, dubbed VidFace, to fully exploit the full-range spatio-temporal information and facial structure cues among multiple thumbnails. Specifically, VidFace handles multiple snapshots all at once and harnesses the spatial and temporal information integrally to explore face alignments across all the frames, thus avoiding accumulating alignment errors. Moreover, we design a recurrent position embedding module to equip our transformer with facial priors, which not only effectively regularises the alignment mechanism but also supplants notorious pre-training. Finally, we curate a new large-scale video face hallucination dataset from the public Voxceleb2 benchmark, which challenges prior arts on tackling unaligned and tiny face snapshots. To the best of our knowledge, we are the first attempt to develop a unified transformer-based solver tailored for video-based face hallucination. Extensive experiments on public video face benchmarks show that the proposed method significantly outperforms the state of the arts.


Introduction
Video-based face hallucination (VFH) targets at recovering a high-resolution (HR) human face from low-resolution (LR) video frames. This task has gained long-standing interest in the research community [12,28,14,33,38,15] due to its wide applications such as emotion recognition [8,13] and video surveillance [40]. As a domain-specific task in video super-resolution, VFH is far less explored due to its high complexity and requirement in jointly modeling both spatio-temporal and facial prior information.
Recent studies [3] have pinpointed two critical steps of VFH approaches, i.e., alignment and aggregation. Previous methods typically conduct pairwise alignment and aggregate frames with estimated optical flow or deformations. Due to the large pose or expression changes in face video snapshots, these approaches may fail to establish frame-wise correspondences accurately especially when input videos are in very low resolution. Furthermore, the pairwise aggregation manner neglects the long-range temporal information offered by an integral sequence but faces the risk of accumulating alignment errors. Last but not least, to the best of our knowledge, previous VFH methods often rely heavily on beforehand aligned faces in terms of facial landmarks. Considering that raw faces in LR videos are naturally tiny and blurry [32], this would drastically hinder the accuracy of face alignment. To date, how to effectively align and aggregate both spatio-temporal and facial prior information for VFH remains challenging.
In this paper, we propose a new video face hallucination method, dubbed VidFace, to super-resolve unaligned tiny face snapshots. VidFace is a full-transformer solver addressing alignment and aggregation in a unified scheme. It capitalizes on the contextual modeling ability of the attention mechanism to harness integral information of all the given snapshots, both spatially and temporally, for face hallucination. Specifically, VidFace starts with splitting snapshots into overlapped patches and flattening them into spatio-temporal tokens. Inspired by the structural design of Token-to-Token network, we aggregate neighboring tokens iteratively to enrich the feature representation of facial details. To further utilize the facial structural information, we propose a novel recurrent landmark positional encodings (RLPE) to equip our transformer with facial priors. RLPE is designed to recurrently estimate the landmarks and attentions together in a mutual-promotion manner, which regularises the feature alignment and aggregation without the need of pretraining transformers.
We also present a new upsampling module, dubbed Detoken Upsampling (DeU). Apart from previous upsampling such as deconvolution and pixelshuffle that operated on each local patch, we design a novel "Detoken Transformer" to decode tokens into dilated ones to enlarge the latent feature resolution and then reconstruct these decoded tokens into HR images. In such dilation process, facial structure information is naturally incorporated with the help of global attention and landmark positional encoding, thus leading to superior face hallucination performance.
Overall, our main contributions are summarized as follows: • We present a unified pure transformer-based solver, namely VidFace for video based face hallucination by utilizing the long-range spatial and temporal information from inputs integrally. To the best of our knowledge, we are the first attempt to super-resolve face videos by a pure transformer-based solver. • We propose a novel recurrent landmark positional encoding (RLPE) to equip our transformer with facial priors. RLPE is designed to recurrently estimate the landmarks and attentions together in a mutual promotion manner, significantly facilitating facial feature alignment and aggregation. • We curate a new large-scale benchmark dubbed TUFS-145K, and LR faces are unaligned in TUFS-145K, posing a more realistic and challenging scenario for existing baselines. • Extensive experiments demonstrate the new state-of-the-art face hallucination performance of the proposed VidFace.

Related Work
Video Super-resolution. Dong et al. proposed a seminar idea to tackle image super-resolution (ISR) with a Convolutional Neural Network [5]. Based on this pioneering work, many deep learning-based approaches have been proposed for video super-resolution (VSR) task [29,1,30,25,10,9,26]. Comparing to ISR, VSR lays more emphasis on exploiting temporal alignment across frames. Some methods [10,1,25] utilize optical flow to estimate the motions between frames. Xue et al. proposed TOFlow [30], an end-to-end trainable network to predict task-oriented motion and fuse all input frames according to estimated motion fields. Besides, several methods [9,29,26] supplant flow estimation with implicit alignment mechanism. TDAN [26] employs deformable convolution to align neighbouring frames at the feature level. Based on [26], EDVR [29] improves previous works with a Pyramid, Cascading and Deformable (PCD) alignment module to align multi-frames.
Face Hallucination. Face hallucination, a domain-specific super-resolution task, aims at generating high-resolution facial images from low-resolution inputs. Based on deep learning, face superresolution methods have been actively researched and achieved impressive progress. Yu et al. [33] super-resolve aligned tiny face images with GAN-based models. In order to process unaligned face images, Yu et al. [34] insert multiple spatial transformer networks (STN) in a generator to hallucinate LR images. Cao et al. [2] learn the recurrent policy network and local enhancement network through deep reinforcement learning. Zhang et al. [39] propose a two-branch super-resolution network (CPGAN) to upsample face images while compensating for low and nonuniform illumination.
Vision Transformer. Transformer models have been heralded as a breakthrough in Natural Language Processing (NLP) domain, and are gradually applied to a wide range of computer vision tasks such as semantic segmentation [21,18,19,20] due to its feature extraction ability and impressive performance. Yang et al. [31] formulated LR and Ref images as queries and keys in a transformer, which contained a hard-attention module and a soft-attention module to discover deep feature correspondence and transfer rich textures to help super-resolving the input LR image. Different from Yang et al. [31], we formulate central frames and other frames as queries and keys in order to search relevant regions between them and fuse the correspondent feature in a pyramid transformer block.
Our work does not follow the strategy of using local-perceptual operator as optical flow or deformable convolution. Instead, a pure-transformer-based method is proposed to harness global information spatially and temporally to exploit better alignment. To our knowledge, we are the first works to propose a full-transformer solution for VFH.

Problem Setting and Overall Idea
Given N unaligned tiny low-resolution face snapshots captured from human video, denoted as {LR n } where n ∈ {1, ..., N }, we aim at super-resolving the reference frame LR ref ∈ {LR n } to restore the facial details. Our task can be clearly distinguished from traditional facial video super-resolution in two folds. First, our method operates on tiny inconsecutive data, e.g., each snapshot is sized 16 × 12 with large pose or expression divergence. Second, the faces are not aligned to their landmarks across snapshots by any pre-process. To our knowledge, such tiny, inconsecutive and unaligned facial snapshots pose a practical scenario that challenges all existing efforts. For convenience, we denote other snapshots as auxiliary snapshots LR aux , which are utilized to supplement the reference one LR ref in hallucination.
Our overall idea is to capitalize on the contextual modeling ability of attention mechanism to harnesses all the spatial, temporal and facial prior information of given face video snapshots. We hence propose the VidFace, an unified transformer-based solver for VFH, which boosts existing methods in all the alignment, aggregation and upsampling steps. We detail the VidFace in what follows.

Network Architecture
The network architecture of VidFace is illustrated in Fig. 1. It is composed of an encoder E, a refiner R and a decoder D, wherein E is responsible for an initiatory spatio-temporal feature alignment and aggregation, R recurrently refines the latent features with facial structure prior and D generates the final HR face with a facial structure-aware upsampling.
Encoder as Token-to-token Transformer. The design spirit of the encoder E is inspired by Tokento-Token (T2T) ViT but also introduces many improvements tailored for VFH. Given N face video snapshots {LR n } where n ∈ {1, ..., N }, we first Unfold them to softly split snapshots into overlapped patches. These patches are then flattened in to tokens T = {T n } (where T n denotes the tokens in LR n ) and forwarded to transformer layer for feature extraction. Finally, the output tokens from transformer are restructured as an image again to bootstrap the next unfold-reconstruction iteration. For a (restructured) LR snapshot LR n ∈ R h×w×c , the number of output tokens can be calculated as: where k is the size of patch, s is the stride and p is the padding on LR n . Then the total number of tokens in N snapshots will be L = N n=1 l n . Different from vanilla T2T process which reduces the number of tokens progressively, VidFace keeps same amount of tokens to maintain the resolution during encoding. VidFace also stacks the tokens in different layers to further enrich the feature with multi-scale representation. As the neighboring (either intra-or inter-snapshot) tokens are aggregated iteratively into single token, the spatio-temporal information are naturally embedded into them. Moreover, the interaction between overlapped tokens also helps to extract more structure information like edges and textures, which is of pivotal for facial detail hallucination.
In order to maintain the spatial structure of snapshots when splitting tokens, we add the spatial position encoding to tokens as in vanilla transformer. It is noteworthy that we do not consider temporal position encoding since we assume that the "snapshots" are unordered compared to "frames".
Refiner as Recurrent Landmark Transformer. We further refine the initiatory spatio-temporal tokens by introducing facial structure information, which has been proven as a robust prior to regularise the feature alignment & aggregation [34,36,7]. However, to utilize such prior effectively in a transformer is non-trivial. Here we propose to additionally estimate the facial landmarks during the training and employ the estimated facial landmarks in positional encoding. To achieve the auxiliary landmark estimation task, we first utilize a linear projection to decrease the dimension of every tokens. Then all tokens from the same snapshot will be flattened into a one-dimensional feature. Followed by a linear layer and a landmark head, these 5 landmark positions of the face will be predicted as: LM n = LMHead(MLP(Flatten(T n,0 , ..., T n,ln ))), (2) where l n = h × w denotes the tokens number of snapshot LR n .
To represent the positions encoding of each token in terms of landmarks, we employ the estimated facial landmarks as the anchors. The landmark position encoding of token T n,i is denoted by RLP E n,i ∈ R 5 , which is reflected in the distance between token T n,i and 5 facial landmarks. Concretely, RLP E n,i can be calculated as: RLP E n,i = [d(P n,i , lm n,1 ), ..., d(P n,i , lm n,5 )], (3) where d(.) indicates the Euclidean distance between two points in a snapshot, P n,i is the position of token T n,i and lm n,j indicates the j-th facial landmark of the n-th snapshot obtained by the recurrent transformer, which serves as the anchor point when calculating RLP E n,i . In order to handle faces that occupy different sizes in images, we normalize RLP E (i.e., RLP E RLP E 1 ) before concatenating it to the tokens. Since the auxiliary head is trained from scratch and the estimated landmarks are inaccurate in early iteration, we propose to recurrently learn the facial landmark and attention map together in a mutual-promotion manner. Specially, the yielded landmarks would recurrently embedded into the position encoding, together with the refined tokens to bootstrap the next refine iteration, as shown in Fig. 1. Afterwards, the final tokens would be forwarded to the decoder.
Decoder as Detoken Upsampling. We propose Detoken Upsampling (DeU) to conduct the facial structure-aware upsampling, which reconstructs the latent tokens into 8× facial images, expressed as: where d model represent the input channels, respectively. DeU is composed of two key modules, i.e., Detoken Transformer (DeT) and Tokens to Image (T2I).
Detoken Transformer (DeT) is designed to overcome the limitation of deconvolution [22] or sub-pixel convolution [24] on their inherent local perceptivity. DeT decodes tokens into dilated ones to enlarge the latent feature resolution. The feature of every token has the dimension of d out * k * k, where k is the kernel size used in T2I and d out is the output dimension of T2I. In such dilation process, facial structure information is naturally employed with the help of global attention and landmark positional encoding.
Residual Image Figure 2: The architectural overview of DeU.
As depicted in Fig. 2, for video tokens T whose feature dimension is d model , we utilize multi-head attention [27] to extract the d qdimensional queries, d k -dimensional keys and d v -dimensional values for h times with different, learnable linear projections. We remove the vanilla feed forward layer after the multihead attention to maintain the global perceptual field. Formally, the selfattention process can be presented in the following function: where the projection matrix For instance, when using d q = d k = 32, d v = 1024 and h = 8, the token feature would be extended, thus can be then folded to a 8× feature map using T2I.
Tokens to Image (T2I) refers to the eventual step for face hallucination to reconstruct tokens into images. We take the tokens T n ∈ T from snapshot n as an example to illustrate T2I as shown in Fig. 2, where T denotes the overall video tokens generated from DeT. The reconstruction process can be formatted as: HR n = Reshape(MLP(T2I(T n ))).
Inspired by the implementation of deconvolutional layer, we implement T2I with fold (a.k.a, col2im) as follows: T2I(T n ) = Fold(T n , k, s, p), (7) where k, s, p denote the kernel size, stride and padding used in Fold, respectively. For instance, when setting k = 16, s = 8 and p = 4 as in our work, we can upsample LR snapshots into 8× HR faces which can be regard as an inverse operation of T2T.

Training Objective
The training objective of VidFace is twofold. For the main task of face hallucination, we adopt the Charbonnier Loss [11] to produce better refined edges during training. Specially, Charbonnier Loss is defined as: whereĤR n is the predicted high-resolution result, and is a small constant.
Therefore, the final loss is formulated as: where λ 1 and λ 2 denote hyper-parameters controlling the relative importance of the two losses.

Experiments
To demonstrate the advantages of VidFace, we have conducted extensive experiments on both public and our new curated datasets. We first briefly introduce the evaluation datasets, the corresponding evaluation protocols, and the implementation details. Then, we perform comprehensive comparisons to verify the superiority of VidFace over all the compared state-of-the-art approaches. Finally, we perform in-depth analysis on each module with ablation study to pinpoint their contributions for VFH.

Datasets and Evaluation Protocols
We employ two datasets for a comprehensive evaluation to validate the effectiveness of VidFace. IJB-C is a video-based face database which contains 3, 531 subjects with 31.3K still images and 117.5K frames from 11, 779 videos. We set a threshold to filter out the over-short video whose length is less than 4 and results in a subset with 6.4K video clips. TUFS-145K is origin from the public voxceleb2 [4] dataset. We first detect and crop the face boundingbox of the videos every several frames with pyramidbox tool. We score the candidate clips with (1) faceQnet to judge the quality of cropped faces and (2) 3ddfa to estimate the pose variance. By combined considering these two scores, we select 145K 7-frame clips from the candidates to constitute TUFS-145K, which has favourable face quality and large pose variance. TUFS-145K is divided into a training set of 116K video clips, a validation set of 14.5K clips and a testing set of 14.5K clips. During the network training, we employ Bicubic downsampling to generate the LR face video snapshots as the input data.
The estimated high-resolution images are evaluated by peak signal to noise ratio (PSNR) and the structural similarity index (SSIM) on Y channel (i.e., luminance) of the transformed YCbCr space and RGB space.

Implementation Details
We use PyTorch [23] for our implementation. We employ T2T-ViT [35] as the backbone. In order to stablize the training, we choose AdamW [17] and Lookahead [37] as the optimizer. In addition, we set 0.05 as the maximum l 2 norm for gradient clipping. We train the network for a total of 600k iterations. And the learning rate is decreased by cosine annealing scheduler [16] without restart. We resize the snapshots to 16 × 12 with Bicubic operation as the LR input and the length of the snapshots is set to 7. These frames are shuffled in training to improve the generalization. The batchsize is set to 16, which is the same with MUCAN [11] and EDVR [29] in all experiments for fair comparison.

Comparison with State-of-the-Art Methods
We compare our method with several state-of-the-art methods, including EDVR [29] and MU-CAN [11]. EDVR is a typical deformable convolution-based method. MUCAN depends on the self-attention modules for alignment and aggregation, but limited to pairwise frames. It is noteworthy that we have not reported the results of optical flow-based methods since these methods are not applicable for over-small unaligned snapshots (the spatial pyramid used in optical flow calculation requires  VidFace hallucinates more facial details specially in "eyes", "mouth" and so on. Zoom in for best view. the image to be larger than 16 × 16). The hallucination results of Bicubic are also reported. The PSNR and SSIM comparisons between VidFace and the above mentioned methods on TUFS-145K as well as IJB-C by 8× upscaling are shown in Table 1. The average Y-channel and RGB-space PSNR and SSIM on all the evaluation snapshots are reported.
Firstly, our proposed VidFace achieves the highest PSNR, yielding 30.94dB on TUFS-145K and 26.35dB on IJB-C and significantly outperforming EDVR over 0.4dB and 0.2dB, which demonstrates the superiority of a full-transformer base solver over the conventional deformable convolutional. The similar result can be also observed on SSIM score.
Secondly, VidFace significantly outperforms MUCAN by 0.45dB and 0.24dB. Since the alignment mechanism of MUCAN also relies on spatial self-attention, such better results draw our attention to the temporal space. Comparing to MUCAN, VidFace handles multiple snapshots all at once and harnesses the spatial and temporal information integrally, thus boost the alignment and aggregation. We also give the visualization results on the above two datasets shown in Fig. 4 and Fig. 6. These qualitative result further verifies that VidFace can actually boost the final face hallucination.

Ablation Study
To assess the importance of various aspects of the VidFace, we run experiments on TUFS-145K under the setting of Deformable convolution-based encoder, T2T transformer-based encoder, RLPE refiner and DeU decoder, deactivating one or a few modules at a time while keeping the others activated. Table 2 reports the face hallucination performance in terms of PSNR under different ablations. To begin with, we compared the deformable and transformer-based backbones. As observed in the table, the employment of global spatio-temporal information in transformer brings significant improvement (0.15 ↑) to PSNR score. Besides, the employment of RLPE and DeU brings about another 0.23 ↑ gain in PSNR. Together with the visualization result of RLPE depicted in Fig. 5, we can conclude that

Conclusion
This work devotes attention to jointly model both spatio-temporal and facial prior information for better video face hallucination with unaligned tiny snapshots. We pinpoint and analyze the shortcomings of previous work and accordingly propose the VidFace. VidFace is a pure-transformer method to address alignment, aggregation and upsampling in an unified scheme. We further curate a large-scale benchmark of tiny unaligned facial video snapshots, dubbed TUFS-145K, which poses more realistic and challenging scenario for existing baselines and hopefully advances the future face hallucination research. To our knowledge, we are the first attempt to propose a unified transformerbased solver tailored for VFH task. Extensive experiments on video face benchmarks show that the proposed method significantly outperforms the state of the arts.