{'fraction_non_alphanumeric': 0.03082741233096493, 'fraction_numerical': 0.010696004278401712, 'mean_word_length': 4.662773091066407, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
MV-TON: Memory-based Video Virtual Try-on network

With the development of Generative Adversarial Network, image-based virtual try-on methods have made great progress. However, limited work has explored the task of video-based virtual try-on while it is important in real-world applications. Most existing video-based virtual try-on methods usually require clothing templates and they can only generate blurred and low-resolution results. To address these challenges, we propose a Memory-based Video virtual Try-On Network (MV-TON), which seamlessly transfers desired clothes to a target person without using any clothing templates and generates high-resolution realistic videos. Specifically, MV-TON consists of two modules: 1) a try-on module that transfers the desired clothes from model images to frame images by pose alignment and region-wise replacing of pixels; 2) a memory refinement module that learns to embed the existing generated frames into the latent space as external memory for the following frame generation. Experimental results show the effectiveness of our method in the video virtual try-on task and its superiority over other existing methods.


INTRODUCTION
Virtual try-on has become an essential task that allows users to try on the desired clothes obtained from social media or shopping websites [37], which implies such a task can be seen as a type of e-commerce application that offers the convenience of shopping. Currently, with the growth of short video technology, generating high-quality videos of a person wearing desired clothes is of vital importance.
Previous virtual try-on methods usually focus on image-based operations [6,10,33], which are able to try on clothing templates for a target person. Clothing templates are clean product images that do not contain any people, while model images are images of models who wear desired clothes. However, directly applying the image-based methods to video generation may cause the problem of lacking smoothness between the adjacent generated frames. As each frame is regarded merely as an image, the frames of videos cannot mutually facilitate each other. Although [5] utilizes optical flow to warp the past synthesized frames for following frame generation, it restricts the number of frames to use and overly relies on the results of the preceding generated frames, which may lead to texture misalignment and blurred results. In addition, the requirements for clothing templates in [5] are typically unavailable and inaccessible.
In order to address the challenges mentioned above, we propose a two-stage Memory-based Video Virtual Try-On Network (MV-TON). As illustrated in Figure 1, given a source video and a model image, MV-TON is capable of synthesizing realistic videos that transfer the desired clothes from the model image to the target person in each frame of the video. Specifically, with an intrinsically learned 3d appearance flow, the key idea of the first stage is to employ pose transfer on the model image and further transfer the corresponding regional pixels from the deformed model image to the frame image using a region-wise replacing block in terms of the individual frame.
However, it is difficult for the first stage to deal with complex scenarios and large deformations due to the non-rigid nature of human bodies. Inspired by the external memory which is normally used in video semantic segmentation [21,25], we propose a memory refinement module in which pixel locations are selected from multiple frames to reconstruct space-time information, which can simultaneously retrieve missing pixels and correct wrong pixels. Still, to the best of our knowledge, this work is the first attempt to apply frame aggregation through a memory mechanism to video synthesis. Without additional masks, a dedicated encoder is used to embed the frames generated from the first module into key and value feature maps. To be more specific, key maps address the similarities between the current frame and the past generated frames, as value maps determine the pixels when-and-where to retrieve. Finally, the key and value maps stored in external memory go through a memory read block to reconstruct the intermediate frames. Besides, we employ a flow consistency loss [17] so that the network can achieve spatio-temporal smoothing during the synthesizing.
To summarize, our MV-TON network comprises two essential modules: 1) a try-on module is responsible for transferring the desired clothes to the target person in each frame of the given video by applying pose alignment and replacing specified pixels, so as to keep prominent appearance details of the person; 2) a memory refinement module aims to combine space-time information of multiple frames with an aggregation mechanism to refine the existing generated frames.
Due to the insufficiency of genuine paired training data (person videos equipped with the desired clothes), lots of virtual try-on methods [10,33] separate pseudo paired data into various representations to reassemble them, with the input being exactly the same as the output. How to leverage the unpaired data ignored in most approaches is non-trivial. Although the unpaired data has a different appearance with model images, we aim to explicitly control the similarity between their clothes. To this end, we design a novel matching discriminator that enforces the clothes of generated frames to be similar to the clothes of model images.
We note that the evaluation metrics adopted in most virtual try-on tasks do not consider their peculiarities, regarding them as normal image synthesis tasks. Thus we propose a novel evaluation matrix Cycle Transfer Score (CTS). Specifically, obtained an initial output video, we apply the try-on method again to recover clothes from the generated frames to the model images. A robust model should get the low 'transfer cost' in the repetitive transformation.
Our contributions can be summarized as below: • A two-stage framework is carefully designed to overcome the difficulty of dealing with a large variety of poses in video virtual try-on. The proposed memory refinement operation improves the details by automatically correcting what is wrong or missing in the initial results. The optical flow of the generated videos is regularized by a flow consistency loss. • We propose a matching discriminator that leverages the unpaired data to better train the model with a hybrid training strategy. • We propose an evaluation metric Cycle Transfer Score (CTS) specially designed for virtual try-on tasks to evaluate the objective quality of generated videos.

RELATED WORKS
Person Image synthesis. With the development of Generative Adversarial Networks (GANs) [8], person image synthesis has made significant progress through adversarial learning. Ma et al. [20] proposed a disentangled representation to generate person images with arbitrary poses. [1,13,19,30] utilized conditional Generative Adversarial Networks (cGANs) [23] to translate skeleton poses into person images. [19] adopted a variant of the conditional DCGAN [27] architecture that conditioned both on a reference image and a specified pose. Li et al. [18] proposed a network that learns the mapping from an input skeleton pose to a target so as to guide the feature warping. In contrast to the above methods, this work handles the transformation with an intrinsic 3D appearance flow since the identity mapping is hard to learn directly from images. Virtual Try-on. 3D try-on methods are mostly based on computer graphics. Guan et al. [9] first allowed 3D human bodies in any pose to be dressed in 2D clothing. Sekine et al. [29] proposed a virtual fitting approach for adjusting 2D clothing images on users by modeling their body shapes from depth images. The principles of [26] were used to capture the garment geometry in motion on 3D models. However, to avoid the time-consuming raised by 3d model rendering, [6,10,12,33,39] directly synthesized a photo-realistic image by transferring the desired clothes onto the person, which is computationally efficient. Han et al. [10] used shape context to align features through clothing-agnostic person representations. Wang et al. [33] described an improved way of [10] which learns to estimate the transformation parameters. Dong et al. [6] proposed a framework that adopts decomposed binary masks to predict human parsing maps Wu et al. [35] so as to guide the generation of person images. In contrast to the above methods requiring clothing templates, Wu et al. [36] directly used available model images from which clothing pixels could be sampled and transferred to the and an arbitrary frame image , MV-TON can achieve the pose transformation according to their poses extracted by a pose estimator to get the warped image . Then, the transferred imageˆcan reserve the appearance of except for clothing items and arms to fuse with the clothes extracted from . The results generated from stage I are further refined by the second stage that consists of a memory refinement module which memorizes the features of past frames to strengthen the current frame.
person's images. Two branches were applied separately in [28] to disentangle the body shape and appearance information, which can be reassembled as desired. There are few explorations in the videobased virtual try-on. Dong et al. [5] exploited optical flow from pose sequences to warp the preceding frame and used a warping module to warp clothes simultaneously. Nevertheless, [2] determines the upper limit of the correctness of the flow used in [5], and this method didn't take full advantage of whole frames since it utilized a few past frames to synthesize the future frame.
Memory Mechanism. Many previous methods for document Q&A [16,22] embedded memory information into key and value feature vectors individually. Recently, memory mechanisms have been applied in diverse domains. For instance, Yang and Chan [38] proposed a dynamic memory network to retrieve the related template for matching objects in the visual tracking task. Na et al. [24] first introduced a network using multi-layer convolutions applied to writing and reading in the movie understanding task. Oh et al. [25] utilized a space-time memory mechanism to store informative knowledge.

METHOD DESCRIPTION 3.1 Problem Formulation and Notations
Formally, given a person video V = { } =1 , where N is the number of the frames of the video, and a model image , we aim to generate a spatial-temporal coherent realistic video O, the person of which wears the desired clothes the same as . Let P={ } =1 denote the pose sequence extracted by the Human Pose Estimator [3], which estimates 18 joints of each human body.
means the pose of . Our pipeline is a two-stage architecture designed for individual frame try-on and multiple frame aggregations. Our goal is to learn the mapping of ( , , , ) →ˆwith respect to the i-th frame and furtherˆ→ O whereˆ= {ˆ} =1 .

Generator
3.2.1 Try-on Module. The try-on module aims to transfer the specified region-wise pixels from model images to target frames, including clothes and arms, which reserves the remaining appearances of the person to the greatest extent possible.
As depicted in the upper of Figure 2, an appearance encoder and a pose encoder are used to encode and in multi-scales for the current processing frame, respectively. Inspired by [18], we estimate the intrinsic 3D appearance flow between ( , ) by modeling SMPL [15] in the 3D space for human pose transfer. The parts of self-occlusion and out-of-plane are distinguished from the image through learning a visibility map. With a spatial transformer layer, the flow warps the feature maps at different scales obtained above and then divides them into exclusive regions according to the visibility map. The warped feature maps are decoded to construct the image which has the same pose as . We subsequently introduce a region replacing block to transfer the specified regionwise pixels from to . Region Replacing Block. With the warped model image in the target pose , we use the pre-trained CIHP [7] to generate the human parsing map S and subsequently convert S into 20 channel heat maps M ∈ 20× × . Each channel is a binary mask and represents one class of person's parts. To extract the clothing and arm pixels, we train a Region of Interest (RoI) generation network that considers a full body mask M as pseudo ground truths. The where ⊙ indicates element-wise multiplication for each pixel, and ⊕ means channel dimension concatenation. ▽ ( ) is an encoderdecoder network aiming to fit two individual parts into one. This module can be used to transfer not only clothes but also other components of dressed people. We observed that faces were prone to losing during the generation [5,18]. It can be solved through replacing the facial region M ∈ 3× × from the ultimate synthesized imageˆto the corresponding pixels of . contains faces, necks, and hair. Our merged component can be expressed as below: In our experiments, the fitting network ▽ ( ) can fit two arbitrary regions into a realistic-looking image with only one training.

Memory Refinement
Module. Our task needs to handle a more challenging scene in which the poses of people are varied, while image scenes merely consider a fixed pose. Therefore, we design a Memory Refinement Module (MRM) that alleviates pixel misalignment and loss by selecting space-time pixel locations to reconstruct the current frame.
As shown in the lower of Figure 2,ˆ= {ˆ} =1 obtained by try-on module are separated into the past frames and the current frame. Each past frame and the current frame are fed into an encoder network that shares weight to embed the visual appearance into the latent space. The latent representations are subsequently separated into key and value maps respectively for reading and writing via a memory read block. Besides, if there exists more than one past frame, their key and value maps are concatenated along the temporal dimension as external memory. Key maps focus on addressing the correspondence between aggregated past frames and the current frame. The generated pixels which have inaccurate values are always prone to suppression during the synthesizing since they have low similarities with others. As high frequency details of the appearance are sometimes lost, the value maps of the current frame reconstruct the pixels at feature level, indicating that it is necessary to provide the value maps of the past frames to compensate.
Memory Read Block. As shown in Figure 4, inspired by [25], we propose a memory read block. However, instead of using masks and frames together in [25], we take the video frames as input without additional masks. To be more specific, this block takes , , , as input. and are the aggregations of the key and value maps of the past frames, respectively. and are the encoded key and value maps of the current frame, respectively. The similarity is calculated by comparing every space-time pixel location of the memory key maps with every spatial pixel location of the current key map. The similarity function f is defined as follows: where i and j are the indexes of the spatial pixel locations. denotes the embedding space and (·) denotes dot-product. To avoid the variable input size, a normalizing factor 1 is necessary, Z can be denoted as: Along the dimension j, it's the same as softmax function by applying 1 (k , k ).
The overall read operation can be summarized as: where [·, ·] denotes the concatenation.
With the final output of the memory read block, a standard decoder is used to yield the ultimate refined image. Figure 3 validates the effects of memory refinement module. We can see that this module is indeed able to correct missing and wrong pixels that can not be recovered well by stage I.
Hybrid Training. We cannot acquire paired training data where people wear different clothes but in the exact same pose. Generally, the image itself serves as paired training data ground truth in most image-based virtual try-on methods [6,10,33]. Training paired and unpaired data simultaneously has proved useful in image synthesis [32,36]. Hence, we take a similar strategy in the memory refinement module.

Matching Discriminator
Although hybrid training has the potential to contribute towards better results, there are relatively few approaches to supervising unpaired data. Hence, we proposed a novel Matching Discriminator (MD) to force the generation to focus on clothing, which constrains the correlation maps between the specified regions extracted from model images and from transferred images.
As shown in Figure 5, the corresponding masks generated by the RoI generation network are sent to extract the clothes to compute correlation maps. Thereafter, for each space-time location, we sample the max value along the dimension. Current discriminators [4,34] have been explored on different scales. However, it is hard to incorporate the correspondence between specified regions since the convolutional operation has an effect on whole feature maps. In contrast, we compute an adversarial loss through correlation maps without direct supervision. If the clothes of the synthesized frames are similar to the model images, the result of each patch on the correlation map will be closer to 1, while the opposite will be closer to 0. To make the training more realistic, we randomly sampled two frames of the original video as inputs to calculate their clothing correlation maps.
In more detail, we brief the correlation map as U in the below. Real U is computed as: where ⊙ denotes element-wise multiplication for each pixel and ⊗ indicates dot-product. and are the different frames of the original video V. and ′ are the regional masks, respectively. Max [· ⊗ ·] is a function that aims to extract the max value along the specified dimension. Conv indicates convolution layers. Fake U is obtained as: where ′ is the i-th frame of the synthesized videos O. is the model image.ˆandˆ′ are the regional masks.
The effects of the MD are visualized in Figure 6 (d)(e). When MRM is applied to our model without MD, it may lead to a blurred texture of the results. The matching discriminator enables improved control over the small details.

Training Losses
We brief the try-on module as TM and the memory refinement module as MRM. In TM, our goal is to generate photo-realistic images with the desired clothes. Specifically, we apply a vanilla adversarial loss L [8], a L1 reconstruction loss L 1 and a perceptual loss L [14] to constrain pose alignment. L is used to precisely classify the semantic label of each pixel. The overall objective function L can be computed as: where hyper-parameters 1 , 2 , 3 , 4 are the weights of losses. In our experiments we set 1 = 4 =0.01, 2 = 3 =1. In MRM, our goal is to enhance frame consistency and refine the initial results generated from TM. We employ a simplified flow consistency loss [17] to guide the synthesized videos to keep consistency, which defines a flow field F from the preceding frames to the current frame. F represents the opposite direction. L can be defined as: where p denotes a pixel location. M is the foreground mask of the frame. N is the number of the foreground pixels in M. We show the effects of flow consistency loss in Figure 7, which enables the generated videos to better keep consistency.
For paired data, inspired by Pix2PixHD [34], we adopt a multiscale GAN to calculate the adversarial loss L . Besides L and L , we also apply a L2 mean squared Loss L 2 , L 1 and L to tackle the challenge of accurate synthesis. The overall objective function L can be computed as: where hyper-parameter , i∈ {1, 2, 3, 4, 5} are the weights of losses.
In our experiments we set 1 = 2 = 5 =1, 3 = 4 =0.1. With the matching discriminator described in Sec.3.3, L ℎ can be formulated as below: where V denotes the original videos andˆindicates the videos generated by TM. means the model images. The overall objective function L can be calculated as: where hyper-parameters 1 , 2 are the weights of losses. In our experiments we set 1 =0.1, 2 =1.

CYCLE TRANSFER SCORE
Among the existing methods, the most widely used evaluation metrics are designed for ordinary synthesis tasks, which ignore the characteristics of virtual try-on. Performing transfer in a cyclic manner, the clothes should keep most of the details so that the model can prove robust in the challenge of pose transformation. In light of this, we propose a novel evaluation metric Cycle Transfer Score (CTS) specially designed for virtual try-on tasks. Figure 8 demonstrates a cyclic transfer process. First, we apply the target virtual try-on method to transfer an arbitrary component from the model image M to the original image F. Then we apply the same method to perform the transfer in an opposite direction -we transfer from image F to image M to generate M . The result M is expected to have a similar appearance as M. We measure the difference between M and M as the cycle transfer score to reflect the level of cyclic consistency for a given virtual try-on method. In particular, any kind of pixel level or feature level loss function can be used to measure such a difference, such as mean squared loss and perceptual loss. In our experiment, we compute CTS through Fréchet Inception Distance (FID) [11] which calculates the mean and covariance matrix of features to evaluate image quality.
Notably, besides virtual try-on, CTS can be widely applied to tasks which are desired to transfer the components of images, in order to evaluate the effectiveness of relevant methods.

EXPERIMENTAL RESULTS
Implementation Details In stage II, we sample five frames to train due to the limited memory and increase the maximum skipping by five every 20 epochs. We utilize a stochastic gradient solver in two stages with a fixed learning rate of 0.0002 and an adam optimizer ( 1 =0.5 , 2 =0.999). The batch size of stage I and stage II are 8 and 2, respectively. All the experiments were conducted on 2 Nvidia TITAN Xp GPUs.
Dataset We trained and evaluated our MV-TON on the VVT dataset proposed by [5], which contains 791 videos of fashion models on the catwalk. We also selected 661 videos for training and 130 videos for testing. Specifically, for stage I, we randomly picked 30  person images of each video in different poses and further combined two of them as paired data, which adds up to 19830 pairs of 661 training videos. In addition, we also picked unpaired images of different videos that have the same number as paired images. For stage II, we crawled 661 person images of whole training sets acted as model images, of which each model image corresponds to one video.

Baseline Methods
We conduct a comparison with the following baseline methods to prove the effectiveness of our proposed MV-TON network. The Characteristic-Preserving Virtual Try-On Network [33] (CP-VTON) fits the desired clothes onto the target person. Since the clothing template is not provided, we use an RoI (Region of Interest) generation to extract the clothes from model images. Model to Everyone (M2E-TON) [36] is similar to our approach, while the former is designed for a fixed pose. We adapt CP-VTON and M2E to our task by assuming each frame to be their user image. Flow-navigated Warping GAN (FW-GAN) [5] warps the past frames and clothes separately to synthesize the video where the person wears the desired clothes. Similarly, we adopt the RoI generation to extract clothes from model images as the clothing template. Finally, we retrain these approaches on the VVT dataset for a fair comparison.

Qualitative Results
We show some results of our method in Figure 9 and more video results are available in Supp. Figure 10 presents a visual comparison with the mentioned baseline methods. Compared with other baseline methods, our approach can better preserve both clothes and human appearance. Figure 11 shows the generated intermediate results from different stages of the MV-TON network. Figure 12 shows some failure results of our method caused by the irregular shape of clothes.

Quantitative Results
CTS. Following [5], we choose FID as a measured function, which uses the pre-trained inception model [31] as a feature extractor to exploit desired vectors of the output after a final average pooling layer. Specifically, it computes the mean and covariance matrix Σ for the feature vectors from the original model images. It also calculates the same statistics˜andΣ for the feature vectors from the synthesized model images. Then the FID can be computed as || −˜|| 2 + (Σ + Σ − 2 √︁ Σ Σ). The lower number indicates better performance in Table 1, demonstrating that our method significantly outperforms the other baseline methods. It also indicates that our method is able to preserve clothing textures in repeatedly synthesizing them.
User Study. We conduct a user study with 30 users to evaluate the quality of the synthesized videos generated by our method

Model
User Study% CP-VTON [33] 4.27 M2E-TON [36] 32.56 FW-GAN [5] 21.02 Ours 42.15 Table 2: User study(%)on VVT dataset. The second column represents the wining percentage in the comparison test. and other methods, which use the same reference model image and video sequences. The user is asked to pick one video with better quality of each video group. Meanwhile, the reference model images are shown to make users aware of clothing ground-truth. All samples in the user study are randomly selected from VVT test sets and shown with full resolution. The comparison results of different methods are shown in Table 2. We can see that the proposed method achieves the best results.

CONCLUSION
In conclusion, we have presented a network that consists of a try-on module and a memory aggregation module. Our approach transfers the clothes from model images to the target person in videos without using any clothing templates. Extensive evaluations demonstrate our approach performs the best among the existing videobased methods in both qualitative and quantitative. Furthermore, we believe that the proposed approach can be potentially adapted to other video synthesis tasks.