{'fraction_non_alphanumeric': 0.02934187075054459, 'fraction_numerical': 0.014753449072546042, 'mean_word_length': 4.644374068554397, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 2, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Collaborative Neural Rendering using Anime Character Sheets

Drawing images of characters with desired poses is an essential but laborious task in anime production. In this paper, we present the Collaborative Neural Rendering (CoNR) method, which creates new images for specified poses from a few reference images (AKA Character Sheets). In general, the high diversity of body shapes of anime characters defies the employment of universal body models like SMPL, which are developed from real-world humans. To overcome this difficulty, CoNR uses a compact and easy-to-obtain landmark encoding to avoid creating a unified UV mapping in the pipeline. In addition, the performance of CoNR can be significantly improved when referring to multiple reference images, thanks to feature space cross-view warping in a carefully designed neural network. Moreover, we have collected a character sheet dataset containing over 700,000 hand-drawn and synthesized images of diverse poses to facilitate research in this area. Our code and demo are available at https://github.com/megvii-research/CoNR.


Introduction
2D Animation is one of the essential carriers of art reflecting human creativity. Artists commonly use character sheets to show their virtual character design. A character sheet is the image collection of a specific character with multiple postures observed from different views, as shown in Figure 1. It covers all the appearance details and is widely used to assist the creation of animations or their derived media. Moreover, character sheets allow many artists to cooperate while maintaining the consistency of the design of this character.
Drawing a sequence of anime frames is extremely timeconsuming, requiring imagination and expertise. Due to the semantic gap between the character sheet and the desired poses, it is challenging to design a pipeline to draw character images automatically. Some non-photorealistic rendering (Gooch and Gooch 2001) methods can simulate the artistic style of hand-drawn animation. For example, Toon Shading is widely used in games and animation production. However, it currently still requires a very complex manual design to approximate a specific artistic style, which is too complicated for animators and painters.
We formulate the task of rendering a particular character in the desired pose from the character sheet. We consider the Figure 1: Based on the desired pose and the character sheet, CoNR renders a new anime image character sheet as dynamical-sized set to fit a wider range of input situations. Based on this formulation, we develop a Collaborative Neural Rendering (CoNR) model based on convolutional neural network (CNN). CoNR fully exploits the information available in a provided set of reference images by using feature space cross-view dense correspondence. In addition, CoNR uses the Ultra-Dense Pose (UDP), an easy-to-construct compact landmark encoding tailored for anime characters to avoid requiring a unified UV texture mapping (Yao et al. 2019;Yoon et al. 2021;Gao et al. 2020) in the pipeline. It can represent the fine details of characters, such as accessories, hairstyles, or clothing, so as to allow better artistic control. It can also be easily generated with existing computer graphics pipelines to adapt interactive applications. Moreover, we collect a character sheet dataset containing over 700, 000 hand-drawn and 3D-synthesized images of diverse poses and appearances (which will be released). Training on this dataset, CoNR achieves impressive results both on hand-drawn and synthesized images.
To sum up, our main contributions are: • We formulate a new task, rendering 2D anime character images with desired pose using character sheets.
• We introduce a UDP representation for anime characters and collect a large character sheet dataset containing diverse poses and appearances.

Related Work Image Generation and Translation for Anime
Recent years have seen significant advancement in applying deep learning methods to assist the creation of anime. For example, Gao et al.; Zheng, Li, and Bargteil propose to apply realistic lighting effects and shadow for 2D anime automatically; Wang and Yu propose to transfer photos to anime style; Siyao et al.; Chen and Zwicker propose frame interpolation tailored for animation. There are also attempts to produce vectorized anime images (Su et al. 2021), similar to the step-by-step manual drawing process. The generative modeling of anime faces has achieved very impressive results (Jin et al. 2017;Gokaslan et al. 2018;Tseng et al. 2020;Li, Jin, and Zhu 2021;He, Kan, and Shan 2021). The latent semantics of generative models has also been extensively explored (Shen and Zhou 2021). A modified Style-GAN2 model (aydao 2021) can generate full-body anime images, although it still suffers from artifacts because of the high degree of freedom of the human body.

Representation of Human Body
Stick-figure of skeletons (Chan et al. 2019), SMPL vectors (Loper et al. 2015), and heat maps of joints (Cao et al. 2019;Siarohin et al. 2021) are widely-used representations obtained from motion capture system. However, when these sparse representations are used for anime characters (Khungurn and Chou 2016), they face a new series of challenges: noisy manual annotations, unexpected occlusions caused by the wide variety of characters' clothing and body shapes, and ambiguity due to hand drawing. Furthermore, the aforementioned human pose representations only represent human joints and body shapes. However, anime characters often require flexible artistic control over other body parts, such as the fluttering of hair and skirts. These representations cannot directly drive these parts. Human parsers or clothing segmenters (Zanfir et al. 2018;Yoon et al. 2021;Gafni, Ashual, and Wolf 2021;Chou et al. 2021) are robust to the uncertainty of joint positions. However, the provided semantic masks are not informative enough to represent the pose or even the orientation of a person. DensePose (Güler, Neverova, and Kokkinos 2018) and UV texture mapping (Yao et al. 2019;Yoon et al. 2021;Gao et al. 2020), greatly enhance the detail of pose representation on the human body or face by adding a universal definition that essentially unwarps the 3D human body surface into a 2D coordinate system. However, due to the great diversity and rich topology of anime characters, it is difficult to find a general labeling method to unwarp them, which makes existing dense representations still not serve as an off-the-shelf solution for anime-related tasks.
There are also some dense representations of the body, such as CSE (Neverova et al. 2020). CSE addresses the task of detecting a continuous surface coding from an image. Similar "define-by-training" methods are inapplicable in anime creation since CSE can only be used for reenactment currently.

Human Appearance Transfer
Most of works create vivid body motions or talking heads from only one single image (Yao et al. 2019;Gafni, Ashual, and Wolf 2021;Sarkar et al. 2020;Wang, Mallya, and Liu 2021;Yoon et al. 2021;Siarohin et al. 2019;Liu et al. 2019). The learned prior of the human body (Loper et al. 2015), head (Blanz and Vetter 1999), or real-world clothing shape and textures (Alldieck et al. 2019) enables the model to solve ill-posed problems like imagining and inpainting the back view even if only the frontal view is available. However, anime has long been featuring a flexible character design leading to high diversity in clothing, body shapes, hairstyles, and other appearance features. A model trained on a huge dataset (e.g.,, CLIP (Radford et al. 2021)) might be able to encode some popular anime character designs implicitly. Still, it is generally more challenging to establish priors or styles of the anime character domain than the real human domain.
There are also some attempts  to extend the pose transfer task by utilizing SMPL (Loper et al. 2015), a realistic 3D mesh of the naked human body that is based on skinning and blend-shapes, to combine appearance from different views. Using multiple reference images would, in principle, allow the model to follow the original appearance of the given person and better suit the needs of anime production.
Some recent works utilize neural rendering models (Mildenhall et al. 2020) which are trained using photometric reconstruction loss and camera poses over multiple images of 3D objects and scenes. Due to their ray-marching nature and capability to in-paint in 3D, they are promising methods in modeling real-world 3D data (Pumarola et al. 2021;Peng et al. 2021). These methods are not influenced by or depend on prior knowledge other than the object to be modeled. However, they have not yet made much progress in modeling hand-drawn data like anime character sheets, which less follow strict geometric and physical constraints. csecsecsecse

Method Task Formulation
Our formulation is inspired by the tasks about drawing or painting art (Zhang et al. 2009;Zheng, Jiang, and Huang 2018;Huang, Zhou, and Heng 2019;Zou et al. 2021;Su et al. 2021). We consider the character sheet I n ∈ S ref in a whole. A target pose P tar representation is also required in the input to provide rendering directives to the model. The task can be formulated as mapping S ref to target image I tar with the desired target pose P tar : (1) We notice that complicated poses, motions, or characters may require more references in S ref than others, so dynamically sized S ref should be allowed.

Ultra-Dense Pose
A UDP specifies a character's pose by mapping 2D viewport coordinates to feature vectors, which are 3-tuple floats When the anime body changes its pose, the landmark at the corresponding body part will remain the same. Compared to existing dense human-pose representations, UDP bypasses the step of unwarping 3D surfaces onto a 2D UV texture mapping, which may not be done in a consistent method for anime characters that continuously and consistently encode body surfaces. In this way, a UDP can be represented as a color image P tar ∈ R H×W ×3 with pixels corresponding to landmarks L (x,y) ∈ R 3 . Non-person areas of the UDP image are masked. It allows better compatibility across a broader range of anime body shapes and enables better artistic control over body details like garment motions.
3D meshes are widely used data representations for anime characters in their game adaptations. Vertex in a mesh usually comprises corresponding texture coordinate (u, v) or a vertex color (r, g, b). Interpolation over the barycentric coordinates allows triangles to form faces filled with color values or pixels looked up from textures coordinates.
Taking a bunch of anime body meshes standing at the center of the world, we ask them to perform the same A-pose (a standard pose) to align the joints. To construct UDPs, we remove the original texture and overwrite the color (r, g, b) of each vertex with a landmark, which is currently the world coordinate (x, y, z), as shown in Figure 2. When the anime body changes its pose, the vertex on the mesh may move to a new position in the world coordinate system, but the landmark at the corresponding body part will remain the same color.
To avoid the difficulty of down-sampling and processing meshes, we convert the modified meshes into 2D images, which are compatible with CNNs. This is done by introducing a camera, culling on occluded faces, and projecting only the faces visible from the camera into an image. The processed UDP representation is a H × W × 4 shaped image recorded in floating-point numbers ranging from 0 to 1. The four channels include the three-dimensional landmark encodings and one-dimensional occupancy for indicating whether the pixel is on the body.
Three properties of UDP could alleviate the difficulties when creating images of 2D animation: 1) UDP is a detailed 3D pose representation since every tiny piece of surface on the anime body could be automatically assigned with a unique encoding without hand-crafted annotations.
2) UDP is a compatible pose representation since the anime characters with similar body shapes will also get out-fits that are consistently pseudo-colorized.
3) UDP can be obtained directly in all existing 3D editors, game engines, and many other up-streams.

Data Preparation
As character sheets used in the anime-related industries are not yet available to the computer vision community, we built a dataset containing more than 20, 000 hand-drawn anime characters by selecting human-like characters from public datasets (Anonymous, community, and Branwen 2021; Jerry 2017). We manually perform matting to remove the background from the character with the help of the watershed algorithm. We also construct a 2D background dataset containing 4000 images with a similar method. Manually annotating hand-drawn anime images with UDP involves significant hardship. To alleviate the problem of label scarcity, we further construct a synthesized dataset from anime-styled 3D meshes.
Finally, We combine the synthesized dataset with the hand-drawn dataset to obtain both high-quality UDP labels and high diversity of hand-drawn styles. We randomly split the whole dataset by a 1/16 ratio into the validation and training sets. The split is on a per-anime-character basis, so the validation set contains characters unseen during training. The whole dataset contains over 700, 000 hand-drawn and synthesized images of diverse poses and appearances. We manually exclude content that is not suitable for public display. Random characters with a random background are shown in Figure 3.

Collaborative Neural Rendering
Overview. We utilize a collaborative inference for a convolutional neural network named CINN, inspired by Point-Net ) and Equivariant-SFM (Moran et al. 2021). CoNR consists of a CINN renderer and an optional Figure 4: Inference pipeline of CoNR. Reference images I 1 · · · I n ∈ S ref from the input character sheet are fed into CoNR using modified U-Nets (Ronneberger, Fischer, and Brox 2015) as sub-networks. The same UDP representation P tar is resized and concatenated into each scale of the encoder outputs in all sub-networks. Blocks with the same color share weights. "D1 to D4" refers to 4 blocks of the decoder. The sub-networks form a fully connected graph using cross-view message passing. Each block will receive the averaged message from corresponding blocks in all other sub-networks UDP Detector. Figure 4 shows the pipeline of the proposed approach. CoNR generates a character image of the desired pose, taking the UDP representation P tar of the target pose and a character sheet S ref , as the inputs. When generating more than one pose, we feed different UDPs and keep the same reference character sheet.
The input UDP representation can be produced by a UDP Detector from reference images or videos. For interactive applications like games, the existing physics engine can be used as a drop-in replacement for the UDP Detector to compute body and cloth dynamics for the anime character directly. Renderer. We apply the following modifications to the U-Net (Ronneberger, Fischer, and Brox 2015): 1) As shown in Figure 4, we rescale and concatenate UDP to each block of the decoder. This input strategy aims to reuse the encoder's results and further allow high efficient inference when there are multiple target UDPs when generating an animation video of a character. 2) Due to the spatial misalignment of the local and the remote branches, we use flow fields to align the features. Specifically, as inspired by (Dai et al. 2017;Dosovitskiy et al. 2015), we approximate a flow field as two channels and warp the features of other channels according to the estimated flow. This operator enhances the long-range look-up ability for CINN. 3) We use the CINN method in the decoders of the network. We split the original up-sampling output feature channels by half, one as the remote branch and the other for the local branch. First, we warp the features on the remote branches to align with the local features. Then the output features from all remote branches are averaged and concatenated with the encoder output, then fed into the next block (illustrated in Appendix). The last decoder block will collect averaged output features from all previous decoder blocks in all branches and output the final generated image I tar . UDP Detector. While training with the synthesized dataset, UDP P tar can be directly obtained. As for the hand-drawn dataset, we design a UDP Detector to estimate UDPs P tar from hand-drawn images. The UDP Detector is a simple U-Net (Ronneberger, Fischer, and Brox 2015) consists of a ResNet-50 (R50) (He et al. 2016) encoder and a decoder with 5 residual blocks. It is trained jointly with the renderer in an end-to-end manner. We share the weight of the renderer's encoder and UDP Detector's encoder.

Experiments Training Strategy
We train CoNR models of m sub-networks (views) on our dataset. To create one training sample, we randomly select a character and then randomly select m + 1 arbitrary poses. These images are split as m image inputs (character sheet) I 1 · · · I m ∈ S ref and one image of the target pose as the ground truth of CoNR's final output.
We paste them onto k random backgrounds and feed them into the UDP Detector. We use the average of the k UDP detection results P i of the same target pose, P tar = 1/k k i=1 P i , as the final UDP detection results. We compute losses at both the output of the detector and the end of the CoNR pipeline. We use L1 loss on the landmark encodings and binary cross-entropy (BCE) loss on the mask to train the detector if the ground truth UDP P GT tar is available: L mask = BCE(sgn( P tar ), sgn(P GT tar )), where sgn indicates that a point is from the body surface or background. We use a consistency loss L cons by computing the standard deviation of k UDP Detector outputs:  Table 1: Comparison on the number of input reference images. We use character sheets of m reference images to train the CoNR model, and then use character sheets of n reference images to evaluate the trained model The UDP Detector and renderer are trained end-to-end simultaneously. The total loss function is the sum of all these losses: where the hyper-parameter α of feature loss is from previous work (Zhang, Ng, and Chen 2018). Our model is optimized by AdamW (Loshchilov and Hutter 2019) with weight decay 10 −4 for 224K iterations. We choose m = 4, k = 4 during training unless otherwise specified. The training process uses a batch size of 24 with all input resolutions set to 256 × 256. Model training takes about three weeks on four GPUs.

Result
For quantitative evaluation, we measure the losses on the validation split. Note that L udp and L photo direct measure the L1 distance between predictions and ground truth. Additionally, we use LPIPS  to measure the perceptual image quality. Inference Visual Effects. CoNR can cope with the diverse styles of anime, including differences between syn-Character Sheet UDP Generated Figure 6: Use UDP Detector and CoNR to transfer appearance based on the character sheet thesized and hand-drawn images. Figure 5 list some random CoNR outputs on the validation split. With the same character sheet input S ref , we replaced the provided UDP P tar , and the results of CoNR changed accordingly. Notably, CoNR goes beyond a naïve correspondence by absolute UDP value (transfer between different cloths). CoNR, as a neural rendering method, can also be used to achieve results similar to what style transfer methods could provide when running a textured image at the reference pose through the UDP Detector. As shown in Figure 6, UDP detection results of one character can be used to render another character. The CoNR inference pipeline for the anime (or game) production will usually be without the UDP Detector. Effectiveness of the Collaboration. Unlike most previous methods for similar tasks that allow only one reference image, CoNR uses a dynamically-sized set of reference inputs during training and inference. Table 1 shows that using an additional number of views m > 1 during training will enhance the quality of generated images. On the opposite, removing images from the character sheet will reduce the coverage of the body surface. When CoNR is trained with m = 1, CoNR can not provide a reasonable solution. For example, given the character's backside, it is hard to imagine the frontal side. In this case, the photo-metric reconstruction and feature losses may encourage the network to learn a wrong solution. Therefore even if enough information in the n > 1 images is provided during inference, the network may not generate the target image accurately. Similarly, keeping m = 4 while reducing the number of inference views (n = 1) will also harm the quality of CoNR.
As shown in Figure 7, CoNR can leverage information distributed across all images to produce better visual effects. This allows CoNR to scale from image synthesis, when only a few shots of the character are given, to image interpolating when a lot of shots are available. The example in Figure 7 shows the behavior of CoNR when it does not have enough information to draw missing parts correctly. Even if very few reference images are provided, the target pose that is similar to some reference will be accurate. Furthermore, users may iterate on the results and feed them back into CoNR to accelerate anime production.

Comparison with Related Work
As CoNR provides a baseline for a new task, we admit that direct comparison with related works can either be impos- Figure 7: Effectiveness of the collaboration. We perform a reconstruction experiment with a video in the wild. The last row shows 8 ground truth frames I GT i ∈ S GT vid from a video. In this experiment, the UDPs are estimated from ground truth frames by a trained UDP Detector. The first two rows show the input and output images of CoNR. The used subsets of character sheet S ref ⊂ S GT vid are marked using the blue background. Generated images for novel poses are marked using the red background sible or unfair. We still try to include comparisons, only to show how our task is related to other existing tasks. Human Pose Synthesis. We compare the results produced by CoNR to a real-world digital human system  using the same target poses as used in their demo. We use two character sheets 1 with both pose and appearance unseen during training. One contains the same 4 images shown in Figure 1, the other character sheet taken from a random video from the internet as used in Figure 7. Figure 8 indicates that the long skirt prevents a high accuracy estimation of the leg joints and that parametric 3D human models like SMPL may not handle the body shape of anime characters correctly. Further diagnosis shows that parametric 3D human models may not handle anime's diverse clothing and body shape, as shown in Figure 2. CoNR can produce images at desired target poses with better quality. Image synthesis pipelines starting with human pose representations are theoretically inapplicable on anime data, as the character's diverse body structure, clothing, and accessories cannot be reasonably represented. Using human pose synthesis methods in anime, we may lose artistic control over garments and fine details, which is crucial in anime creation workflows. Style Transfer. Style transfer typically refers to applying a learned style from a certain domain (anime images) to the input image taken from the other domain (real-world images) (Chen, Liu, and Chen 2019;Wang and Yu 2020b). The models usually treat the target domain as a kind of style and require extensive training to remember the style. Some methods use a single image to provide a style hint during the inference. For example, one could use Swapping Auto 1 One anime character is from www.youtube.com/watch?v= m6k t8yEyvE and another character is illustrated by an amateur artist.

Setting
112K   Encoders (Park et al. 2020), a recent style-transfer method to swap the textures or body structures between two characters. Although the model has a lot of parameters (3× the size of CoNR), our comparison shows that it is still not enough to remember and reproduce the diverse sub-styles of the textures, pose, and body structure in the domain of anime.

Ablation Study
We perform ablation studies on the UDP Detector, renderer, and loss functions. Table 2 shows that UDP representation can be inferred from images using a U-Net (Ronneberger, Figure 8: Comparison with the SMPL-based method. We compare the results of our method with the results of Liquid Warpping GAN (LWG) ) when trying to resemble the target poses. Note that CoNR uses 3D model editor software to generate pose-related UDPs based on the real person. And LWG uses the real person images as input.
The short skirt of the second character is tightened on the legs instead of sagging naturally Fischer, and Brox 2015). An original U-Net, which takes a concatenated tensor of 4 reference images and the target UDP as the input, does not provide acceptable results on this task, as shown in Table 3. The proposed CoNR method with both the feature warping and the CINN method significantly increases the performance, thus establishing a stronger baseline for the proposed task. We further ablate the loss functions in Table 4.

Limitations
CoNR is unable to model the environmental lighting effects. The inputs of CoNR can not provide any information about the environmental or contextual information that could be utilized to infer lighting effects. Users may have to look for sketch relighting techniques (Zheng, Li, and Bargteil 2020;Gao et al. 2018;Su et al. 2018).
CoNR is unable to model the dynamics of the character. The CoNR model accepts target poses detected from a video of a character with a body shape similar to S ref , and could inherit the dynamics. However, it requires such a pose sequence beforehand. To bypass the UDP Detector, we can rely on additional technologies like garment cap- Figure 9: Comparision between detection results of handdrawn anime character images using OpenPose (Cao et al. 2019), the UDP Detector and SMPLify (Bogo et al. 2016). The images are from the validation split of the hand-drawn dataset (Anonymous, community, and Branwen 2021) with the ID number. The detected SMPL body can not fully handle the diverse body shapes of anime characters. UDP Detector produces relatively reasonable results

Setting
224K iter   Tiwari et al. 2021) or existing 3D animation workflows to obtain a synthesized UDP P tar . CoNR focuses on the rendering task. Obtaining a suitable 3D mesh with all the body parts rigged and all clothing computed with proper dynamics is beyond the scope of this paper. Using P tar from an inappropriate body shape may lead to incorrect CoNR results. The dataset may not fully follow the distribution of anime characters in the wild. The collected dataset contains only human-like anime characters from 2014 to 2018. As the character meshes are aligned using joints, the models trained on this dataset may not be applied to animal-like characters. Research on broader datasets will be the future work.

Conclusion
In this paper, we explore a new task to render anime character images with the desired pose from multiple images in character sheets. We develop a UV-mapping-free method, CoNR achieving encouraging effectiveness. We show the potential of this data-driven method in assisting animation art. We hope the method and the datasets presented in this paper will inspire our community and further researchers.

Experiment Details
The synthesized part of the dataset contains multiple-pose and multiple-view images. Each of them is paired with UDP annotations. A randomly-sampled subset of 3D mesh data obtained in the same way as in (Khungurn and Chou 2016) was used in this work, which contains 2, 000 anime characters with more than 200 poses. The scope of permissions was confirmed before using existing assets in this work. The contribution in this work is mostly about years of manual data clean-up, which includes converting mesh formats, restructuring the bone hierarchy, and aligning the A-poses before synthesizing RGB images for each pose, and their corresponding UDPs with MikuMikuDance (MMD) software.
The mixed dataset also contains unlabeled hand-drawn samples and one-shot samples, i.e. characters with the only single available pose, which are randomly sampled from existing datasets (Anonymous, community, and Branwen 2021; Jerry 2017). We make full use of the mixed data by performing semi-supervised learning. For the detector, we skip the L udp as we do not have ground truth but keep the L cons to ensure the detector makes the same prediction of the same character under k random augmentations. As for the renderer, we have to reuse the image of the same pose multiple times to fill all the m views if the provided images in the character sheet are not enough. Random-crop augmentation is applied to I m ∈ S ref ,P tar and I gt tar . With the random-crop augmentation, in order to generate the image of body parts in the right position, CoNR has to learn a cross-modality feature matching instead of a trivial solution by simply selecting one of the provided images as the output.

Ablation on Message Passing
We perform ablations on the number of messages passing for the CoNR. More ghosting can be observed when subnetworks communicate less than three times.

Comparison with Style Transfer method
To best of our knowledge, fusion of pose and texture for virtual character cannot be solved directly by implicit style transfer. As shown in Figure 11, SwapAE fails in this scene. Figure 10: Sub-network structure of CoNR. Every input I n ∈ S ref goes into a U-Net shaped sub-network. The warping and averaging operation are performed on the output of every decoder block. The semi-transparent blocks represent corresponding blocks in all other sub-networks. Blocks with the same color share weights. Each block in a sub-network would pass part of the outputs as messages to corresponding successive blocks in all other sub-networks, in addition to forwarding outputs to its following blocks in the current sub-network. The path of UDP is omitted.