{'fraction_non_alphanumeric': 0.02997814629129708, 'fraction_numerical': 0.01493765265458285, 'mean_word_length': 4.8428721646387265, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Speech Fusion to Face: Bridging the Gap Between Human's Vocal Characteristics and Facial Imaging

While deep learning technologies are now capable of generating realistic images confusing humans, the research efforts are turning to the synthesis of images for more concrete and application-specific purposes. Facial image generation based on vocal characteristics from speech is one of such important yet challenging tasks. It is the key enabler to influential use cases of image generation, especially for business in public security and entertainment. Existing solutions to the problem of speech2face renders limited image quality and fails to preserve facial similarity due to the lack of quality dataset for training and appropriate integration of vocal features. In this paper, we investigate these key technical challenges and propose Speech Fusion to Face, or SF2F in short, attempting to address the issue of facial image quality and the poor connection between vocal feature domain and modern image generation models. By adopting new strategies on data model and training, we demonstrate dramatic performance boost over state-of-the-art solution, by doubling the recall of individual identity, and lifting the quality score from 15 to 19 based on the mutual information score with VGGFace classifier.


Introduction
Driven by the explosive growth of deep learning technologies, [1,2,3], computer vision algorithms are now capable of generating high-fidelity images, such that humans can hardly distinguish real images from synthesized ones. Researchers in computer vision and other areas are seeking concrete applications to fully unleash the power of image synthesis. One of the most promising applications is the automatic generation of facial images based on the vocal characteristics extracted from human speech [4,5], referred to as speech2face in the rest of the paper. It is believed to be the key enabler to new business opportunities in public security, entertainment, and other industries. Specifically, given an audio clip containing the speech from a target individual, the speech2face system is expected to visualize the individual's facial image, based on the voice of the target individual only.
It is straightforward to understand vocal characteristics reflect important personal features, such as gender and age, which can be easily inferred over an individual's speech. Existing studies have also demonstrated interesting observations on the presence of additional physical features of human face, strongly correlated to the vocal features in the voice. Unfortunately, the quality of the output facial images from these studies remains far from satisfactory, due to the following technical limitations. Firstly, the poor quality of paired dataset of individuals' facial image and clean speech hinders the effective training of generative model for speech2face. Secondly, the integration of vocal features into state-of-the-art image synthesis models does not fully utilize the rich information from the speech of the individuals.
To address these technical challenges, we propose speech fusion to face, or SF2F in short, in this paper. Fig. 1 shows sample results of our method. We attempt to improve on top of the existing approaches in two general directions. To tackle the problem of poor data quality, we enhance the VoxCeleb dataset [6,7] by filtering and rebuilding the face images of the celebrities. In Fig. 2, we present example images from filtered VGGFace dataset used in [8] and our refined face image dataset. Common problems associated with the original face images from VGGFace dataset include (1) plenty of monolithic photos containing limited details of the faces; (2) the shooting angle and facial expression are highly diverse and technically difficult to normalize; (3) the background of the face image is not appropriately eliminated. Given these problems, we target to rebuild a high-quality face database for individuals in VoxCeleb database. By carefully retrieving images of these celebrities from the Internet and filtering with specific conditions, we generate HQ-VoxCeleb database with example figures shown in Fig. 2, with highly normalized facial images on different aspects.
Existing studies on speaker recognition, e.g., [9], imply that vocal characteristics are highly unstable in the speech. Speaker recognition/verification algorithms usually extract short-time vocal features from speech and combine these features for final decision making. Such strategies cannot be directly copied to the solution for speech2face system because simple aggregation may lose some key information only available in certain pronunciations. In order to exploit the short-term features for face image generation, we introduce a new fusion strategy over speech extraction algorithms, which applies merging over the face images with different embeddings instead of the original embeddings. A new training scheme, with a new combination of loss functions, is introduced accordingly in order to maximize the information flowing from speech domain to facial imaging domain.
The key contributions of the paper are summarized as follows: (1) we introduce an enhanced face image dataset based on VoxCeleb, containing over 3,000 individual identities with high-quality front face images; (2) we present a new fusion strategy over the short-term vocal characteristics to stabilize the facial features for the synthetic image generation; (3) we propose a new loss function for the neural generative model in order to better preserve the vocal features in the facial generation process; (4) we evaluate the performance of the proposed model on both image fidelity and facial similarity to ground-truth and validates the performance gain over the state-of-the-art solutions.

Related Work
Generative Models. Plenty of generative frameworks have been proposed in recent years. Autoregressive approaches such as PixelCNN and PixelRNN [10] generate images pixel by pixel, modeling the conditional probability distribution of sequences of pixels. Variational Autoencoders (VAEs) [11,12] jointly optimize a pair of encoder and decoder. The encoder transforms the input into a latent distribution, and the decoder synthesizes images based on the latent distribution. Remarkably, Generative Adversarial Networks (GANs) [13,14] which adversarially train a pair of generator and discriminator, have achieved the best visual quality.
Visual generation from audio. The generation of visual information from various types of audio signals has been studied extensively. There exist approaches [15,16,17] to generate an animation that synchronizes to input speech by mappings from phoneme label input sequences to mouth movements. Regarding pixel-level generation, Sadoughi and Busso's approach [18] synthesizes talking lips from speech, and X2Face model [19] manipulates the pose and expression of conditioning on an input face and audio. Wav2Pix model [20] is most relevant to our work, in the sense that it generates face images from speech, with the help of adversarial training. Their objective is to reconstruct the face texture of the speaker including expression and pose. In this paper, our objective is different, targeting to reconstruct a frontal face with a neutral expression, such that most facial attributes of the groundtruth are preserved.
Face-speech association learning. The associations between faces and speech have been widely studied in recent years. Cross-modal matching methods by classification [21,22,23,24] and metric learning [25,26] are adopted in identity verification and retrieval. Cross-modal features extracted from faces and speech are applied to disambiguate voiced and unvoiced consonants [27,28]; to track active speakers of a video [29,30]; or to predict emotion [31] and lip motions [28,32] from speech.
Face reconstruction from speech. Speech2face is an emerging topic in computer vision and machine learning, aiming to reconstruct face images from a voice signal based on existing sample pairs of speech and face images. Oh et al. [4] propose to use a voice encoder that predicts the face recognition embedding, and decode the face image using a pre-trained face decoder [33]. As the most similar approach to ours, Wen et al. [5] employ an end-to-end encoder-decoder model together with GAN framework, based on filtered VGGFace dataset [8]. In this paper, we propose a suite of new approaches and strategies to improve the quality of face images on top of these studies.

Data Quality Enhancement
As demonstrated in Fig. 2, the poor quality of training dataset for speech2face is one of the major factors hindering the improvement of speech2face performance. To eliminate the negative impact of the training dataset, we carefully design and build a new high-quality face database on top of VoxCeleb dataset [6,7], such that face images associated with the celebrities are all at reasonable quality. To fulfill the vision of data quality, we set a number of general guidelines over the face images as the underlying measurement on quality, covering attributes including face angles, lighting conditions, facial expressions, and image background. In order to fully meet the standards as listed above, we adopt a data processing pipeline to build the enhanced HQ-VoxCeleb dataset for our speech2face model training, with details available in Appendix B.1. In Table 1, we summarize the statistics of the result dataset after the adoption of the processing steps above. In Appendix B.1, we compare HQ-VoxCeleb with existing audiovisual datasets, to justify HQ-VoxCeleb is the most suitable dataset for end-to-end learning of speech2face algorithms. HQ-VoxCeleb dataset will be released later.

Model and Approaches
The overall architecture of our proposed SF2F model is illustrated in Fig. 3. The pipeline of our model follows the popular encoder-decoder architecture used in most of the generative models. We   refine the structure to make a better fit for the speech2face task. At inference, we apply a rolling window over the input speech audio. Each window is regarded as an individual local speech segment and the corresponding features from one segment are extracted and stored in a 1D vector by a speech encoder. The global facial features are generated by applying an attention fuser over the sequence of local facial features from the sliding windows. As shown in Fig. 3, the whole training procedure consists of two stages. In the first stage, the voice encoder and face decoder are first optimized to correlate facial features with vocal features extracted from short-time audio windows ranging from 1 second to 1.5 seconds and reconstructing faces based on the features. In the second stage, longer range of audio (global audio) ranging from 6 seconds to 25 seconds are converted into a series of local audio segments with a sliding window; the voice encoder converts the audio segments into corresponding embeddings. An embedding fuser follows to obtain a global facial embedding conditioned on the sequence of local facial embeddings. The weights of voice encoder and face decoder remain fixed in the second stage, while the two training stages share the same conjugated loss function. On the other hand, our approach enables the model to save efforts on learning useless variations mostly contained in long audios, such as the prosody of the speaker and linguistic features of the transcript, and consequently prevents the model from over-fitting. Moreover, the proposed framework maximizes the information extraction from short audios containing most important vocal features with matching features in the facial imaging domain.
Pre-Processing. Over every speech audio waveform, we apply a sliding window at fixed width of 1.25 seconds with 50% overlap. Short-time Fourier Transform (STFT) [34] computes the mel-spectrogram of each window. The input waveform is thus converted into a sequence of mel-spectrograms M = m i ∈ R T ×F , where T denotes the number of frames in the time domain, and F denotes frequency. As we use an end-to-end training pipeline, instead of relying on pre-trained face decoder  as in [4], we expect the groundtruth face images to contain least irrelevant variations, including pose, lighting, and expression. This is enforced by our data enhancement scheme in Section 3. Specifically, each groundtruth face image is denoted by I ∈ R 3×H×W in the rest of the paper.
Voice Encoder. We use a 1D CNN composed of several 1D Inception modules [35] to process melspectrogram. The voice encoder module converts each short input speech segment into a predicted facial feature embedding. The architecture of the 1D Inception module and the voice encoder is described with details in Appendix D. The encoder's basic building block is a 1D Inception module, where each Inception module consists of four parallel 1D CNNs, each with kernel size ranging from 2 to 7 and stride 2 with convolution along the time domain, followed by ReLU activation and batch normalization [36]. The Inception module models various ranges of short-term mel-spectrogram dependency, and enables more accurate information flow from voice domain to image domain, compared to plain single-kernel-size CNN used in [5]. The number of channels grows from lower layers to higher layers in the stack, in order to reflect the bandwidth of the frequency domain to the corresponding layers. On top of the stack of basic building blocks, an average pooling layer outputs a predicted facial embedding e i ∈ R d for each mel-spectrogram segment m i . The embedding e i is normalized as e i = ei max( ei 2, ) , in which = 10 −12 by default. Face Decoder. The decoder reconstructs target individual's face image based on embeddings from the individual's speech segments. Since HQ-VoxCeleb dataset eliminates unnecessary and redundant variance on irrelevant features, the face decoder is expected to target the most distinctive features of faces. The basic building blocks of our face decoder include a 2D bilinear upsampling operator, a 4×4 convolution operator, a ReLU activation, and a batch normalization. On top of stacked basic building blocks, a 1×1 convolution with linear activation is deployed to generate the output I .
Embedding Fuser. The most important part of our model is the fuser of the embeddings from the speech segments. The rationale behind the independent extraction of local features from the speech segment is to enable the model to focus on the vocal characteristics of the individual. When the voice encoder is directly applied to long speech audio, for example, the output features may contain features over the texts in the speech, as well as prosodic features on the emotion and presentation style of the speakers. These features are speaker-independent and therefore irrelevant to the features of the individuals in facial imaging domain. However, embeddings from speech segments are not stable by themselves. Because of the variance of text content in the speech segment, each embedding may reflect completely different vocal characteristics useful to the speech2face task. The embedding fuser is illustrated in the example in Fig. 4.
Given a sequence of mel-spectrograms M = {m 1 , m 2 , ..., m T −1 , m T } generated Short-Time Fourier Transform (STFT) over T consecutive speech segments, voice encoder transforms M into a sequence of embedding vectors E 1:T = {e 1 , e 2 , ..., e T −1 , e T }, each of which contains certain vocal features correlated to different facial features. In order to preserve the useful information from all of the embeddings, embedding fuser first processes E 1:T with a self-attention layer. Given a voice embedding vector e i , its attention score to another voice embedding vector e j is calculated by s(e i , e j ) = e T i W a e j . Based on the scores between the embedding vectors, β i,j indicates the degree of e i attending to e j , measured by β i,j = exp(s(ei,ej )) N j=1 exp(s(ei,ej )) . The output of e i , for each 1 ≤ i ≤ T , attending to the whole sequence E is finalized as a i = T j=1 β i,j e j . Here, attention output a i is concatenated with voice embedding vector e i and then goes through a linear transformation, generating the output of a fine-grained feature Discriminators. To improve the generative capability of our proposal, two discriminators, namely D real and D id , are employed as the adversary to the generative model. The discriminator D real is the traditional binary discriminator as introduced in standard Generative Adversarial Network (GAN) [13] that distinguishes the synthesized images from the real images, by optimizing over the following objective function: x ∼ p fake represent the distributions of real and synthesized images, respectively. Instead of telling difference between real and fake images, the discriminator D id attempts to classify the images into the identity of the individual in the image. A standard identity classifier aims to optimize the following objective function: Conditioned on human speech, the generator network is expected to synthesize human face images, which can be correctly classified by D id . This is achieved by minimizing the objective function Training Scheme. As shown in Fig. 3, the weights associated with the encoder-decoder and the embedding fuser are updated in two stages, respectively. The unified conjugated loss function consists of four different types of losses: (1) Image Reconstruction Loss. L 1 = I −Î 1 penalizes the L 1 differences between the ground-truth image I and the reconstructed imageÎ. (2) Adversarial Loss. Image adversarial loss L G from D real encourages generated face image to appear photo-realistic; (3) Auxiliary Classifier Loss. L C ensures generated objects to be recognizable and well-classified by D id ; (4) FaceNet Perceptual Loss. Image perceptual loss L P penalizes the L 1 difference in the global feature space between the ground truth image I and the reconstructed imageÎ, while object perceptual loss. Inspired by [37], we add a lightweight perceptual loss between generated images and ground truth images to keep the perceptual similarity. This loss not only improves the quality of the generated images, but also enhances the similarity of the output images to the ground truth faces. The loss function of our model is finally formulated as, are the hyper-parameters used to balance the loss functions.

Experiments
Implementation Details. We train all our SF2F models using Adam optimizer [38] with learning rate 5e-4 and batch size 256; the encoder-decoder training takes 120,000 iterations, and the fuser training takes 360 iterations. λ 1 , λ 2 , λ 3 , λ 4 are set at 10, 1, 0.05 and 100, respectively. We implement discriminator with ReLU activation and batch normalization. More information of the dataset is covered in Section 3.

Metric on Similarity.
We evaluate the quality of synthesis outputs by measuring the perceptual similarity between the reconstructed face and its corresponding groundtruth [4]. By feeding the generated face image to a pre-trained FaceNet [39], we generate the face embedding by retrieving the output of the last layer prior to softmax. Given the embeddings from the groundtruth, denoted by U = {u 1 , u 2 , . . . , u N }, and the embeddings from the synthesized images, denoted by U = {u 1 , u 2 , . . . , u N }, over N individuals in our test dataset, we calculate and report the average cosine similarity, i.e., N i=1 cos(u i , u i )/N , and the average L 1 distance, i.e., Metric on Retrieval Performance. We also validate the usefulness of the output face images by querying the test face dataset with the generated face as the reference image, aiming to retrieve the identity of the speaker [4]. Specifically, Recall@K with K = 1, 2, 5, 10 are reported, which is the ratio of query face images with its groundtruth identity included in the top-K similar faces.
Image Quality Evaluation. Existing studies [4,5] do not report the general quality of the generated face images, partly because of the missing of appropriate metric on face image synthesis quality. While Inception Score [40] is commonly employed in studies of image generation [41,42,43,44], it is designed to cover a wide variety of object types in the images. We introduce and report a new quality metric based on VGGFace classifier, which is calculated in a similar way as Inception Score. Inception classifier is simply replaced with VGGFace classifier. The mutual information based on the probabilities over the identities, i.e., VFS(G) = exp 1 , is used to measure image quality. Here, x (i) stands for the i-th image synthesized by the generator G, y is the prediction by VGGFace, and D KL denotes KL Divergence. To justify the adoption of VGGFace Score, we compare it against Inception Score over the real images in HQ-VoxCeleb dataset, at different resolutions. The results in table 2 show that Inception Scores are locked in a small range between 2.0 and 2.6 regardless of the image resolution, implying the total information of the real faces could be encoded with 1 bit only! This is because Inception V3 is not designed for face classification and thus skips the variance of the details on the faces. VGGFace Score, on the other hand, demonstrates a much more reasonable evaluation of face image quality. Comparison with existing work. We include voice2face [5] as the state-of-the-art baseline in this group of experiments. To make a fair comparison, we implement voice2face by ourselves and train its model with our HQ-VoxCeleb dataset. Both voice2face and our SF2F are trained to generate images with 64×64 pixels, and evaluated with 1.25 seconds and 5 seconds of human speech clips. As is presented in Table 3, SF2F outperforms voice2face by a large margin on all metrics. By deploying fuser in SF2F, the maximal recall@10 reaches 36.65%, almost doubling the recall@10 of voice2face at 20.82. Performance comparison on filtered VGGFace dataset is discussed in Appendix C.2. Effect of input audio length. We examine different audio length for the extraction of vocal features. It is crucial to capture the most important vocal features while reserving minimal irrelevant linguistic and prosody information from speech. In Table 4, the results show speech audios between 1 second and 1.5 seconds achieve the best performance on most of the feature similarity and retrieval recall metrics. This justifies the assumption that only short audio's vocal characteristics are useful to the face reconstruction task. VFS remains stable with different audio lengths, implying the capacity of the information flow from vocal domain to facial domain is fairly static.  Effect of output resolution. SF2F is also trained to generate large images with 128×128 pixels. Two different approaches are tested over higher resolution, including a single-resolution SF2F model optimized to generate 128×128 pixels only, and a multi-resolution model optimized to generate both 64×64 pixels and 128×128 pixels. Both of the solutions are equipped with the same conjugated loss function. Table 5 compares the performance of SF2F with fusers trained under different schemes with speech length at 5 seconds. Single resolution model performs worse than the other two, suffering from the unstable optimization gradient over a deeper face decoder. Multi-resolution model achieves comparable performance to single-resolution model. This is because most of the speaker-dependent facial features are already included in 64×64 pixels. The addition of more details with higher resolution does not benefit at all. Qualitative results. We also directly compare the visual performance of the generated face images from SF2F and voice2face. We compare SF2F and voice2face trained on HQ-VoxCeleb, to justify the performance boost brought by our model design. We also provide results generated by voice2face pre-trained and released by Wen et al. [45], to prove the improvement brought by HQ-VoxCeleb. All of the images, from all three of the models, are reconstructed based on speech ranging from 4-7 seconds. As shown in Fig. 5, faces reconstructed by SF2F contain obviously preserves more accurate and meaningful facial features. The pose, expression, and lighting over the faces from SF2F are generally more stable and consistent. More qualitative results of 128 × 128 face image generation are available in Appendix A.
Ablation study. Ablation experiments are discussed in Appendix C.1.

Conclusion and Future Work
In this paper, we present Speech Fusion to Face (SF2F), a new strategy of building generative models converting speech into vivid face images of the target individual. We demonstrate the huge performance boost, on both face similarity and image fidelity, brought by the enhancement of training data quality and the new vocal embedding fusion strategy. In the future, we will explore the following research directions for further performance improvement of speech2face system. Firstly, we plan to introduce more accurate vocal embedding methods, in terms of the capability to distinguish different speakers. Secondly, we will evaluate the possibilities of hierarchical attention mechanisms, in order to link the vocal features to the corresponding facial features on the correct layers.

Broader Impact
Speech Fusion to Face (SF2F) makes the following positive impacts to the society: (1) with the help of SF2F, law enforcement departments can convert voiceprint evidence to face portraits, to facilitate tracking or identifying culprits; (2) SF2F can be used together with video processing techniques to create entertaining videos. On the other hand, the privacy protection of personal facial images will become more challenging. SF2F could be abused to infer facial images, even when the individual prefers to keep anonymous by using his voice only. When SF2F is applied for public security, inaccurate face reconstruction could wrongly identify innocent people as suspects, therefore, the police departments should only use SF2F as an auxiliary system in their crime fighting campaign. If SF2F is trained on a biased population with unbalanced geographical attributes, such as race, age, and gender, the performance of SF2F may worsen when used to predict facial images of minorities, consequently leading to potential fairness problem.

A Qualitative Results
In the main paper, we compare 64 × 64 face images generated by SF2F and voice2face [5]. In this section, we compare 128 × 128 face images reconstructed by both models. As the original voice2face [5,45] is only trained to generate 64 × 64 images, we compare SF2F and voice2face trained on HQ-VoxCeleb dataset. To ensure fairness in the comparison, both models are trained until convergence, and checkpoints with the best L1 similarity are used for inference. As shown in Fig. 6, although voice2face can capture attributes such as gender, SF2F generates images with much more accurate facial features as well as face contours. The pose, expression and lighting condition over the faces from SF2F are generally more stable and consistent than the face images from voice2face. In group (f), for example, SF2F predicts a face almost identical to the groundtruth, when the respective output from voice2face is hardly recognizable. This verifies our model enables more accurate information flow from the speech domain to the face domain.

B.1 HQ-VoxCeleb Dataset
As an overview of the HQ-VoxCeleb dataset is presented in the main paper, in this section, we elaborate on the standards and process in our data enhancement scheme. As demonstrated in Fig. 7, the poor quality of training dataset for speech2face is one of the major factors hindering the improvement of speech2face performance. To eliminate the negative impact of the training data itself, we carefully design and build a new high-quality face database on top of VoxCeleb dataset, such that face images associated with the celebrities are all at reasonable quality.
To fulfill the vision of data quality, we set a number of general guidelines over the face images as the underlying measurement on quality, as follows: • Face angle between human's facial plane and the photo imaging plane is no larger than 5 • ; • Lighting condition on the face is generally uniform, without any noticeable shadow and sharp illumination change; • Expression on the human faces is generally neutral, while minor smile expression is also acceptable; • Background does not contain irrelevant information in the image, and completely void background in white is preferred.
To fully meet the standards as listed above, we adopt the following methodology to build the enhanced HQ-VoxCeleb dataset for our speech2face model training.

HQ-VoxCeleb
Filtered VGGFace Problems with Filtered VGGFace Data Collection. We collect 300 in-the-wild images for each of the 7,363 individuals in VoxCeleb dataset, by crawling images of the celebrities on the Internet. The visual qualities of the retrieved images are highly diverse. The resolution of the images, for example, ranges from 47×59 to 6245×8093 in pixels. Moreover, some of the images cover the full body of the celebrity of interest, while other images only include the face of the target individual. It is, therefore, necessary to apply pre-processing and filtering operations to ensure 1) the variance of the image quality is reasonably small; and 2) all the images are centered at the face of the target individuals.
Machine Filtering. To filter out unqualified images from the massive in-the-wild images, we deploy an automated filtering module, together with a suite of concrete selection rules, to eliminate images at poor quality, before dispatching the images for human filtering. In the filtering module, the algorithm first detects the landmarks from the raw face images. Based on the output landmarks of the faces, the algorithm identifies poorly posed faces, if the landmarks from left/right sides cannot be well aligned. Low-resolution face images, with distance between pupils covers fewer than 30 pixels, are also removed. Finally, a pre-trained CNN classifier [46] is deployed to infer the emotion of the individual in the image, such that faces not recognized as "neutral" emotion are also removed from the dataset.
Image Processing. Given the face images passing the first round machine-based filtering, we apply a two-step image processing, namely face alignment and image segmentation in the second round of filtering. In the first step of face alignment, the images are rotated and cropped to make sure both the pupils of the faces in all these images are always at the same coordinates. In the second step of image segmentation, we apply a pyramid scene parsing network [47] pre-trained on Pascal VOC 2012 dataset [48] to split the target individuals from the background in the images. The background is then refilled with white pixels. Note that the second step is helpful because irrelevant noise in the background may potentially confuse the generation model.
Human Filtering. To guarantee the final quality of the images in HQ-VoxCeleb dataset, human workers are employed to select 3 to 7 images at best qualities for each celebrity. Only celebrities with at least 3 qualified face images are kept in the final dataset.

B.2 Comparison with Existing Datasets
In the main paper, we summarize the statistics of the result dataset after the adoption of the processing steps above. In this section, we compare HQ-VoxCeleb with existing audiovisual datasets [26,25,49,7,6] in terms of data quality and its impact to model training, in order to justify the contribution of HQ-VoxCeleb. Table 6 shows the attributes of existing audiovisual datasets.
Existing datasets, including VoxCeleb [6,7] and AVSpeech [49], contain a massive number of pairwise data of human speech and face images. However, the existing datasets are constructed by cropping face images and speech audio from in-the-wild online data, and the face images thus vary hugely in pose, lighting, and emotion, which makes the existing datasets unfit for end-to-end learning of speech-to-face algorithms. To the highest image quality of existing datasets, Wen et al. [5] use the intersection of the filtered VGGFace [26] and VoxCeleb with the common identities. However, as shown in Fig. 7, the filtered VGGFace cannot meet the quality standards defined in section B.1. Moreover, HQ-VoxCeleb has triple as many identities as filtered VGGFace as shown in Table 6. In conclusion, the high quality and reasonable amount of data make HQ-VoxCeleb the most suitable dataset for speech2face tasks.

C.1 Ablation Study
In this section, we focus on the effectiveness evaluation over the model components, loss functions, and dataset used in our SF2F approach. The ablated models are trained to generate 64 × 64 images, with all the results are summarized in Table 7. When removing any of the four loss functions in {L1, LG, LC , LP }, the performance of SF2F drops accordingly. This shows it is necessary to include all these components to ensure the learning procedure of the model is well balanced. 1D-CNN encoder is used in the baseline approach [5], while SF2F employs 1D-Inception encoder instead. We report the performance of SF2F by substituting our encoder with 1D-CNN, referred to as Baseline Encoder in Table 7. This replacement causes a significant drop of the performance on all metrics. Similarly, by adopting the deconvolution-based decoder used in [5,4] instead of the upsampling-and-convolution-based decoder in SF2F, referred as Baseline Decoder in Table 7, we observe a slight yet consistent performance drop. The impact of data quality is also evaluated, by training the SF2F model with the manually filtered version of VGGFace dataset [8] instead of HQ-VoxCeleb, by including overlap celebrity individuals included in both datasets. Poor data quality obviously leads to a huge performance plunge, which further justifies the importance of training data quality enhancement.

C.2 Performance on Filtered VGGFace
We also compare the performance of SF2F and voice2face [5] on the filtered VGGFace dataset [8], to evaluate how SF2F functions under less controlled conditions. To make a fair comparison, we train both models with filtered VGGFace dataset. Both voice2face and our SF2F are trained to generate images with 64×64 pixels, and evaluated with 1.25 seconds and 5 seconds of human speech clips. As is presented in Table 8, SF2F outperforms voice2face by a large margin on all metrics. By deploying fuser in SF2F, the maximal recall@10 reaches 21.34%, significantly outperforming voice2face. We use a 1D-CNN composed of several 1D Inception modules [35] to process mel-spectrogram. The voice encoder module converts each short input speech segment into a predicted facial feature embedding. The architecture of the 1D Inception module and the voice encoder is summarized in Table 9. The Inception module models various ranges of short-term mel-spectrogram dependency, and enables more accurate information flow from voice domain to image domain, compared to plain single-kernel-size CNN used in [5].  Table 10. We show the structure of the face decoder generating 64×64 images, and the face decoder generating 128×128 images can be built by adding UpBlock 7 after UpBlock 6. Our empirical evaluations prove that our decoder based on upsampling and convolution renders better performance in all the metrics compared to the deconvolution-based decoder employed in existing studies [4] and [5].

D.2.2 Multi-resolution Decoder
Inspired by [50], the multi-resolution decoder is optimized to generate images at both low resolution and high resolution, which is shown in Fig. 8. With the multi-resolution approach, the decoder learns to model the multiple target domain distributions of different scales, which helps to overcome the difficulty of generating high-resolution images.
Face Decoder

D.3 Discriminators
The network structure of image discriminator D real and identity classifier D id is described in Table 11. Both networks are convolutional neural networks, followed by fully connected networks. In Table 11, we demonstrate the structure of discriminators for 64 × 64 images, and the discriminator of 128 × 128 images can be simply implemented by adding another convolution layer before average pooling layer.  Table 12

E.2 Evaluation
Evaluation is conducted on the same GPU machine. The details of the implementation are provided in this part of the section.
Ground Truth Embedding Matrix. As mentioned in the main paper, the ground truth embedding matrix U = {u1, u2, . . . , uN } extracted by FaceNet [39] is used for similarity metric and retrieval metric. Given that one identity is often associated with multiple, i.e., K, face images in both datasets, for an identity of index n, the ground truth embedding is computed by un = K j=1 unj/K. The embedding un is normalized as un = un max( un 2 , ) , because the embeddings extracted by FaceNet are also L2 normalized. Building embedding matrix with all the image data helps remove variance in data. This makes the evaluation results fair and stable.
Evaluation Runs. In each experiment, we randomly crop 10 pieces of audio with desired length for each identity in the evaluation dataset. With the data above, each experiment is evaluated ten times, the mean of each metric is calculated based on the outcomes of all these ten evaluation runs. We additionally report the variance of VGGFace Score.

E.3 Hyperparameter
The hyperparameters for SF2F's model training are listed in Table 13. The detailed configuration of SF2F's network is available in Section D as well as the main paper. Hyperparameters are carefully tuned. As the training is time-consuming and the hyperparameter space is large, it is difficult to apply grid search directly. We adopt other methods for parameter tuning instead.
We first train a SF2F model with λ1, λ2, λ3 = 1 and λ4 = 0, we adjust the parameters λ1 and λ3 and find that increasing the Image Reconstruction Loss weight λ1 and decreasing the Auxiliary Classifier Loss weight λ3 both improve the model performance. We adjust λ1 and λ3 gradually and find out the model achieves the best performance when λ1 = 10 and λ3 = 0.05 . Afterward, we apply Perceptual Loss to our model training with initial weight λ4 = 1. We find increasing λ4 improves SF2F's performance, which is because the original scale of Perceptual Loss is much smaller than the scale of the other three losses. We gradually increase λ4 until we observe SF2F achieving the best performance when λ4 = 100. Therefore, λ1, λ2, λ3, λ4 are set at 10, 1, 0.05 and 100, respectively.
Afterwards, we apply a grid search over learning rate and batch size. We test with different values, which are listed in Table 14 with optimal values singled out in the last column.