{'fraction_non_alphanumeric': 0.03979927323066274, 'fraction_numerical': 0.010324738997519756, 'mean_word_length': 4.76955074875208, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Towards 3D Dance Motion Synthesis and Control

3D human dance motion is a cooperative and elegant social movement. Unlike regular simple locomotion, it is challenging to synthesize artistic dance motions due to the irregularity, kinematic complexity and diversity. It requires the synthesized dance is realistic, diverse and controllable. In this paper, we propose a novel generative motion model based on temporal convolution and LSTM,TC-LSTM, to synthesize realistic and diverse dance motion. We introduce a unique control signal, dance melody line, to heighten controllability. Hence, our model, and its switch for control signals, promote a variety of applications: random dance synthesis, music-to-dance, user control, and more. Our experiments demonstrate that our model can synthesize artistic dance motion in various dance types. Compared with existing methods, our method achieved start-of-the-art results.


INTRODUCTION
Dance and concepts thereof are embroidered in our society, culture, and history [3]-whether it is freestyle (i.e., on-the-fly), a specific dance to a certain song (e.g., the macarena), a spiritually or culturally inspired dance-off, or even just solo acts of dancing while alone and replaying a melody from memory. Hence, dance moves have the power to allow one-to-many individuals express emotions, all the while having the persistence to inspire, spread knowledge, show culture, and promote believes. As a part of the mobile-age, dance performances are easily main-streamed, making it possible to broadcast or view from just about anywhere at given moment. Now, we aim to best leverage our scientific research to directly enhance the melody of life we call dancing.
Dancing can be considered a form of art. It requires professional choreographers to create and design artistic movements to express emotions. For this, professional dancers are trained and equipped with a rich repertoire of dance steps -the more creative, the better. Different dancers perform quite differently, even to the same music or melody. Nonprofessionals would typically find it challenging to create a dance. Therefore, to acquire the ability to automatically create dances is a daunting task, as dancing contains high kinematic complexity that span long-term spatio-temporal structures (i.e., temporal 3D human dance motion), which make it difficult to synthesize realistic dance. More importantly, dance motion is diverse, irregular, complex, and often designed for specific music or melody. In addition, it is important to note that dancing is inherently a multi-modal problem [20], spanning multiple views (i.e., various dances for the same song). Lastly, different music or melodies should yield a whole variety of dance types. The challenges and specifications mentioned here demand an effective generative model to handle the complex and diverse dance motions. Furthermore, with such high powerful model, it should be adequate for various applications: freestyle(random synthesis [25]), dancing with music(music2dance [40]), and support ordinary users to create dance(user control).
Early on, researches mainly adopted similarity-based retrieval methods to synthesize simple locomotion [24,29,33]. Then, others proposed methods to synthesize long-term dance motions in accordance to musical inputs [7,23,30]. However, these strategies lack flexibility, creativity, and are difficult to apply to irregular, complex dance motion. More recently, deep learning-based motion synthesis algorithms have shown higher potential [8,25,26,31,32]. The deep neural network (NN) can well model spatio-temporal structures with high kinematic complexity without taking up too much memory. Recurrent Neural Network (RNN) [8,26] has been proposed in recent years to model human motions for human motion prediction. However, these methods can easily fall into the temporal accumulation error(i.e., get stuck in static poses). Li et al. proposed the auto-condition training strategy to train the model based on RNNs to synthesize complex dance motion [25]. The method can only achieve random motion synthesis(long-term motion prediction), but not controllable motion synthesis. Considering the control signal, the method based on long short-term memorys (LSTMs) [22] can synthesize controllable simple locomotion to interact with the environment, and it is difficult to synthesize complex controllable dance motion. Recently, some learning-based methods [20,35,40] have been used to synthesize controllable dance motion from music. Lee et al. designed a NN based on variational autoencoder (VAE) and generative adversarial network (GAN) to synthesize 2D dance movement [20]. The LSTMs-autoencoder (AE) [35] is proposed to synthesize 3D dance motion from music feature, but the synthesized dance motion is far from realistic and diverse.The DanceNet, based on Temporal Convolution Network (TCN), have been applied to synthesize diverse dances for different dance types [40]. However, the method can not synthesize various dances for the same music. Different from the direction and speed control in the locomotion synthesis [22], the control signal adopted by the method is directly extracted from the music. Although there is the rhythmic consistency between music and dance, it is difficult to completely determine the dance motion. This is a weak (i.e., not strong) control signal, so the controllability of their synthesized dance motion is lacking. Moreover, existing methods do not span various applications with the same model.
We first propose a dance control signal. Unlike in [40], we introduce a control signal, called dance melody line, and it is highly correlated to the dance since it is extracted directly from the dance motion. We sum up the speed of the salient joints frame-by-frame to capture the melody control signal (i.e., a 1D control signal). Provided a low-dimensional signal produced directly by motion, its coupling is notably strong, i.e., different dance motions correspond to the same dance melody line. This helps to synthesize different dance motions conditioned with the same control signal.
We also introduce a novel generative model with encoding and decoding stages. TCNs are robust to noisy inputs [40], so we adopt it to extract motion features and fuse control signals to obtain controllable motion features, which is done as part of the the encoding stage. To strengthen the long-term spatio-temporal dependence of the output frames, we adopt the LSTMs as the decoder. Our model overcomes the shortcomings of the LSTMs that is not robust to noise, while ensuring that the output frames leverage long sequence dependency. Our model obtains the controllable motion features based on the TCN, and then LSTMs decode to synthesize controllable dance motion. As a whole, we call our framework the Temporal Convolution-LSTMs (TC-LSTMs). The output of TC-LSTMs is designed as a probability density function (PDF) (i.e., Gaussian mixture model), which also makes our model more robust. With a careful training strategy (i.e., mix training), our model supports switch melody control signal to synthesize dance motion, meaning that our model can use the same parameters for random and controllable motion synthesis.
Applications. We ran experiments on music-dance pair dataset [40]. The results show that our model can generate realistic and diverse dance motions for different applications. Listed as follows: Random synthesis. We can switch off the melody control signal, and our model synthesizes long-term (i.e., arbitrary length), diverse dance sequences(without the temporal accumulation error).
Music2Dance. From the analysis of the music-dance paired data [40], we found that the melody lines of music and dance are highly matched, so we directly use the music melody line as the melody control signal. The type control signal can be obtained by a classifier as in [40] or user given. Through the music melody line and dance type to synthesize the dance motion consistent with the melody and style of music.
User control. The dance melody line is a 1D signal, which is easily given by ordinary users. Therefore, our approach allows ordinary users to design dance motions. We can synthesize the controllable dance through the user-defined melody line(i.e., drawing) as the melody control signal.
Research contributions. Along with the direct, tangible benefits, we propose the following contributions in research: Table 1: Compare against state-of-the-art (SOTA) methods about 3D dance synthesis for different applications. Model extensibility means the model can synthesis dance for different dance types. ✓ means the method can achieve the application. × means the method can not achieve the application.

Method
Model extensibility Random synthesis Music2Dance User control ac-LSTM [25] × Controllability. To our knowledge, we are the first to propose a controllable dance synthesis framework.
Robust to noisy inputs and long-term dependencies. Our encoderdecoder structure ensures robustness to noise,and building longterm spatio-temporal dependence of the output frames.
SOTA results with various applications. Our experiments show that our approach can achieve SOTA results for different applications.

BACKGROUND
We review three research area most related to the proposed (i.e., motion synthesis and control, dance motion, and generative model).
Motion synthesis and control. Researchers tend to synthesize motions via data-driven methods, i.e., , Hidden Markov Models (HMM) [2,4], spatial-temporal dynamic models [6,19,37,38], and low-dimensional statistical models [5,11]. In addition, other methods to synthesize locomotion were based on motion graphs [18,21,24,29,33]. A common strategy among the aforementioned methods is the formulation of similarity-based retrieval to synthesize simple locomotion, which are completely dependent on the availability of dataset, hence, lacks flexibility and tend not to generalize well. Nowadays, deep NN-based methods [8,15] gradually started being used to synthesize motion. For instance, the RNN-based methods that were proposed to predict short-term human motion, while being unable to synthesize long-term motion due to the temporal accumulation error [8,16,26]. Li et al. adopted the auto-conditioned training strategy to synthesize long-term motion, but lacking the ability to control the generated motion [25]. Phase-functional networks [14] and LSTMs-based method [22] were introduced to synthesize controllable locomotion. However, the these methods are still limited -either by simple-random or simple-controlled locomotion, thus they are unable to synthesize complex dance motion with complete control. These limitations can be overcome in our approach. Our method can synthesize realistic, complex, diverse and controllable dance motion sequences.
Dance motion. As mentioned, earlier research tended to focus on synthesizing dance motions by adopting similarity retrieval strategies (e.g., motion graph [21,24]). Fan et al. divided the long-term dance motion into multiple short-term clips, which were then used to build a motion graph [7]. Shiratori et al. retrieved each dance segment where the music and dance rhythm were consistent [34]. However, these methods rely entirely on the dataset, and lack the ability to truly creativity are music-consistency. Recently, various types of NNs emerged as solutions to generate dance movements. For instance, VAE and GAN models were proposed to synthesize 2D dance motion from music [20]. Tang et al. built an LSTM-AE to generate 3D dance motions; however, the generated motions are unrealistic [35]. Li et al. proposed auto-conditioned LSTM to synthesize 3D dance motion; however, this work lacked motioncontrol(just random synthesis) [25]. A model based on temporal convolution was proposed to generate 3D, controllable dance motions, but the controllability is limited in its inability to synthesize multimodal dances provided the same control signals [40]. Our model overcomes the limitations in being controllable, as we are able to synthesize realistic, diverse, and controllable multimodal dances. Furthermore, we use just a single model in a wide-range of applications: random synthesis, music2dance, and user control (Fig 1).
Generative model. In the motion synthesis model, the common autoregressive model is an LSTMs [8,16,25,26]. Like most others, these models can only generate random outputs, and lack the ability to control the synthesized motion. Lee et al. designed the control signal at the model input, but lacked robustness to noise [22]. Zhuang et al. proposed an autoregressive model based on temporal convolution [40], although the model is robust to input noise, it was unable to capture temporal dependencies across output frames. We carefully consider the pros and cons of the temporal convolution and LSTMs-based methods -leveraging the strengths of each to design our model. During the encoding phase, we train our model to be insensitive to input noise via temporal convolution to encode features. Thereafter, we employ an LSTMs to decode, as it enhances the temporal correlation of the output motion so that our method synthesizes realistic, complex, diverse and controllable dance motions.

OVERVIEW
The proposed framework is shown in Figure 1. We take 3D dance synthesis as an autoregressive process conditioned on control signals. specifically, to synthesize current frame, we take the previous dance frames and control signals(dance melody line and dance type) as inputs. We will introduce dance motion and control signal processing in Section 4. The model consists of two parts: encoder based on temporal convolution and decoder based on LSTMs. We will elaborate on our model in Section 5. After decoding, the model outputs the PDF of current dance frame, then we sample from the PDF to get the current frame. Our model can realize different applications with a set of parameters: dance random synthesis, Music2Dance, and user control, which are introduced in Section 6. Figure 2: Depiction of the melody line from a modern dance. We proposed quantifying the melody lines as 1D signals.

DATA PROCESSING
Zhuang et al. introduced a high-quality music-dance pair dataset to synthesize dance-from-music [40]. This dataset consists of two types of dances -modern dance (≈26.15 minutes, 94,155 frames at 60 FPS) and Korean dance (≈31.72 minutes, 114,192 frames at 60 FPS) -we used to train the proposed model. The aim of this work was to generate controllable dance motions with the control signal made-up of two parts, i.e., dance melody line (characterize the dance rhythm, local condition), and dance type(characterize the dance style, global condition). The dance type can be represented as an one-hot vector c s , similar to [40]. We next describe the dance melody line (Section 4.1), and then the dance motion representation (Section4.2).

Dance melody line
Professional choreographers taught us that human dance moves are more than just random movements [13]. There is an internal melody for dance founded on higher-level information that reflects the rhythm and speed of a dance, which mirrors that of the main melody line in music theory [27]. For this, we introduce a kinematic melody line extractor to encode speed information of the motion. Theoretically, we can either use the angular velocities or position velocities to encode the speed of a dance. In practice, we chose to use joint translation velocities for the purpose of motion control, similar to [17]. Inspired by [7], we extract the speed of the motion for just a few key joints, opposed to all of them. Specifically, we focused on the left and right shoulders, elbows, hands, knees, feet, and the head. From these joints, we can express the salient kinematic movement, along with the importance of the positions in motion features [21]. The speed of motion is determined by the change in position between neighboring poses. Specifically, the speed of the motion at frame t is the sum of speed of each key joint k. Mathematically speaking, where the p k t is the position of joint k at frame t. To ensure smoothness, we use a Gaussian filter д to smooth the motion speed to get the motion melody line L(t). Thus, L(t) (i.e., the motion speed) is a strong signal (Fig. 2). Meanwhile, the signal is highly coupled due to summing of speeds-different motions may have the same melody line.
Since dance focuses on melody changes, we use the melody change trend as the control signal instead of directly using the melody line. Specifically, we extract the melody line relative to the value of current frame in a one second interval, and use it as the melody control signal. We represent the melody control signal c l t by sparsely sampling in the temporal domain. Hence, starting from frame t of the melody line clip, a one second sequence of future frames is extracted (i.e., 60 frames from our 60 FPS dance motion). Then, we down-sampled in the temporal domain to 12-1D points. Finally, we subtract the speed of frame t to produce the change in speed and, thus, the melody control signal, where t int is the sampling interval and n is the sampling index.

Motion representation
Human motion is modeled as an articulated figure with rigid links (i.e., socket joints) connected ball-to-ball. Each frame is represented by the root translation (p x , p y , p z ), rotation (r x , r y , r z ), and the other joint rotations with respect to the parents (i.e., r jx , r jy , r jz , and j is the joint index). However, such a motion representation is a relative feature with local information related their parents. In addition, the translation and rotation of the root joint is relative to world coordinates. Ultimately, this increases the motion feature space, which increases the modelling complexity. Thus, we adopt the relative translation and rotation of root joint and add 3D joint positions, angle, and position velocities to the representation. This allows us to better model human motions. In our method, the motion feature at frame t is represented as the rotations using quaternion exponential mapping x r We extract motion features from two aspects (i.e., rotation and position) to maximize the amount of motion information. Then, we add the angular velocity and linear velocity to more fully represent the motion feature. In addition, the information about the foot contact x f t is added to reduce the foot sliding in the generated frames. Like in [14,40], the foot contact labels(ground-truth) are detected by the height and speed of toe end per frame.

GENERATIVE MODEL
Next, we introduce the structure of the proposed model, and then we discuss the training details.

Encoder-decoder structure
Our end-to-end framework is shown in Figure 1: the proposed models the PDF of the predicted motion conditioned on control Pr (x t |X pr e , c l t , c s ), Pr (x t |X pr e , c l t , c s ) = Γ(X pr e , c l t , c s ) = Γ D (Γ E (X pr e , c l t , c s )), (7) where X pr e = x t −k −1 , . . . , x t −1 is the motion of the previous k frames and Γ is our model made-up of encoder Γ E and decoder Γ D . Encoder. Inspired by [40], we adopt temporal dilated convolution to extract input features for improved robustness to noise. The motion controllable feature m t is extracted during encoding via Due to the high complexity of motion data, we first encode the input with a two stacked Conv1D+Relu module Γ e E . Then, the residual control module Γ r,i E fuses motion coded features and control signals similar to [40]. We stack 10-Γ r,i E and sum the outputs to produce motion controllable feature m t . Furthermore, the dilated convolution in Γ r,i E can extract the temporal fusion information of the motion sequence by increasing the receptive field. Also, Γ r,i E accepts the control signals as input to the 1D-conv layer with a kernel size of 1. Then, the fused temporal motion feature is fused with the control features by summing. However, the coupling formed via addition lacks in strength, which allows us to swap the control signals from training-to-inference. In our experiments, the input channel of Γ r,i E is 128D, while the motion controllable feature m t is 512D.
Decoder. To improve the temporal correlation of the output motion, we use two LSTMs and a fully connected layer to decode m t and to predict the PDF of current frame (Fig. 3). Mathematically speaking, where h 1 t and o 1 t represent the hidden state and cell memory of the first LSTM layer, respectively, and h 2 t , o 2 t are the second layer (Fig. 3). The hidden states and cell memories of the LSTMs ensure temporal correlation between output frames such that the output motion is smooth and realistic.

Training
Training loss. The output of TC-LSTM is the PDF of frame t, which we model as a Gaussian Mixture Model (GMM), with a loss defined as the negative log likelihood. Specifically, where the balance parameters λ = 0.1 and β = 1. Then, at inference, the generated motion frame can be obtained by sampling from the predicted PDF. Implementation details. To achieve dance synthesis with and without the dance melody line control signal, we adopt a mix training strategy. That is, we pass the probabilistic input (i.e., melody control signal) during training with a probability of 0.5. To obtain robustness, we apply data augmentation: (1) mirror transformations for additional dance motion, (2) added Gaussian noise (i.e., µ noise = 0 and σ noise = 0.05) to the input and ground-truth to learn to handle temporal accumulation error, and (3) apply dropout (i.e., 0.4) at the input to resolve the problems of over-fitting. We initialize our model we use Xavier normal [9], and optimize via RM-Sprop [36]. Training runs for 500 epochs, starting with a learning rate of 4 × 10 −4 , and then dropping by a factor of 10 at epoch 300.
The batch-size is 128, setting each sample as a motion sequence of 600 continuous frames. Our system is implemented using PyTroch 1.2 on a PC with Intel I7 CPU, 32G RAM, and a GeForce-GTX 1080Ti.

EXPERIMENT
The proposed melody control signal allows it to be toggled on and off for different applications (i.e., random synthesis, Music2Dance, and user control). As far as we know, this is the first dance motion synthesizer proposed for a wide range of applications with the same model(the same parameters). Furthermore, this is the first user-controlled dance motion synthesizer for specific motions.
In this section, we describe each application, and compare with SOTA methods ( Table 1). The SOTA random synthesis model (i.e., the ac-LSTM [25]), adopts a unique strategy (i.e., auto-condition) to train the LSTM. However, it lacks controllability for dance motion. Tang et al. proposed an AE-based LSTM (i.e., LSTM-AE) to synthesize dance motion from music [35], but its synthesized dances are unrealistic and out of sync with the music. Furthermore, it lacks an ability to be used in other applications. DanceNet was proposed to synthesize dance from music [40]. However, it lacks the ability to synthesize a variety of dance motions from the same music. Our model synthesizes realistic and diverse dance motions of different dance types, and the dance motions synthesized from same music

Random synthesis
Given the dance type and initial dance motion frames(30 frames), our model can generate realistic and diverse dance sequences with its own style. With the same initial input frames, our model can generate different dance motion sequences, as shown in Figure 5. Since the predicted motion frame needs to be sampled from thePDFof the model output, which can effectively increase motion diversity. Then the sampled motion frame is fed back to the input to generate follow-up frame. To demonstrate our method, we compared with the SOTA model: ac-LSTM [25]. We randomly synthesize 15 dance motion sequences for each dance type, and every three synthesized sequences share the same initial input frames. We evaluate the random synthesized dance motion from realism and diversity. The realism can be evaluated by Fréchet Inception Distance(FID) [12], similar to [20,39]. We adopt 3 temporal convolution layers and 1 Bi-LSTM layer as a feature extractor to obtain the FID score since the FID needs an action classifier to extract dance features. Our method can synthesize diverse sequences with the different/same initial frames, so we can evaluate the diversity by the dance features extracted by the action classifier, that is, the average feature distance among different sequences. Diversity-I evaluates the diversity of synthesized different dance sequences conditioned on different initial frames, Diversity-II evaluates the diversity conditioned on same initial frames, and the result is shown in Table 2.
For better comparison, we use user study to score the realism and diversity of dance(10 users), as shown in Figure 4a. Our model can synthesize more realistic dance motion sequences, close to real dance sequences. It is worth noting that the dances generated by our model are diverse, while ac-LSTM [25] can not synthesize diverse dances at all for the same initial frames. Their method can only synthesize the same dance with the same initial frames. One explanation is that their training strategy avoids the temporal accumulation error, but the model completely loses diversity(just overfit to the train data). Our method can achieve the random synthesis of diverse motion sequences. Because our model has complex and robust modelling capabilities and the output of our model is the probabilistic density (we need to sample for predicted frame), which increases dance diversity.

Music2Dance
Synthesizing music-consistent dance is an interesting and challenging task. It requires that the synthesized dance can be consistent with the music rhythm, style and melody. However, music and dance are weakly related, and it does not determine the specific dance posture, that is, music does not determine whether the dance moves are leg lifting, jumping, or circling. Therefore, how to establish a correlation between music and dance is very important. Tang et al. [35] synthesized dance directly from music, and did  Figure 6: The melody lines of dance and music in musicdance pair dataset.
not explicitly establish the relationship between music and dance, so the synthesized dance is not realistic. Zhuang et al. [40] added music feature in the process of auto-regressive synthesis motion, but did not establish the relationship between music and dance. How to determine the relationship between music and dance is the core difficulty of music2dance task. From professional choreographers, we know the relationship between music and dance is reflected in the melody and rhythm. Therefore, we construct the relationship between music and dance through the melody line of music and dance. In section 4.1, we propose the dance melody line to express the melody and rhythm of dance. In order to extract the music melody line, we introduce a simple and effective extraction method: extract onset strength by librosa [28] or madmom [1] and then smooth it through a Gaussian filter.
We select a segment from the music-dance pair data to obtain the melody lines of music and dance, and compare the relationship between them, as shown in Figure 6. Although the melody value of each frame is not necessarily the same, the change trend and peak value of the two melody lines are basically the same, indicating that the consistency of the melody rhythm between music and dance can be reflected through the melody line. In section 4.1, we introduce that the melody control signal of the model is the change trend of the melody line relative to the current frame, so we can directly adopt the change trend of the music melody line as the melody control signal.
To demonstrate the effectiveness of our approach, we compare with two SOTA methods, LSTM-AE [35] and DanceNet [40]. We use the FID to evaluate the dance realism, and the average feature distance to evaluate the dance diversity. For this application, dance diversity is evaluated from three views: (1) Diversity-I, the diversity of the different synthesized dance sequences conditioned on different initial frames and different melody line; (2) Diversity-II, the diversity is conditioned on same initial frame and different melody lines; and (3) Diversity-III, the diversity is conditioned on same initial frame and the same melody line. Diversity-III reflects the multi-modality power of Music2Dance. In addition, we use the rhythm consistency (i.e., the rhythm hit rate) to evaluate our methods like in [20,40]. We randomly select 5 initial sequences to synthesize 30 dance motion sequences for each dance type using the two methods mentioned and our approach. For each, three were synthesized conditioned on the same initial sequence and the same melody line, and three were synthesized conditioned on the same initial sequence and a different melody line. The quantitative results are shown in Table 3. We conduct a user study to evaluate the realism of the music-consistency (Fig. 4, b & c). Our results significantly outperform LSTM-AE. We believe that directly mapping music to get the dance movement is unreasonable due to the weak correlation between dance and music. DanceNet [40] uses music features as the conditions to synthesize dance, but this method directly inputs music features without explicitly analyzing the correlation between music and dance. So, the model takes a long Table 3: Comparison of realism (FID), diversity (Diversity-I: synthesized conditioned different initial frames and different melody line, Diversity-II: synthesized conditioned same initial frames and different melody line, Diversity-III (Multi-modal): synthesized conditioned same initial frames and same melody line), rhythm-consistent (rhythm hit rate, higher is better).  Figure 8: User controls -the melody line is drawn by users.
time to train (i.e., >1,000 epochs). However, the consistency in the synthesized dance and the music is low. In addition, the output of DanceNet is a probability distribution -a model trained for too long would cause it to collapse (i.e., it lacks in diversity), and especially the synthesized dance sequences conditioned on same initial frame. We explicitly analyze the relationship between music and dance represented by the melody lines. During training, the dance melody line is used as the control signal. Then, at inference, the music melody line is used as the control signal. This strategy ensures the controllable effect (i.e., music consistency) of the synthesized dance. More importantly, the highly coupled of the melody lines and low training difficulty (i.e., about 500 epochs), our method prevents the model from collapsing to synthesize various dances. In addition to music in WAV format, our method also works with audio in other formats (e.g., MIDI, an electronic music format): melody lines are captured as a temporal sequence, as it is easier to obtain melody line (i.e., obtained from the change of music note).

User control
Given the dance type, our model synthesizes realistic, diverse dance motions conditioned on the melody line. The melody line is a simple 1D signal (Fig. 2, 6, and 7). Thus, allowing an ordinary user to create dances, opposed to depending on a professional choreographerthere is a variety of ways the melody line can be described: e.g., drawing (Fig. 8). We built an end-to-end system based on two steps: (1) the user draws the melody line via mouse inputs and (2) the synthesizer generates a dance according to the melody line. Note that since the lines drawn by the user are evenly sampled as melody lines, our model synthesizes melody-consistent dance motions.
In the end, this is the first model with such controls for synthesizing dances. Thus, we evaluate our method by user study. We asked 10 users draw 3 melody lines. Then, they scored each dance sequence separately by measuring dance realism and melodyconsistency (Fig. 4 d). Our method synthesizes realistic dances, while ensuring melody-consistency, especially for the modern dance.

Discussion and future work
Ablation study. We propose the TC-LSTM model, which is composed of two parts: an encoder (temporal convolution) and a decoder (LSTM). In order to verify the ability of our model, we conducted a comparative experiment. We directly use the temporal convolution model (without LSTM) to model the dance, and adopt same training strategy. We perform quantitative evaluations on two different applications, random synthesis and Music2Dance (Table 2 and 3). When the LSTMs is omitted from the decoder, the synthesized dances worsen, and especially the in realism (i.e., FID), which shows that building temporal dependencies via LSTMs in decoder improves modeling capabilities.
Result discussion. Our method realizes different dance synthesis applications (i.e., more than the three applications described above). For example, musical notation synthesis dance (i.e., to extract a melody line from musical notation). For these applications, we found a worthwhile phenomenon in the experiment. That is, the melody-consistency (i.e., controllable effect) of modern dance is superior to that in Korean dance; also, the diversity of Korean dance is inferior to modern dance. The first reason lies in the dance data. The dance steps of Korean dance in dataset are inconsistently distributed, which makes it difficult to model such unevenly distributed data. The second reason is that the rhythm and melody of Korean dance is very fast, which causes poor temporal smoothness and high dynamic complexity.
Future work. To the best of our knowledge, we are the first to propose a generative model with controllable dance synthesis via the simple and effective use of 1D control signals. However, there are some topics worth discussing: i.e., how to quantitatively evaluate the controls effectiveness, mediocre modeling ability for fastrhythm Korean dance (mentioned above), the foot sliding (it is difficult to solve by IK for complex dance motion). The topics raised here are subject of future work.

CONCLUSION
We introduced a novel generative model, TC-LSTMs. Based on temporal convolutional neural networks (CNNs) and LSTMs, TC-LSTM synthesizes realistic, diverse dances (i.e., motion sequence). Our model can handle different dance types for various applications: random synthesis, music2dance, and user control. We demonstrated quantitative results and user studies establishing the effectiveness of our method, and our model can synthesize more realistic and diverse dance motion sequences, achieving state-of-the-art results.