{'fraction_non_alphanumeric': 0.022030651340996167, 'fraction_numerical': 0.007719179625873338, 'mean_word_length': 4.94606365159129, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Analysis of Torso Movement for Signing Avatar Using Deep Learning

Avatars are virtual or on-screen representations of a human used in various roles for sign language display, including translation and educational tools. Though the ability of avatars to portray acceptable sign language with believable human-like motion has improved in recent years, many still lack the naturalness and supporting motions of human signing. Such details are generally not included in the linguistic annotation. Nevertheless, these motions are highly essential to displaying lifelike and communicative animations. This paper presents a deep learning model for use in a signing avatar. The study focuses on coordinating torso movements and other human body parts. The proposed model will automatically compute the torso rotation based on the avatar’s wrist positions. The resulting motion can improve the user experience and engagement with the avatar.


Introduction
Interactive avatars have grown popular as learning tools for spoken languages. Virtual reality has become a new tool to aid deaf or hard-of-hearing learners with specialized guidance in learning core academic concepts, such as mathematics and science (Zirzow, 2015). A signing avatar has been proposed to assist deaf students in a comprehensive educational environment (De Martino et al., 2017). Avatars are evaluated as a potential communication medium to facilitate language learning in babies (Nasihati Gilani et al., 2019). Avatars are also increasingly popular in social media, personalizing users' contributions to interacting and representing users and their behaviors. People prefer to have an avatar in their profile to secure their visual anonymity or pseudo-anonymity. Anonymity enables them to express and observe opinions they would not necessarily be comfortable with elsewhere while holding personal characteristics (Vasalou et al., 2008). Historically, people have adopted a pen name or alias to express themselves anonymously for several reasons. Deaf experience in signing online is inherently not anonymous. An avatar would help signers who do not want to reveal their identity. There is a rise of a new generation of AI avatars for speech interaction, such as Amelia, that serves as virtual cognitive assistant (Davenport et al., 2020). Deep learning capacities support her ability to learn human interaction continuously and create an engaging user experience to drive higher business value. The use of avatars in signing can be equally exciting and has potential benefits over video recordings of a sign. One can see a sign from a different angle or zoom in, or the pace or rhythm of the signing can be customized based on users' needs. The scene's background can also be adapted based on the context or better clarity of the sign (Jaballah and Jemni, 2014). The modeling and animation of signed contents generated once can automatically become reusable software components, which can be re-purposed for novel utterances. Though the need to make avatars natural is widely recognized in the animation industry, the current quality of signing avatars is still not satisfactory for producing human-like user experiences, making them less acceptable to the Deaf community (Jancso et al., 2016). Since speech is missing in sign language, supporting movements are essential to engage the communication. This research recognizes that torso movements are critical for direct linguistic communication (McDonald et al., July 8 11 2014). However, incorporating the coordination for each movement is a timeconsuming process for animators. The avatars driven by linguistic input become robotic because linguistic descriptions lack the subtleties of human motion. Motion capture can automatically incorporate natural torso support but is inflexible for generating new signing that has not been specifically recorded. Furthermore, multiple processes in signing can affect the torso simultaneously, and the effects can be difficult to separate or isolate in such recordings (Filhol et al., 2017). This paper introduces a novel application of deep learning to predict the torso movement of a signing avatar. The method will build a deep sequential neural network, implement it in the avatar and test it against the source motion capture data for validation.

Importance of Torso Motion in Signing
Analyzing and modeling the supportive motions, such as torso movements, is crucial to make the avatar mimic human movements accurately. The motions supported by the torso include reach, balance, emotion, or the turning of the body to assume participants' positions in reported speech or to indicate a side-facing object. The principles of overlapping movements are essential for the avatar to get a natural and believable feel (Burleigh et al., 2018). The following figures show three illustrations of torso movements during the signing of a scene description. In Figure 1, the signer is twisting her torso to produce a sidefacing sign, and in Figure 2, the signer is leaning her torso to the side to balance. In Figure 3, the signer is bending backward to illustrate a scene. An arm raised outwards and another arm moved across the body impacts how the torso is twisted and should be positioned to look natural. Shoulder, wrist, and hand movements must be carefully considered, especially when transitioning from one type of composition to another. One must  consider how much the wrist is bent and how the elbow is raised to orient the palm. All these specific actions can make the avatar more realistic in its movements. Modeling and automating such coordination would result in a practical, accurate, and interactive synthesis. It will elevate the avatar to drive a deeper connection with users.

Related Work
Though the natural movements of the spine are captured and can be directly replayed from motion capture data, the segmentation and synthesis of novel discourse from motion capture data is a complicated process that is the focus of ongoing research (Gibet, 2018). Many efforts for sign synthesis focus on describing sign language using a phonetic description called the Hamburg Notation System (Hanke, 2004) (Efthimiou et al., 2010). It subdivides the movements of the signer into a string of individual specifications for the parts of the body. The linguistic descriptions do not encode the motions of the torso unless they have a specific linguistic meaning (Kennaway, 2015). The Paula avatar uses a heuristic adjustment (McDonald et al., 2016) for the torso position and does not consider other features, such as the neck, shoulder, or wrist orientations. The heuristic model was created based on the artist's profi- It modeled a precise interaction of the spine and the arms to save the artist time setting up initial poses rather than general movements of the arms and torso. Furthermore, the kind of movements discussed in the last section applies to only the reaching action of the torso. There is no research yet to coordinate torso movement with hand movement in a general way for a signing avatar using a data-driven model. This paper addresses this need by studying a motion capture data set through deep learning. Neural networks (Bishop, 1994) have been used in sign language synthesis. They are employed to combine motion capture sequences for novel utterances for Japanese Sign Language (Brock et al., 2018). It is also used to classify hand positions for signing avatars (Jaballah and Jemni, 2014). Neural Networks have also been explored for their ability to generate continuous 2D skeletal signing motion based on video (Stoll et al., 2018). However, these have not considered direct 3D models of torso postures driven by the positions of the signers' hands.

Proposed Solution
This study focuses on the coordination of the torso with other body parts during the signing. The resulting framework predicts the torso movements of a signing avatar. The framework is based on a deep neural network, which learns from large motion capture (Mocap) data sets of human signers. A neural network in this context can learn and detect nonlinear relationships between independent and dependent variables. A sequential neural network model was used since it is the simplest model and can learn without prior application knowledge to find human motions (Baccouche et al., 2011). Implementing the proposed solution on the avatar will produce lifelike natural postures. Due to the opacity of the neural network for interpretation, a regression model was also trained to compare with the neural network result. This companion model aids the interpretation of the primary relationships computed by the more sophisticated black-box neural network model. This study is focused on creating a framework that will produce an improved natural movement of the torso in the avatar, which is based solely on the position and orientation of the signer's wrists. The suggested solution must be accurate compared to the actual human signer.

Data set
The LIMSI, CNRS laboratory collected human motion data for Langue des Signes Française, (LSF) in BVH format. The data were recorded with a mocap system, video, and annotations of signed descriptions of scenes elicited by a picture from human signers (Benchiheub et al., 2016). The mocap data is recorded with sensors along the spine, neck, head, shoulders, elbow, and wrist orientations. Since the signing consists of descriptions of scenes, it has very few lexical signs in the data that would differ highly from one sign language to another (Baker et al., 2016). So, even though the recorded signing is in LSF, the body postures captured are applicable across sign languages. However, this should not be generalized to all types of signers. The 3DS Max software package was used to import the motion capture data, convert the data to match the avatar's coordinate system, resolve data issues, such as outliers, derive new variables required for the ML model, and finally save the data to CSV files. Python scripts tested the data, combined all CSV files into a master file, and performed other intermediate tasks.
The data covered four signers, 25 descriptions, 25 frames per second and roughly 800 frames per description in the study. The final data set has 66644 rows and 34 columns. A specific signer was chosen to train the model to avoid confusion with different signing styles because the specific signer's style is highly consistent, while other signers use more excessive body movements.

Target definition
Three attributes for the spinal movement in the data were the primary targets: the torso's twist, side, and forward motions. The three Twist, Side, and Forward attributes across the spine bones were summed up to a derived variable to simplify the computation. It helped reduce the target or dependent variable set from 12 to only three attributes and gave a better idea of the overall movement of the spine.

Name
Action Rotation Axis Twist Transverse twisting Z-axis Side Lateral bending X-axis Forward Sagittal bending Y-axis

Linear regression
The primary motivation to start with linear regression is that it is highly interpretable and enables a better understanding of the independent variables' impact on the dependent variable. The linear regression model enabled to match the coordinate systems of the motion capture data, where the data comes from with the signing avatar, where the model is implemented. It helped to calibrate the model. It also served as the baseline model, critical for capturing the evaluation metrics before initiating the deep learning model. The steps followed from start to end are shown in Figure 4. The independent variables employed in the study are linear X, Y, and Z positions of the left and right wrists. Based on the exploratory correlation analysis shown in Figure 5 and experiments with linear regression, it was determined that chained regression between the dependent variables was appropriate. The analysis identified a linear sequence to arrange three models. The first model uses all independent variables and predicts spine twist. The second model uses all independent variables and the prediction output from the previous model to predict the spine side rotation, and so on. The chained regression approach increased the predictive power of the model significantly.
High accuracy is the main priority to make the signing avatar natural. However, the evaluation metrics from Linear Regression, such as mean squared error and R-Squared, are not satisfactory. The regression formulas for each of the three movements are displayed in equations (1) -(3).
The equations helped a more intuitive knowledge of the relationship between the independent and dependent variables, such as if wrist X increases, the twist also increases, and so on.

Applying the neural network
Deep learning techniques can train the nonlinear representation of data through multiple hidden layers. The deep learning structure can perform feature extraction and transformation without prior knowledge. Keras, an open-source neural network library (Chollet and others, 2015) was used. Keras runs on top of the TensorFlow platform (Bisong, 2019), used to run computations requiring tensors. A tensor can be considered a machine that accepts vectors as inputs and produces another vector as output. The most straightforward way to build a deep learning model in Keras is a sequential model. The sequential model is suitable for a typical stack of layers where each layer has precisely one input tensor and one output tensor. Figure 6 shows the sequential model summary used in the study. The model used a simple multi-layer perceptron with three layers with the shape of the independent variables (predictors) as a parameter. The first and second layers contain 64 units with rectified linear activation function (ReLU), and the output layer contains just one unit. The network used the "Adam" optimizer, a stochastic gradient descent method for the training model. The 'Mean Squared Error' served as the regression loss function that the model minimized during training. The model was trained on movement information from the descriptions of the first 80% of scenes and held out the rest as a test set. The training sample was used to build a deep learning model and the test sample to evaluate the model. The regression metrics reported are loss, root mean squared error (RMSE), and

Chained regression approach
As indicated by the regression models, the neural network also followed a chained approach. The first model predicts spine twist by using all independent variables. The second model uses all independent variables and the prediction output from the spine twist model to predict the spine side. The third model uses all independent variables, predicted output from the previous two models, and predicts spine forward.

Results
The models are evaluated based on the resulting predictive performance on the holdout test data using loss, RMSE, and R-Squared of the test set with 100 epochs. Table 2 shows the performance metrics of three dependent variables.

Implementation
The model is successfully implemented in the avatar using Python and tested against the original mocap positions. We scaled the torso movements to adapt the morphology of the avatar to that of the skeleton of the captured data. Examples for each of the three key spine movements are displayed in Figures 7 to 9. Naturalness is a piece of subjective information, and there is an effort to figure out how to measure it. A Figure 9: Signing avatar leaning forward the torso to depict objects user survey from the ASL community, which combines the Deaf community and the experts in the ASL domain, will be requested to compare the avatar with and without the proposed solution. The outcomes of the survey will serve as a measure of naturalness. Currently, the performance is fast enough for the avatar to respond to user interaction in real-time. This framework will be updated once future data is collected, so the model will learn using new data.

Conclusions and Future Work
This paper describes the potential power of the proposed model to compute the torso positions of the avatar, which will improve the interaction and engagement of users with the avatar. The proposed model is implemented on an avatar using motion capture data. The initial testing and validation produce satisfactory results. In sign language, signing style varies from person to person. Different signers use the torso in very distinct ways. Some signers like to move more than others. The future effort has started incorporating personal signing styles and refining the models to include additional independent variables and data. Additionally, work is in progress to create a multi-target neural network model to combine the current implementation's three models. The unified model will streamline the implementation process and may deliver better predictions than individual models. Deep neural networks have many parameters, and it is usually prone to overfitting. Since the model will soon include more independent variables, it may have overfitting issues. The companion linear regression model will be leveraged to prevent the overfit. There is a plan to handle overfitting by applying regularization techniques or tuning the neural network parameters. Though the primary focus of the study is sign language avatars, the model can be implemented in any other human animation.
There are plans to apply this framework to other sign languages, such as German or Mexican.

Acknowledgement
This paper and the study would not have been possible without the outstanding support of Dr. John McDonald. The assistance provided by Dr. McDonald, Dr. Wolfe and