{'fraction_non_alphanumeric': 0.037012654524175265, 'fraction_numerical': 0.013459147099700095, 'mean_word_length': 4.983150984682713, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Skeletal Graph Self-Attention: Embedding a Skeleton Inductive Bias into Sign Language Production

Recent approaches to Sign Language Production (SLP) have adopted spoken language Neural Machine Translation (NMT) architectures, applied without sign-specific modifications. In addition, these works represent sign language as a sequence of skeleton pose vectors, projected to an abstract representation with no inherent skeletal structure. In this paper, we represent sign language sequences as a skeletal graph structure, with joints as nodes and both spatial and temporal connections as edges. To operate on this graphical structure, we propose Skeletal Graph Self-Attention (SGSA), a novel graphical attention layer that embeds a skeleton inductive bias into the SLP model. Retaining the skeletal feature representation throughout, we directly apply a spatio-temporal adjacency matrix into the self-attention formulation. This provides structure and context to each skeletal joint that is not possible when using a non-graphical abstract representation, enabling fluid and expressive sign language production. We evaluate our Skeletal Graph Self-Attention architecture on the challenging RWTH-PHOENIX-Weather-2014T(PHOENIX14T) dataset, achieving state-of-the-art back translation performance with an 8% and 7% improvement over competing methods for the dev and test sets.


Introduction
Sign languages are rich visual languages, the native languages of the Deaf communities. Comprised of both manual (hands) and non-manual (face and body) features, sign languages can be visualised as spatio-temporal motion of the hands and body [58]. When signing, the local context of motions is particularly important, such as the connections between fingers in a sign, or the lip patterns when mouthing [44]. Although commonly represented via a graphical avatar, more recent deep learning approaches to Sign Language Production (SLP) have represented sign as a continuous sequence of skeleton poses [51,56,71]. Due to the recent success of Neural Machine Translation (NMT), computational sign language research often naively applies spoken language architectures without signspecific modifications. However, the domains of sign and spoken language are drastically different [55], with the continuous nature and inherent spatial structure of sign requiring sign-dependent architectures. Saunders et al. [49] introduced Progressive Transformers, an SLP architecture specific to a continuous skeletal representation. However, this still projects the skeletal input to an abstract feature representation, losing the skeletal inductive bias inherent to the body, where each joint upholds its own spatial representation. Even if spatio-temporal skeletal relationships can be maintained in an latent representation, a trained model may not correctly learn this complex structure.
Graphical structures can be used to represent pairwise relationships between objects in an ordered space. Graph Neural Networks (GNNs) are neural models used to capture graphical relationships, and predominantly operate on a high-level graphical structure [4], with each node containing an abstract feature representation and relationships occurring at the meta level. Conversely, skeleton pose sequences can be defined as spatio-temporal graphical representations, with both intra-frame spatial adjacency between limbs and inter-frame temporal adjacency between frames. In this work, we employ attention mechanisms as global graphical structures, with each node attending to all others. Even though there have been attempts to combine graphical representations and attention [15,61,70], there has been no work on graphical self-attention specific to a spatio-temporal skeletal structure.
In this paper, we represent sign language sequences as spatio-temporal skeletal graphs, the first SLP model to operate with a graphical structure. As seen in the centre of Figure 1, we encode skeletal joints as nodes, J (blue dots), and natural limb connections as edges, E, with both spatial (blue lines) and temporal (green lines) relationships. Operating on a graphical structure explicitly upholds the skeletal representation throughout, learning deeper and more informative features than using an abstract representation.  Figure 1. An overview of our proposed SLP network, showing an initial translation from a spoken language sentence using a text encoder, with gloss supervision. A subsequent skeletal graphical structure is formed, with multiple proposed Skeletal Graph Self-Attention layers applied to embed a skeleton inductive bias and produce expressive sign language sequences. Additionally, we propose Skeletal Graph Self-Attention (SGSA), a novel spatio-temporal graphical attention layer that embeds a hierarchical body inductive bias into the selfattention mechanism. We directly mask the self-attention by applying a sparse adjacency matrix to the weights of the value computation, ensuring a spatial information propagation. To the best of our knowledge, ours is the first work to embed a graphical structure directly into the self-attention mechanism. In addition, we expand our model to the spatiotemporal domain by modelling the temporal adjacency only on N neighbouring frames.
Our full SLP model can be seen in Figure 1, initially translating from spoken language using a spoken language encoder with gloss supervision. The intermediary graphical structure is then processed by a graphical sign language decoder containing our proposed SGSA layers, with a final output of sign language sequences. We evaluate on the challenging RWTH-PHOENIX-Weather-2014T (PHOENIX14T) dataset, performing spatial and temporal ablation studies of the proposed SGSA architecture. Furthermore, we achieve state-of-the-art back translation results for the text to pose task, with an 8% and 7% performance increase over competing methods for the development and test sets respectively.
The contributions of this paper can be summarised as: • The first SLP system to model sign language as a spatio-temporal graphical structure, applying both spatial and temporal adjacency.
• A novel Skeletal Graph Self-Attention (SGSA) layer, that embeds a skeleton inductive bias into the model.
• State-of-the-art Text-to-Pose SLP results on the PHOENIX14T dataset.

Related Work
Sign Language Production The past 30 years has seen extensive research into computational sign language [35,62]. Early work focused on isolated Sign Language Recognition (SLR) [20,36], with a subsequent move to continuous SLR [6,28]. The task of Sign Language Translation (SLT) was introduced by Camgoz et al. [7] and has since become a prominent research area [8,46,68]. Sign Language Production (SLP), the automatic translation from spoken language sentences to sign language sequences, was initially tackled using avatar-based technologies [17,38]. The rulebased Statistical Machine Translation (SMT) achieved partial success [26,31], albeit with costly, labour-intensive preprocessing.
Recently, there have been many deep learning approaches to SLP proposed [23,42,48,50,52,56,63,71], with Saunders et al. achieving state-of-the-art results with gloss supervision [52]. These works predominantly represent sign languages as sequences of skeletal frames, with each frame encoded as a vector of joint coordinates [51] that disregards any spatio-temporal structure available within a skeletal representation. In addition, these models apply standard spoken language architectures [60], disregarding the structural format of the skeletal data. Conversely, in this work we propose a novel spatio-temporal graphical attention layer that injects an inductive skeletal bias into SLP.
Graph Neural Networks A graph is a data structure consisting of nodes, J , and edges, E, where E defines the relationships between J . Graph Neural Networks (GNNs) [4] apply neural layers on these graphical structures to learn representations [45,74], classify nodes [64,67] or generate new data [34,66]. A skeleton pose representation can be structured as a graph, with joints as J and natural limb connections as E [53,57]. GNNs have been proposed for operating on such dynamic skeletal graphs, in the context of action recognition [25,40,53,64] and human pose estimation [57].
Attention networks can be formalised as a fully connected GNN, where the adjacency between each word, E, is a weighting learnt using self-attention. Expanding this, Graph Attention Networks (GATs) [61] define explicit weighted adjacency between nodes, achieving state-of-theart results across multiple domains [5,30,54]. Recently, there have been multiple graphical transformer architectures proposed [15,29,33,70], which have been extended to the spatio-temporal domain for applications such as multiple object tracking [13] and pedestrian tracking [69].
However, there has been no work on graphical attention mechanisms where the features of each time step holds a relevant graphical structure. We build a spatio-temporal graphical architecture that operates on a skeletal representation per frame, explicitly injecting a skeletal inductive bias into the model. There have been some applications of GNNs in computational sign language in the context of SLR [14,18,24,41,59]. We extend these works to the SLP domain with our proposed Skeletal Graph Self-Attention architecture.
Local Attention Attention mechanisms have demonstrated strong Natural Language Processing (NLP) performance [2], particularly with the introduction of transformers [60]. Although proposed with global context [2], more recent works have selectively restricted attention to only a local context [12,37,65] or the top-k tokens [73], often due to computational issues or to enable long-range dependencies. In this paper, we propose using local attention to represent temporal adjacency within our graphical skeletal structure.

Background
In this section, we provide a brief background on selfattention. Attention mechanisms were initially proposed to overcome the information bottleneck found in encoderdecoder architectures [2,39]. Transformers [60] apply multiple scaled self-attention layers in both encoder and decoder modules, where the input is a set of queries, Q ∈ R d k , and keys, K ∈ R d k , and values, V ∈ R dv . Selfattention aims to learn a context value for each time-step as a weighted sum of all values, where the weight is determined by the relationship of the query with each corresponding key. An associated weight vector, W Q/K/V , is first applied to each input, as shown in Figure 2, as: are weights related to each input variable and d model is the dimensionality of the self-attention layer. Formally, scaled self-attention (SA) outputs a weighted vector combination of values, V W , by the relevant queries, Q W , keys, K W , and dimensionality, d k , as: Multi-Headed Attention (MHA) applies h parallel attention mechanisms to the same input queries, keys and values, each with different learnt parameters. In the initial architecture [60], the dimensionality of each head is proportionally smaller than the full model, d h = d model /h. The output of each head is then concatenated and projected forward, as: where W O ∈ R d model ×d model . In this paper, we introduce Skeletal Graph Self-Attention layers that inject a skeletal inductive bias into the self-attention mechanism.

Methodology
The ultimate goal of SLP is to automatically translate from a source spoken language sentence, X = (x 1 , ..., x T ) with T words, to a target sign language sequence, G = (g 1 , ..., g U ) of U time steps. Additionally, an intermediary gloss 1 sequence representation can be used, Z = (z 1 , ..., z P ) with P glosses. Current approaches [51,56,71] predominantly represent sign language as a sequence of skeletal frames, with each frame containing a vector of body joint coordinates. In addition, they project this skeletal structure to an abstract representation before being processed by the model [49]. However, this approach removes all spatial information contained within the skeletal data, . We propose novel Skeletal Graph Self-Attention layers to operate on the sign language skeletal graphs, G.
restricting the model to only learning the internal relationships within a latent representation. Contrary to previous work, in this paper we represent sign language sequences as spatio-temporal skeletal graphs, G, as in the centre of Figure 1. As per graph theory [3], G can be formulated as a function of nodes, J and edges, E. We define J as the skeleton pose sequence of temporal length U and spatial width S, with each node representing a single skeletal joint coordinate from a single frame (blue dots in Fig. 1). S is therefore the dimensionality of the skeleton representation of each frame. E can be represented as a spatial adjacency matrix, A, defined as the natural limb connections between skeleton joints both of its own frame (blue lines) and of neighbouring frames (green lines).
As outlined in Sec. 3, classical self-attention operates with global context over all sequence time-steps. However, a skeletal inductive bias can be embedded into a model by restricting attention to only the natural limb connections within the skeleton. To embed a skeleton inductive bias into self-attention, we propose a novel Skeletal Graph Self-Attention (SGSA) layer that operates with sparse attention. Modeled within a transformer decoder, SGSA retains the original skeletal structure throughout multiple deep layers, ensuring the processing of spatio-temporal information contained in skeletal pose sequences. In-built adjacency matrices of both intra-and inter-frame relationships provide structure and context directly to each skeletal joint that is not possible when using a non-graphical abstract representation.
In the rest of this section, we outline the full SLP model, containing a spoken language encoder and a graphical sign language decoder, with an overview shown in Figure 3.

Spoken Language Encoder
As shown on the left of Figure 3, we first translate from a spoken language sentence, X , of dimension E × T , where E is the encoder embedding size, to a sign language rep-resentation, R = (r 1 , ..., r U ) (Fig. 1 Left). We build a classical transformer encoder [60] that applies self-attention using the global context of a spoken language sequence. R is represented with a spatio-temporal structure, containing identical temporal length, U, and spatial shape, S, as the final skeletal graph, G. This structure enables a graphical processing by the proposed sign language decoder. Additionally, as proposed in [52], we employ a gloss supervision to the intermediate sign language representation. This prompts the model to learn a meaningful latent sign representation for the ultimate goal of sign language production.

Graphical Sign Language Decoder
Given the intermediary sign language representation, R ∈, we build an auto-regressive transformer decoder containing our novel Skeletal Graph Self-Attention (SGSA) layers (Figure 3 middle). This produces a graphical sign language sequence,Ĝ, of spatial shape, S, and temporal length, U.
Spatial Adjacency We define a spatial adjacency matrix, A ∈ R S×S , expressed as a sparse attention map, as seen in Figure 4. A contains a spatial skeleton adjacency structure, modelled as the natural skeletal limb connections within a frame (blue lines in Fig. 1). A can be formalised as: where Con(i, j) = T rue if joints i and j are connected. For example, the skeletal elbow joint is connected to the skeletal wrist joint. We use an undirected graph representation, defining E as bidirectional edges.
Temporal Adjacency We expand the spatial adjacency matrix to the spatio-temporal domain by modelling the inter-frame edges of the skeletal graph structure (green lines in Fig. 1). The updated spatial-temporal adjacency matrix can be formalised as A ∈ R S×S×U . We set N as the temporal distance that defines 'adjacent', where edges are established as both same joint connections and natural limb connections between the N adjacent frames. In the standard attention shown in Sec. 3, each time-step can globally attend to all others, which can be modelled as N = ∞. We formalise our spatio-temporal adjacency matrix, as: where t is the temporal distance from the reference frame, t = u − u ref .

Self-loops and Normalisation
To account for information propagation loops back to the same joint [3], we add self-loops to A using the identity matrix, I ∈ R S×S . In practice, due to our multi-dimensional skeletal representation, we add self-loops from each coordinate of the joint both to itself and all other coordinates of the same joint, which we define as I * ∈ R S×S . Furthermore, to prevent numerical instabilities and exploding gradients [3], we normalise the adjacency matrix by inversely applying the degree matrix, D ∈ R S . D is defined as the numbers of edges a node is connected to. Normalisation can be formulated as: where A * is the normalised adjacency matrix.
Skeletal Graph Self-Attention We apply A * as a sparsely weighted mask onto the weighted value calculation, V W = V · W V , of Eq. 1, ensuring that the values used in the weighted context for each node is only impacted by the adjacent nodes of the previous layer: where Figure 4 shows a visual representation of the sparse adjacent matrix A * containing spatio-temporal connections, applied as a mask to the weighted calculation. With a value matrix containing a skeletal structure, V ∈ R S , A * restricts the information propagation of self-attention layers only through the spatial and temporal skeletal edges, E, and thus embeds a skeleton inductive bias into the attention mechanism.
We formally define a Skeletal Graph Self-Attention (SGSA) layer by plugging both the weighted variable computation of Eq. 1 and the adjacent weighted computation of Eq. 7 into the self-attention Eq. 2, as: where d model = S. This explicitly retains the spatial skeletal shape, S, throughout the sign language decoder, enabling a spatial structure to be extracted.
To extend this to a multi-headed transformer decoder, we replace self-attention in Eq. 3 with our proposed SGSA layers. To retain the spatial skeletal representation within each head, the dimensionality of each head is kept as the full model dimension, d h = d model = S, with the final projection layer enlarged to h × S.
We build our auto-regressive sign language decoder with L multi-headed SGSA sub-layers, interleaved with fullyconnected layers and a final feed-forward layer, each with a consistent spatial dimension of S. A residual connection [22] and subsequent layer norm [1] is employed around each of the sub-layers, to aid training. As shown on the right of Figure 3, the final output of our sign language decoder module is a graphical skeletal sequence,Ĝ, that contains U frames of skeleton pose, each with a spatial shape of S.
We train our sign language decoder using the Mean Squared Error (MSE) loss between the predicted sequence, G, and the ground truth sequence, G * . This is formalised as L MSE = 1 U u i=1ĝ 1:U −g * 1:U ) 2 , whereĝ and g * represent the frames of the produced and ground truth sign language sequences, respectively. We train our full SLP model endto-end with a weighted combination of the encoder gloss supervision [52] and decoder skeleton pose losses.  Table 1. Impact of Skeletal Graph Self-Attention layers, L, on model performance.

Sign Language Output
Generating a sign language video from the produced graphical skeletal sequence,Ĝ, is then a trivial task, animating each frame in temporal order. Frame animation is done by connecting the nodes, J , using the natural limb connections defined by E, as seen in Fig. 1.

Experiments
Dataset We evaluate our approach on the PHOENIX14T dataset introduced by Camgoz et al. [7], containing parallel sequences of 8257 German sentences, sign gloss translations and sign language videos. Other available sign datasets are either simple sentence repetition tasks of nonnatural signing not appropriate for translation [16,72], or contain larger domains of discourse that currently prove difficult for the SLP field [10,21]. We extract 3D skeletal joint positions from the sign language videos to represent our spatio-temporal graphical skeletal structure. Manual and non-manual features of each video are first extracted in 2D using OpenPose [11], with the manuals lifted to 3D using the skeletal model estimation model proposed in [71]. We normalise the skeleton pose and set the spatial skeleton shape, S, as 291, with 290 joint coordinates and 1 counter decoding value (as in [49]). Adjacency information, A, is defined as the natural limb connections of 3D body, hand and face joints, as in [71], where each coordinate of a joint is adjacent to both the coordinates of its own joint and all connected joints. We define the counter value as global adjacency, with connections to all joints.

Implementation Details
We setup our SLP model with a spoken language encoder of 2 layers, 4 heads and an embedding size, E, of 256, and a graphical sign language decoder of 5 layers, 4 heads and an embedding size of S. Our best performing model contains 9M trainable parameters. As proposed by Saunders et al. [49], we apply Gaussian noise augmentation with a noise rate of 5. We train all parts of our network with Xavier initialisation [19], Adam optimization [27] with default parameters and a learning rate of 10 −3 . Our code is based on Kreutzer et al.'s NMT toolkit, JoeyNMT [32], and implemented using PyTorch [43].
Evaluation We use the back translation metric [49] for evaluation, which employs a pre-trained SLT model [9] to translate the produced sign pose sequences back to spoken language. We compute BLEU and ROUGE scores against the original input, with BLEU n-grams from 1 to 4 provided. The SLP evaluation protocols on the PHOENIX14T dataset have been set by [49]. We share results on the Text to Pose (T2P) task which constitutes the production of sign language sequences directly from spoken language sentences, the ultimate goal of an SLP system. We omit Gloss to Pose evaluation to focus on the more important spoken language translation task. Skeletal Graph Self-Attention Layers We start our experiments on the proposed Skeletal Graph Self-Attention layers, evaluating the effect of stacking multiple SGSA layers, L, each with a multi-head size, h, of 4. We first ablate the effect of using no SGSA layers, and replacing them with 4 standard self-attention layers, as described in Section 3. We then build our graphical sign language decoder with 1 to 5 SGSA layers, with each model retaining a constant spoken language encoder size and a global temporal adjacency. Table 1 shows that using standard self-attention layers achieves the worst performance of 14.25 BLEU-4, showing the benefit of our proposed SGSA layers. Increasing the number of SGSA layers, as expected, increases model performance to a peak of 14.72 BLEU-4. A larger number of layers enables a deeper representation of the skeletal graph and thus provides a stronger skeleton inductive bias to the model. In lieu of this, for the rest of our experiments we build our sign language decoder with five SGSA layers.
Temporal Adjacency In our next set of experiments, we examine the impact of the temporal adjacency distance, N , defined in Sec. 4.2. In order to logically set N , we analyse the trained temporal attention matrix of the best performing decoder evaluated above. We notice that the attention predominantly falls on the last 3 frames, as the model learns to attend to the local temporal context of skeletal motion. Manually restricting the temporal attention provides this information as an inductive bias into the model, rather than  Table 2. Impact of Temporal Adjacency, N , on SGSA model performance relying on this being learnt. Table 2 shows results of our temporal adjacency evaluation, ranging from an infinite adjacency (no constraint) to N ∈ [1,5]. It can be seen that a temporal adjacency distance of one achieves the best BLEU-4 performance. Note: Although we report BLEU of n-grams 1-4 for completeness, we use BLEU-4 as our final evaluation metric to enable a clear result. Although counter-intuitive to the global self-attention utilised by a transformer decoder, we believe this is modelling the Markov property, where future frames only depend on the current state. Due to the intermediary gloss supervision [52], the defined sign language representation, R, should contain all frame-level information relevant to a sign language translation. The sign language decoder then has the sole task of accurately animating each skeletal frame. Therefore, a single temporal adjacency in the graphical decoder makes sense, as no new information is required to be learnt from temporally distant frames.

Baseline Comparisons
We compare the performance of the proposed Skeletal Graph Self-Attention architecture against 4 baseline SLP models: 1) Progressive transformers [49], which applied the classical transformer architecture to sign language production. 2) Adversarial training [47], which utilised an adversarial discriminator to prompt more expressive productions, 3) Mixture Density Networks (MDNs) [51], which modelled the variation found in sign language using multiple distributions to parameterise the entire prediction subspace, and 4) Mixture of Motion Primitives (MOMP) [52], which split the SLP task into two distinct jointly-trained sub-tasks and learnt a set of motion primitives for animation. Table 3 presents Text to Pose results, showing that SGSA achieves 15.15/14.33 BLEU-4 for the development and test sets respectively, an 8/7% improvement over the state-of-the-art. These results highlight the significant success of our proposed SGSA layers. We have shown that representing sign pose skeletons in a graphical skeletal structure and embedding a skeletal inductive bias into the self-attention mechanism enables a fluid and expressive sign language production.

Conclusion
In this paper, we proposed a skeletal graph structure for SLP, with joints as nodes and both spatial and temporal connections as edges. We proposed a novel graphical attention layer, Skeletal Graph Self-Attention, to operate on the graphical skeletal structure. Retaining the skeletal feature representation throughout, we directly applied a spatio-temporal adjacency matrix into the self-attention formulation, embedding a skeleton inductive bias for expressive sign language production. We evaluated SGSA on the challenging PHOENIX14T dataset, achieving state-of-theart back translation performance with an 8% and 7% improvement over competing methods for the dev and test set. For future work, we aim to apply SGSA layers to the wider computational sign language tasks of SLR and SLT.

Acknowledgements
This work received funding from the SNSF Sinergia project 'SMILE' (CRSII2 160811), the European Union's Horizon2020 research and innovation programme under grant agreement no. 762021 'Content4All' and the EPSRC project 'ExTOL' (EP/R03298X/1). This work reflects only the authors view and the Commission is not responsible for any use that may be made of the information it contains. DEV Table 3. Baseline comparisons on the PHOENIX14T dataset for the Text to Pose task.