{'fraction_non_alphanumeric': 0.03735282176207876, 'fraction_numerical': 0.01619975639464068, 'mean_word_length': 4.69898195279963, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Dicta-Sign-LSF-v2: Remake of a Continuous French Sign Language Dialogue Corpus and a First Baseline for Automatic Sign Language Processing

While the research in automatic Sign Language Processing (SLP) is growing, it has been almost exclusively focused on recognizing lexical signs, whether isolated or within continuous SL production. However, Sign Languages include many other gestural units like iconic structures, which need to be recognized in order to go towards a true SL understanding. In this paper, we propose a newer version of the publicly available SL corpus Dicta-Sign, limited to its French Sign Language part. Involving 16 different signers, this dialogue corpus was produced with very few constraints on the style and content. It includes lexical and non-lexical annotations over 11 hours of video recording, with 35000 manual units. With the aim of stimulating research in SL understanding, we also provide a baseline for the recognition of lexical signs and non-lexical structures on this corpus. A very compact modeling of a signer is built and a Convolutional-Recurrent Neural Network is trained and tested on Dicta-Sign-LSF-v2, with state-of-the-art results, including the ability to detect iconicity in SL production.


Introduction
A great number of sign languages have been naturally developed within Deaf communities, and even the most widely-spoken are still to be thoroughly described in terms of linguistics (Braffort, 2016). They all have their own number of very conventionalized signs as well as highly iconic structures, such as classifiers constructions (Liddell, 1977) used to depict common entities (see Fig. 1). While the research in automatic Sign Language Processing (SLP) has focused on recognizing lexical signs which are by definition very conventionalized, iconicity is usually ignored even though it is crucial for SL understanding. In this paper, three major contributions to the field of continuous SLP are presented: 1. A public remake of the French Sign Language (LSF) part of the Dicta-Sign Corpus (Matthes et al., 2012), with cleaned and reliable annotations. These annotations include lexical data and more refined linguistic categories. As the corpus is based on dialogue with very loose elicitation guidelines, it is highly representative of natural SL.
2. The manufacturing of a relevant modeling of a signer as a proper input to a convolutional-recurrent neural network (CRNN) for automatic SLP. This modeling is made compact, and easily generalizable to any SL recording. Preprocessed data is also made public; 3. We prove the relevance of our modeling and network on Dicta-Sign-LSF-v2. State-of-the-art accuracy for the classification of video frames into a number of manual unit types is also attained on NCSLGR, a similar -although elicited with more constraints -American Sign Language corpus.
This paper is organized as follows: in Section 2, the linguistics of SLs is discussed and shortcomings of current research are pointed out, then different categories of datasets and associated research in SLP are analyzed. Section 3 introduces Dicta-Sign-LSF-v2: its elicitation and recording conditions, its annotations and key statistics. In Section 4, a first baseline method for automatic SLP is laid out and experimented on Dicta-Sign-LSF-v2 in Section 5. In the same section, the learning model is also tested on another SL corpus for a comparison with existing results.

Related work and limitations 2.1. Sign Language Linguistics
In this section, some fundamental aspects of SL linguistics are discussed. Conversely to written English for instance, in which every word is conventionalized and the grammar is structured by the use of some specific words, SLs are not fully described by their lexicon. Lexical signs do exist, and can be defined as follows, from Johnston and Schembri (2007): "fullylexical signs are highly conventionalised signs in both form and meaning in the sense that both are relatively stable or consistent across contexts. Fully-lexical signs can easily be listed in a dictionary". However, the hypothesis -usually used but unstated in the field of SLP -that SL production can be reduced to a se-quence of lexical signs is wrong. Indeed, some characteristics of SLs make them fundamentally different from unidimensional sequential languages: • They are multi-channel: information is conveyed through hand motion, shape and orientation, body posture and motion, facial expression and gaze; • They are strongly spatially organized: events, objects, people and other entities are placed in the signing space and related to each other in a visual way. The grammar of SLs is structured by the use of space; • They allow signers to generate new signs -that would not appear in a dictionary -on the go, in an iconic way, or even modify lexical signs. More generally, SL do not only consist of lexical signs but they also make use of more complex iconic structures: classifiers, pointing signs, buoys (see Section 3.2 for more detail). As one can appreciate on the random example of Fig. 1, a SL utterance can be completely iconic.
Thus, SL production should not be seen and analyzed as a succession of lexical signs. Classifiers and other gestural units like pointing -used to link entities in a SL production -are crucial in the visual grammar of SLs.

Sign Language Datasets and Recognition
One of the most basic SLP tasks consists in the recognition of isolated lexical signs. The following datasets have been used for this task, and are all publicly available: • ASLLVD  consists of 2284 ASL isolated lexical signs, realized by 6 signers, with a sign count of 9800. Different views are provided, with RGB recording at 60 fps.
• Devisign-L (Chai et al., 2015) covers 2000 CSL (Chinese Sign Language) lexical signs, realized by 8 signers, with a sign count of 24000. Videos are recorded frontally, with RGBD and skeleton data provided.
• MS-ASL (Joze and Koller, 2018) is a collection of 1000 isolated lexical signs from YouTube videos, with 222 signers and a total sign count of 25513.
Continuous Sign Language Recognition (CSLR) is a more challenging task. Many datasets have been made public, although most of them only include lexical annotations, and are quite artificial in the sense that they consist of simple elicited sentences: • Purdue RVL-SLLL (Wilbur and Kak, 2006) results from the elicitation of pre-defined "paragraphs" (1-2 sentences) from 14 signers, each of them realizing 10 so-called paragraphs. RGB data is provided, and 104 lexical signs are annotated.
• CSL-25k (Huang et al., 2018) results from the elicitation of pre-defined sentences from 50 signers, with 178 annotated CSL lexical signs and more than 100 hours of RGBD video. Skeleton data is also given.
• Signum (Von Agris and Kraiss, 2007) is a DGS (German Sign Language) dataset, with 5 hours of RGB video from 25 signers. Pre-defined sentences are elicited, with 465 lexical signs.
• RWTH Phoenix (Forster et al., 2014) is made from 11 hours of live DGS interpretation of weather forecast on German TV. It has established itself as a reference dataset for SLR  even though the language variability and complexity are quite limited. Furthermore, one should note that interpreted SL is different from spontaneous SL: there is a good chance that the translation will be strongly influenced by the original speech (in German), especially in terms of syntax, and make little use of the structures typical of SL (Section 2.1). Also, RGB frame resolution is low at 210 × 260. A signer-independent version was released in Koller et al. (2017).
At the other end of the spectrum, some datasets are -at least partially -made of natural sign language, often in the form of conversations or narratives, and include annotations other than lexical. For instance: • Corpus NGT (Zwitserlood et al., 2008, Sign Language of the Netherlands, 72 hours) • BSL Corpus Project (Schembri, 2008, British Sign Language) • DGS Korpus (Prillwitz et al., 2008, 50 hours) • Auslan Corpus (Johnston, 2009, Australian Sign Language, 300 hours) • NCSLGR (Neidle and Vogler, 2012, ASL, 2 hours) • Corpus LSFB (Meurant et al., 2016, French Belgian Sign Language, 150 hours) are continuous SL RGB datasets and include annotations like "Depicting signs", "Pointing signs" and "Buoys" (cf. Section. 3.2). However, annotations are often incomplete and these datasets are usually ignored in the field of automatic SLR. To the authors knowledge, there is only one experiment of extra-lexical automatic Sign Language Processing on a continuous SL dataset in the literature (Yanovich et al., 2016, see Section 5.1 for more detail). Therefore, a continuous French Sign Language corpus, including very natural conversations on a wide-ranging subject, annotated on lexical and extra-lexical levels, is presented in the next section and will be used to evaluate SLP models.

Dicta-Sign-LSF-v2
This section presents a remake of the LSF part of the Dicta-Sign Corpus (Matthes et al., 2012).

Elicitation and recording conditions
Dicta-Sign-LSF-v2 is a dialogue corpus: two signers face each other, with one camera above each of them, and a third camera on the side. In order to ensure consistency, these three views are released in Dicta-Sign-LSF-v2, with identical resolution (720 × 575 at 25 fps). Other views or better Eight couples were formed, and each couple was given nine different tasks, on the common theme of "travel". Since elicitation was quite loose, conversation was very direct, with pauses, laughter, hesitance etc. occurring naturally.

Annotations
Focusing on the manual activity, the corpus was annotated within three main annotation categories, according to the classification of Johnston and De Beuzeville (2014): Fullylexical signs (FLS), Partially-lexical signs (PLS) and Nonlexical signs (NLS). Other structures like constructed action/dialogue 1 (or role shift) were not annotated, or only partially. An illustration sequence is shown on Fig. 3, with more detail on the annotations in associated Table 1.

Fully Lexical Signs (FLS)
FLS form the basic lexicon of SL, as defined in Section 2.1. They only account for a fraction of what can be analyzed in SL production.

Partially Lexical Signs (PLS)
PLS are annotated into three sub-categories: • Pointing signs (PT), as mentioned in the name, are used to point towards an entity in the signing space, that is to link what is said to a spatial referent. Since SL are spatially organized, they are of prime importance to understand a discourse.
• Depicting Signs (DS) form a broad category of signs, the structure of which is easily identified. They are also referred to as classifier signs, classifier constructions or classifier predicates (see Liddell (1977)). Their definition is close to what is called highly iconic signs in Cuxac (2000). They are used to describe the location, motion, size, shape or the action of an entity, along with trajectories in the signing space. They sometimes consist in the tweaking/enrichment of a lexical sign.
1 From Johnston and De Beuzeville (2014): enactment of the external physical actions or behaviour of a character • Fragment buoys are hand shapes held in the signing space (usually on the weak hand) while signing activity continues on the other hand (Liddell, 2003). They can be seen as a referent, and can be used for specific linguistic functions, like what was called qualification/naming structures in Filhol and Braffort (2012).

Non Lexical Signs (NLS)
NLS comprise fingerspelling (FS), numbering (N) and gestures that are not typically specific to SL and can be culturally shared with non SL signers (i.e. speakers).

Statistics
There are 16 signers in the dataset, with 94 fully annotated videos -one for each signer and each task. In total, 11 hours are annotated, that is 1007593 frames. However, one should note that since each signer is recorded continuously during dialogue, approximately half of these 11 hours correspond to one person looking at the other person signing. Table 2 presents the detailed frame and manual unit count for each of the main annotated categories, along with their cumulative distribution. Fully-lexical signs, depicting signs and pointing signs account for 93.6% of non blank frames, and 97.6% of all manual units.

Baseline SLP method
In this section, a first Sign Language Processing model is presented. It is based upon a compact and generalizable modeling of a signer, and can be used to learn the detection and recognition of different types of linguistic annotations.

Generalizable signer modeling
In this section, the manufacturing of relevant upper body pose, face and hand features is described.  The left hand is being used as a fragment buoy -here a fragment of the tower -, which is a non-lexical SL function helping the interlocutor understand that what is being said still relates to the same scene.
The right hand has a typical hand shape for the legs of a person -known as a proform -and depicts a straight motion from the bottom to the middle of the tower, indicating the action of using the elevator. It is annotated as a depicting sign of motion type. 7 Pointing sign to a precise location, at the middle of the tower. It indicates the location of what is going to be introduced. The left hand is still used as a fragment buoy.

8, 9
Both hands are used to depict an outer shape -thus annotated as depicting sign of shape type. Its base is a square, and it is rather slim (which is stressed by the crinkled eyes, even though not in the scope of the annotations).

10
Lexical sign "Restaurant". The location and shape that have just been described apply to it.  Fig. 3. It points out many key elements necessary to understand this sequence -namely lexicon, buoys, proforms, pointing, iconic structures and spatial structure.
• Position, speed and acceleration of: each hand w.r.t. body center, parent elbow and face center; one hand w.r.t. other hand; each elbow w.r.t. parent shoulder; each shoulder w.r.t. body center; • Angle and orientation of elbows/shoulders, and 1 st /2 nd order derivatives.

Head pose
A 3D face estimate is directly obtained from video frames thanks to a CNN model trained on 230, 000 images (Bulat and Tzimiropoulos, 2017). Handcrafted features are used instead of raw data -the head feature vector is of size 16: • Euler angles, speed and acceleration for axes X, Y and Z of the centroid of the head; • Horizontal and vertical mouth openness; relative motion of the eyebrows to the eyes; • Nose to body center distance.

Hand modeling
A lot of information in SL production, if not most of it, is conveyed through the hands. More specifically, the location, shape and orientation of both hands are critical, along with the dynamics of these three variables, that is: hand trajectory, shape deformation and hand rotation. Ideally, one would greatly benefit from a frame-wise 3D hand pose estimate on RGB images. Although such algorithms have been developed (Xiang et al., 2018), they have not proven reliable on real-life 25 fps SL videos -Open-Pose, which also provides a 2D estimate on hand pose, faces the same issue. Indeed, because hands move constantly and relatively fast in SL, motion blur makes the frame-wise estimation of hand pose very difficult.

Public access
Along with video files, metadata and annotation data, all previously defined preprocessed body, face and hand data is publicly accessible at https://www.ortolang.fr/ market/item/dicta-sign-lsf-v2 (Belissen et al., 2019). Some more detail on the computation of this data is included.

Learning network
This section outlines the architecture of the CRNN that was built to learn a generalizable gesture representation, in order to learn the detection and recognition of different SL features.
With time t = 1 . . . T , the input and output sequences are defined as follows: x t as a flattened input vector, its size N in f corresponding to the total number of pre-computed motion features (computed from body pose and facial landmarks) and hand features (see Section 4.1); Y t as the output of the model -it consists of N out f predictions, one for each output type of the model. The model then learns the frame-wise conditional probabilities with framewise categorical cross-entropy as training loss.
The architecture of the model starts with a convolutional layer applied on the input x t , and helps build a first set of temporal features. Then, recurrent Long Short-Term Memory (LSTM) layers are added to the network -LSTM units handle vanishing gradient issues (Gers et al., 2000), which in the case of high frequency data is critical. Since this work does not target real-time applications, the recurrent layers are bidirectional. Dropout is used to prevent overfitting in the LSTM layers (Srivastava et al., 2014). Output layers use softmax activation, while other layers use Rectified Linear Unit (ReLU). RMSProp optimizer is used, and the network is built with Keras (Chollet and others, 2015) on top of Tensorflow (Abadi et al., 2016).

Experiments
In this section, the learning model is tested on Dicta-Sign-LSF-v2. When possible, results are also compared with another learning model and dataset (see Section 5.1).

Manual unit classification
5.1.1. Comparison with a unique reference point A first experiment consists in the classification of video frames into a small number of types of manual units. Yanovich et al. (2016) developed a multiple instance learning (MIL) model of this kind, and trained and tested it on the NCSLGR ASL corpus (Neidle and Vogler, 2012) -in a signer-dependent fashion. The model is trained on video frames that are classified into three main categories -lexical signs, fingerspelling, depicting signs. For some unknown reason, one shortcoming is that the prediction model never outputs blank frames. That is, frames will automatically be misclassified when there is no annotation or when there is an annotation of another type. The claimed accuracy of the model is 91.27%, but it is computed on non blank frames: when considering the whole dataset it drops to 44.0%. Using the same three annotation categories while including a fourth one dedicated to blank frames, the CRNN learning model presented in Section 4 is trained and tested on NCSLGR. The dataset was split 70%-20%-10% (trainingvalidation-test) in a random fashion. Results are given on Table 3. The CRNN model gets 87.9% accuracy on the whole NCSLGR dataset (91.8% when ignoring blank frames).

A relevant baseline for LSF
In order to establish a more relevant baseline for LSF, another frame classification model is trained, with three to six annotation categories, following their distribution given in Table 2. For instance, fingerspelling is much more frequent in ASL than in LSF, so it should not be included when considering only three categories. Models are trained and tested either in a signer-independent fashion or not -signer-independent makes learning more difficult while increasing model generalizability. See Table 3 for an overview of the results. A test sequence illustration with framewise probabilities is given on Fig. 5. This example shows that the model is able to accurately classify and segment most signs, even when only one blank frame separates them.

Detection of specific signs
A second experiment tackles the problem of detecting specific signs. The learning model is trained for one annotation type at a time, with binary cross-entropy as training loss. F 1-score is used, from the evaluation of true positives, true negatives and false positives within a sliding window of four seconds length. The corpus was split 70%-20%-10% respectively for training, validation and testing sets, in a signer-independent fashion. Two lexical signs are considered -"Same" (208 training instances) and "Line" (47 training instances) -, as well as pointing signs, depicting signs, fragment buoys and fingerspelling. Advantage is also taken from this experiment to evaluate the relevance of using the hand shape classifier and 3D modeling that are detailed in Section 4.1. Results are given on Table 4. The benefit of the modeling presented in Section 4.1 over a simple OpenPose estimate is clear: the performance of every output is improved. The models performs rather poorly on fragment buoys and fingerspelling, which can be explained by the unclear annotation guidelines for the former, and too few training examples for the latter. A few factors are known to lessen model performance, including: • The chopping of sequences at the middle of signs; • A non fully optimized network architecture; • An imperfect modeling, especially regarding hands; • Annotation errors and bias or subjectivity; • A great variability between signers; • A lot of variability between signs because of the continuous nature of the corpus.
• More generally, the continuous nature of Sign Language in itself that makes it very difficult to classify its parameters into a finite number of categories.

Measuring the iconicity
Although the annotations for depicting signs are binary (signs are either annotated as depicting or they are not), a continuous transition from very conventionalized fully lexical signs to completely iconic structures can be observed in LSF. Cuxac (2000) refers to these two ends as saying without showing and saying with showing. In this third experiment, we illustrate with a few exemples that the learning model, when fed with annotations corresponding to depicting signs, seems to be learning a measure of iconicity in SL production.
For that purpose, the setting is identical to that of Section 5.2: binary cross-entropy as training loss with depicting signs as the only output of the learning model, signerindependent 70%-20%-10% split of the dataset, 3D+HS signer modeling. Fig. 6 presents four excerpts from a LSF sequence of Dicta-Sign-LSF-v2. Most annotated depicting signs are accurately detected, and many "false positives" actually seem to reveal a certain degree of iconicity that was not annotated. Provided these preliminary results are proven true in a future thorough analysis, this model could actually be used to build more appropriate linguistic categories of signs, which are still part of continuing discussions amongst linguists. Although this analysis will require further examination, it can only be conducted on datasets that include such annotations, that is very few of them.

Conclusion
In this paper, we introduced Dicta-Sign-LSF-v2, a continuous dialogue French Sign Language corpus. This corpus is finely annotated, with lexicon, classifier constructions and many more manual unit types. A convolutional-recurrent learning network was built, drawing on a compact and generalizable modeling of people signing. A first baseline was established, showing that video frames can be accurately classified into the main annotation categories of the corpus. The classification network was also tested with state-of-the-art performance on NCSLGR, an ASL corpus. The classifier can be used to segment signs, even though it was not explicitly trained with this objective.     Table 4: Best F1-score on test set (Dicta-Sign-LSF-v2), for the detection of two lexical signs, pointing and depicting signs, fragment buoys and fingerspelling. Two modelings are considered: the one presented in Section 4.1 (3D+HS), and a simple 2D OpenPose modeling of a signer. Furthermore, specific lexical signs or iconic structures can also be detected with the same learning framework.
Hopefully, this work will raise awareness on the importance of Sign Language Processing as a whole, that is beyond lexical-only perspectives, and lead to progress regarding the automatic analysis of iconic structures. It is intended as a baseline, so that different approaches can be compared. Future work will focus on a finer analysis of iconicity in Dicta-Sign-LSF-v2, possibly using annotations of the corpus as weak labels. (a) A very iconic situation is detected around frame 25, even though it appears to be longer than annotated. Around frame 100, some unannotated form of constructed action may be detected. (d) Between frames 30 and 45, the lexical sign "under" is clearly produced in a very iconic way and detected as such. The shape and size classifier construction around frame 55 is accurately detected. Figure 6: Four excerpts from Dicta-Sign-LSF-v2 (video reference: S7_T2_A10), with annotated (full lines) and predicted (dashed lines) probability of depicting signs. Many false positives actually appear to come from a real but unannotated degree of iconicity in some signs. Only some key frames are shown.