{'fraction_non_alphanumeric': 0.026432824044186214, 'fraction_numerical': 0.007173086579155011, 'mean_word_length': 5.230837988826815, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
The state of play of ASC-Inclusion: An Integrated Internet-Based Environment for Social Inclusion of Children with Autism Spectrum Conditions

Individuals with Autism Spectrum Conditions (ASC) have marked difficulties using verbal and non-verbal communication for social interaction. The running ASC-Inclusion project aims to help children with ASC by allowing them to learn how emotions can be expressed and recognised via playing games in a virtual world. The platform includes analysis of users' gestures, facial, and vocal expressions using standard microphone and web-cam or a depth sensor, training through games, text communication with peers, animation, video and audio clips. We present the state of play in realising such a serious game platform and provide results for the different modalities.


INTRODUCTION
Three decades of research have shown that children and adults with Autism Spectrum Conditions (ASC) experience significant difficulties recognising and expressing emotions and mental states [3].These difficulties are apparent when individuals with ASC attempt to recognise emotions from facial expressions [14,17,23,27], from vocal intonation [12,24], from gestures and body language [26,44], and from the integration of multi-modal emotional information in context [22,50] Limited emotional expressiveness in non-verbal communication is also characteristic in ASC, and studies have demonstrated individuals with ASC have difficulties directing appropriate facial expressions to others [31,32,30], modulating their vocal intonation appropriately when expressing emotion [34,37,43] and using appropriate gestures and body language [1]. Integration of these non-verbal communicative cues with speech is also hampered [16]. Individuals with ASC lack the sense of social reciprocity and fail to develop and maintain age appropriate peer relationships [8], [40]. Current findings suggest 1 % of the population might fit an ASC diagnosis [4].
The use of Information Communication Technology (ICT) with individuals with ASC has flourished in the last decade for several reasons: the computerised environment is predictable, consistent, and free from social demands, which individuals with ASC may find stressful. Users can work at their own pace and level of understanding, and lessons can be repeated over and over again, until mastery is achieved. In addition, interest and motivation can be maintained through different and individually selected computerised rewards [10,39,42]. For these reasons, and following the affinity individuals with ASC show for the computerised environment, dozens of ICT programs, teaching various skills to this population were created. However, most of these tended to be rather specific (e. g., focusing only on recognition of facial expressions from still photos) and low budget, and have not been scientifically evaluated [25]. ICT programs that teach socio-emotional communication have been evaluated including I can Problem-Solve; others aim to teach social problem solving [9], such as FEFFA, e. g., by emotion recognition from still pictures of facial expressions and strips of the eye region [11]. Emotion Trainer teaches emotion recognition of four emotions from facial expressions [49]; Lets Face It is teaching emotion and identity recognition from facial expressions [52], and Junior Detective program combines ICT with group training in order to teach social skills to children with ASC [7]. These examples demonstrate how focused most ICT solutions are in their training, focusing mostly on emotion recognition from facial expressions and contextual situations, with very little attention given to emotional gestures of emotional voices. In addition, there were no reports of ICT programs teaching emotional expressiveness. Further training programs such as Mind Reading implement an interactive guide to emotions and teaches recognition of 412 emotions and mental states, systematically grouped into 24 emotion groups, and 6 developmental levels (from the age of four years to adulthood). The evaluations of Mind Reading resulted in limited generalization when adults with ASC used the software [21,25]. However, when 8-11 year old children with ASC had used it, improved generalization was found.
In the last decade, with the rapid development of internetbased communication, web applications have been increasingly used for social interaction, forming online communities and social networks. Anecdotal reports of the emergence of online autistic communities, and the use of forums, chat rooms and virtual-worlds, show the great promise the internet holds for better inclusion and social skills training for adolescents and adults with ASC [13,28]. However, there has been no scientifically documented attempt to use the internet for structured training of socio-emotional communication for individuals with ASC. Furthermore, since intervention into ASC has been shown to be more effective when provided early in life, using the internet as a platform for the support of children with ASC could significantly promote their social inclusion.
Virtual Environments (VE) form another domain with immense possibilities for those with ASC and related social difficulties. VE are artificial computer generated threedimensional simulations and come in single-or multi-user forms. In either format, the user can operate in realistic scenarios to practice social skills, conversations, and social problem solving. Moore and colleagues investigated VE and children and youth with ASC and found that over 90 % of their participants used the VE to recognise basic emotions [38]. Other studies have also shown the potential for those with ASC to use VE for socio-emotional skills training and for other learning purposes [41,42].
The ASC-Inclusion project 1 [47] -dealt with herein -suggests advanced ICT-enabled solutions and serious games for the empowerment of children with ASC who are at high risk of social exclusion. It combines several state-of-the-art technologies in one comprehensive game environment, including analysis of users' gestures, facial, and vocal expressions, training through games, text chatting, animation, video and audio clips. Despite the innovative technologies involved, the ASC-Inclusion is aimed for home use. Though designed to assist children with ASC, ASC-Inclusion could serve other population groups characterised by deficient emotional understanding and social skills, such as children with learning difficulties [6], attention deficit and hyperactivity disorder (ADHD) [15], behavioural and conduct problems [51], or socio-emotional difficulties [45].
The remainder of this paper is structured as follows: first, a detailed description of the user requirements and specification is given (Section User Requirements); then we describe the three modalities namely face, voice and body gesture (Sections Face Analysis, Voice Analysis, Gesture Analysis) and the platform (Section Platform). We next comment on content creation (Section Content Creation) before concluding the paper in Section Conclusions.

USER REQUIREMENTS
Previous literature reports social communication difficulties in individuals with ASC as well as enhanced abilities in other non-social areas such as systemising [5]. This provided us 1 http://www.asc-inclusion.eu with a greater insight into the social, cognitive and behavioural abilities of these potential end users. Review of the design, content and effectiveness of the Mind Reading DVD as an intervention [21] and The Transporters DVD [20] which have shown to enhance the ability of children with ASC to recognise emotions, permitted identification of a method through which learning material can be presented in an engaging manner which capitalises on the enhanced systemising skills seen in individuals with ASC.
A preliminary set of focus groups in Israel and Sweden were observed to help identify the preferences and interaction style of children with ASC when using virtual environments. The results from Hebrew user focus groups found that the children enjoyed using the virtual environments and the majority of children could navigate independently in the environment, activate different functions of the VE and change the settings. The results from these focus groups found that the games that were easy to understand and had a good game concept were rated highest by the children. It was noted that a number of the games presented to the children and which the children enjoyed playing could be easily adapted to become emotion recognition games, but that better reward systems and the ability to change difficulty level was necessary to keep the children engaged and motivated.
Further focus groups were observed to asses if the children could maintain attention to a teacher talking about emotions and understand how emotional dimensions are expressed in the voice. The results from this study showed that the maturity level appears to affect childrens ability to maintain attention and comprehend the topic of emotions in voices. Additional instructions are required for younger children when teaching emotions in the voice. Illustrative examples may help the children to grasp the concept of the different vocal dimensions such as pitch and vocal speed.
A second batch of focus groups were conducted in Israel with the main aims of observing the children's use and reactions to the virtual environment, the four educational games, and the 'anger' lessons provided in the current ASC-Inclusion platform by assessing the level of attention and learning. Additionally, the first version of the voice analyser as further described below was tested. Generally, the Focus Group children loved the virtual environment. They responded in a positive way to the environment and navigated freely and instinctively through it. The children in particular liked the functions of the environment that they could design themselves such as the avatar character and their virtual home, these features of the environment are significantly motivational. The children also enjoyed being able to equip their avatar character. All the children listened throughout the anger lessons, and the material was understood. Most of the children also liked the emotional games with particular preference to "Emotion Tetris" and "Emotion Memory Game". All the children liked the activity with the voice analyser software. They related to the plots and understood how to modulate vocal parameters after a short explanation from the experimenter. The children had fun playing with their voices and influencing the plots shown to them based on their speech acoustics. Further user testing was conducted with additional games embedded into the platform such as "Who wants to be a Millionaire" and "Everyday Stories". The former assesses the child on the level of learning in the unit just completed; performance on this quiz determines the child's progression to the next learning unit. The latter allows the child to practice and demonstrate the knowledge he/she acquired in a learning unit. The child deals with integrative presentation of the social situations, including face, body, and voice materials. The results show that in the former game, the percentage of questions correctly answered increased with age. The information gathered from review of literature and technologies developed for children with ASC, as well as the results from the focus groups have been fundamental in designing both the visual and functional aspects of the VE to match the needs of the intended user -children with ASC 5-10 years of age.

FACE ANALYSIS
The previous Facial Affect Computation Engine (FACE) [47] is using a dimensional model of affect to fit with the other inference systems from voice and from body motion. The dimensional model requires a different approach to affective inference, building on our earlier experience with categorical inference [29]. We want to model the affect in a continuous space rather than as discrete categories as in our initial prototype. Furthermore, we want to model the temporal relationships between each time step, since emotion has temporal properties and is not instantaneous. The constrained local model (CLM) tracker [2] has been retained, but the mental state inference now applies linear support vector regression (SVR) for inference. The dimensional inference system uses the CLM tracker to extract 34 features from live video of the face. These include per-frame descriptors of the facial expression and measures of variation in head pose over time. SVR was trained on the AVEC 2012 corpus [48] to predict values for valence and arousal. The output from the FACE engine is encoded in EmotionML [46] and delivered through an ActiveMQ 2 communication infrastructure, allowing integration with vocal analysis and gesture analysis to provide multi-modal inference in the central platform.

VOICE ANALYSIS
For the training of a voice analyser, a data set of prototypical emotional utterances containing sentences spoken in English, Hebrew, and Swedish by a total number of 60 children has been collected with typically developing children and children with ASC. A complete description of the database containing sentences in Hebrew is given in [36]. Considering the outcome of the evaluations described in [36] and [35], we defined potential features and descriptors that are relevant for emotion perception, such as energy by the sum of auditory spectrum at different frequency bands (from 20 Hz to 8 kHz) and root-mean-square signal frame energy, pitch by fundamental frequency contour, and duration by modelling temporal aspects of F0 values, such as the F0 onset segment length. Applying the openSMILE audio feature extractor [19,18], features are extracted and tracked over time: in order to assess a child's performance in expressing emotions via speech, the extracted parameters are compared to the respective parameters extracted from pre-recorded prototypical utterances. The vocal expression evaluation system is shown in Figure 2. In the "Target emotion" box the player chooses the emotion she or he wants to play with. Once the emotion has been selected, a reference emotion expression is played back to the child and the according quadrant of the arousal / valence plane is highlighted. Then, the child is prompted to repeat the selected emotion. According to the expressed emotion the evaluation system is providing a visual feedback on the aurousal / valence plane via a red dot indicating the coordinates of the expressed emotion in the two dimensional space. Additionally, in the "Recognised Emotion Group" box the detected emotion is expressed. According to the correctness of the expressed emotion virtual coins are earned and on the bottom right part of the GUI a corrective feedback is shown. The 'traffic light' on the top of the gauge bars indicates if the extracted parameters are distant or close to the reference values. If a green light is shown the childs expression is close to the reference; the red light indicates high distance between the reference value and the extracted one.
The output from the voice analyser is also encoded in Emo-tionML [46] and delivered through the ActiveMQ communication infrastructure, allowing integration with the face analysis and the gesture analysis to provide multi-modal inference in the central platform.

GESTURE ANALYSIS
Full-body expressive features to be analysed in the game are inspired by several sources: biomechanics, computer vision, experimental psychology, and humanistic theories, including Laban's Effort Theory [33]. The list of candidate features is the following: kinetic energy of body parts (hands, head, upper body), symmetry of hands movements and posture with respect to the vertical axis and forward/backward, forward/backward leaning of upper part of the body, directness (direct/flexible in terms of Laban's Effort), impulsivity, fluidity, light/heavy gesture quality, the relative position between head and shoulder, openness of the posture (contraction/expansion), rigidity and swinging of the whole body or of body parts.
A dataset of 15 people expressing the six basic emotions (happiness, anger, sadness, disgust, fear, and surprise) was collected. Each person was asked to express one of the six emotions with their body from three to seven times in front of a Microsoft Kinect sensor. On average, each recording session lasted about 10 minutes. After the recording sessions, each video was manually segmented to find the prototypical expressive gestures of each emotion. Those segments were used to evaluate the data with human subjects. 60 judges labelled 10 segments, each. The material was used to train SVM for the classification modules. A real-time feature extractor has been developed using the EyesWeb 3 XMI development environment; the existing library for gesture analysis has been extended and modified according to the descriptors and feature set introduced before. The EyesWeb development environment main characteristics are: the multi-modal input processing capabilities, modularity, flexibility and extensibility. The key features are as follows: Multimodal input support, where data from an arbitrary input stream (file, webcams, RGB-D cameras, microphones, sensors, etc.) is pushed through the processing chain sample by sample and frame by frame, modularity, which allows modifying the extracted features set, and easy addition of new feature extractors via a dedicated API, and finally easy configuration of feature extractor parameters that can even be tuned during run-time through the EyesWeb XMI development environment.
Higher level computations such as machine learning techniques are carried by the MetaEyesWeb external module. This adds to the Python programming language the possibility to remotely communicate with the EyesWeb Environment, controls the developed applications execution and tunes directly the module parameters through a dedicated communication API. MetaEyesWeb Mobile enables the use of Python scripts and thus the integration of related machine learning libraries. Output from the body analyser is again encoded in EmotionML [46] and delivered through the ActiveMQ communication infrastructure, allowing integration with face analysis and voice analysis to provide multi-modal inference in the central platform. The framework described so far was used as the main part of two game demonstrations. Both the games perform a 3 http://www.infomus.org/eyesweb_ita.php live automatic emotion recognition, and interact with the user by asking her to guess an emotion and to express an emotion with her body (Figure 3).

PLATFORM
The general architecture presented in [47] has been updated and extended. An overview of the new integrated platform architecture is given in Figure 4. The container of the game engine in flash is ZINC 4 ; the container loads the game environment and the subsystems services. The flash game engine is lying under the game logic and the user interface. It is responsible for the communication with the different services through Apache ActiveMQ by applying Streaming Text Oriented Messaging Protocol 5 (STOMP). The communication with the different services are based on three components. The subsystem control component sends control messages to the different services -the control messages are sent to a unique ActiveMQ queue for each of the services (face, body, voice). When a message is sent to the activeMQ queue, the subsystem service reads it and processes the command. The STOMP message receiver gets XML text messages from ActiveMQ topic based on STOMP protocol. The component connects to ActiveMQ through STOMP and starts receiving messages from the server. Messages are then used by the game engine in order to provide a certain feedback to the user. As above, the XML messages are encoded using the EmotionML standard. Since the game needs to show the web camera input, or to play back the voice recorded during a game task, the subsystem 'media component' obtains such data from the subsystem services. Each service runs a web server which provides (by request) a stream of information from the camera or the recorded audio. The ActiveMQ process is used as the pipe to push the information from the different subsystems to the game engine. ActiveMQ includes one Topic (named asc) which contains all classification results and parameters from the different modalities (based on the running services). Ac-tiveMQ also includes three queues (one per subsystem) which contain control messages produced by the platform (start, stop, shutdown, etc.). The three subsystems provide output in the same dimensional space. The subsystems provide the actual values of the recognised emotion in the arousal and valence space and the platform compares them according to the target space.
Besides the platform integration, a new Learning Manager Application (LMA) was developed (cf. Figure 5). This application is the main tool to control, personalise, and present the learning material to the user. It also enables the navigation within the program including access to the practice games. The LMA enables to track the user's play patterns for data collection and for system improvement purposes. The LMA also manages the rewards granted to the child while going through the program. The 'monetary system' is the basis for the VE 'economy', planned to motivate the child and encourage longterm engagement. The learning material overall consists of an intro (five animated general introduction sessions, interactive activities), emotion lessons (20 interactive animated 'research sessions'), curricular basic games (six simple task games, two for each modality (face, voice, body)), integrative stories (six interactive stories with accompanying activities, two for each of the emotions learning units), advanced games (three crossmodal games, one integrative quiz game), a comprehensive testing game (one per emotion unit, 300 questions arranged in nine levels of difficulty, systematically covering all the unit content), and three semi-curricular dynamic fun games with emotions content.
Advancing in the program by actively participating in 'research sessions' (lessons) and playing the practise games to a pre-defined achievement, is the only way to earn virtual money in the VE, and virtual money is needed for virtually anything the child might want to do in the VE apart from 'doing research' (learning) -namely: play non-curricular games, buy goodies, go to fun locations out of the 'camp' -the learning and training activities are embedded in a virtual world themed as a 'research camp', where each user is a 'young researcher' exploring human behaviour -etc. This way we hope to maintain an efficient balance between coping with the possibly challenging emotions content, and having the opportunity for recreation and pure fun. This is also meant to prevent a situation of ignoring the program or 'getting stuck' on a certain activity or comfort zone. Additionally, further motivational elements were added, such as the avatar, the camp-home, and the virtual wallet. Four new characters have been designed to serve and support the learning sessions and add interest and fun. Further, four fun locations were designed as out-of-camp 'tourist' recreation destinations. The new locations contain fun games with themed content that also has an enrichment element. The child needs to earn the ticket to go there, and usage is limited.
A first expression game 6 has been developed. It integrates the face, voice, and body language analysis technologies. This game is the main activity in the 'expression sub-unit' at the end of each of the learning units. The game is designed as a 'race' board game. Each turn the child is asked to express an emotion in a chosen modality. If she/he expressed it well enough, the racing robot advances one step on the board. Whoever gets to the ending point first wins.

CONTENT CREATION
In order to determine the most important emotions for social interaction in children with ASC, an emotion survey was developed and completed by parents, typically developing adults and parents in particular of children with ASC. 20 emotions -happy, sad, afraid, angry, disgusted, surprised, excited, interested, bored, worried, disappointed, frustrated, hurt, kind, jealous, unfriendly, joking, sneaky, ashamed, and proud -were identified by the survey as being the most important for social interaction and selected to be taught through the platform. Additionally, 496 face stimuli, 82 body gesture stimuli and 95 social scene stimuli previously recorded with professional actors were validated. The stimuli were divided into 21 validation surveys with each survey containing 30-45 emotion stimuli. Each survey was completed by 60 typically developing adults  (20 per country), at minimum age of 18 years. For each stimulus validated a percentage of correct recognition score and a chance corrected percentage recognition score was calculated. These two scores detail the percentage of participants that correctly identified the emotion. However, the chance corrected score takes into consideration the possibility of the person selecting the correct emotion label by chance. Only emotions with a chance corrected percentage score greater than 50 % are included in the platform. A further 20 UK-based voice surveys have been created and validated along side voice recordings in Hebrew and Swedish: As outlined above, to assist with the development of the voice analysis module, a series of emotion phrases were recorded by in UK, Israel, and Sweden. A total of 20 children (10 typically developing and 10 children with ASC) were recorded at each site, in English, Hebrew and Swedish. The discrete emotions were portrayed within phrases that the child had to repeat; each phrase was embedded within a story to provide context for the child to express the emotion. Four phrases were written for each of the emotions.

CONCLUSIONS
We introduced the current state of the gaming platform ASC-Inclusion targeted to children aged 5 to 10 years with ASC. User requirements and specification have been refined and further analysed, tailoring the needs of the target audience and creating user scenarios for the platform subsystems. The integration of the three subsystem into the game platform was implemented. The technical partners collaborated to design communication protocols and the platform architecture for the main game engine. The face analyser implements the required messaging components and provides output in the arousal and valence dimensional space. The voice analyser has new components to enable software integration and is being refined through a spiral developmental process. A new release of the analyser is currently evaluated during recent focus group meetings and a new vocal expression evaluation system has been developed. The body gesture analyser can provide ActiveMQ messages and infrastructure to integrate the subsystem in the platform. the gesture analyser also refined the feature extraction modules and the emotion recognition system. Further, interactive training program lessons and expression games on emotions by using the integrated modalities were developed. The content to be used by the general platform and the subsystems has been identified and created. The video material recorded as examples has been extensively validated such that out of 60 raters, at least half agree. Two surveys were initially developed and piloted to ensure the validation method chosen was methodologically sound. The evaluation protocols for the upcoming randomised control clinical trial are prepared. As an outlook, evaluation is needed, providing the basis for the further refinement of the analysers, and fine-tuned interplay with the platform.