{'fraction_non_alphanumeric': 0.03714704368143836, 'fraction_numerical': 0.014375668290364739, 'mean_word_length': 5.074573009381766, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Affect Recognition in Autism: A single case study on integrating a humanoid robot in a standard therapy

Autism Spectrum Disorder (ASD) is a multifaceted developmental disorder that comprises a mixture of social impairments, with deficits in many areas including the theory of mind, imitation, and communication. Moreover, people with autism have difficulty in recognising and understanding emotional expressions. We are currently working on integrating a humanoid robot within the standard clinical treatment offered to children with ASD to support the therapists. In this article, using the A-B-A’ single case design, we propose a robot-assisted affect recognition training and to present the results on the child’s progress during the five months of clinical experimentation. In the investigation, we tested the generalization of learning and the long-term maintenance of new skills via the NEPSY-II affection recognition sub-test. The results of this single case study pilot suggest the feasibility and effectiveness of using a humanoid robot to assist with emotion recognition training in children with ASD.


Introduction
Affect recognition is the process of identifying human emotion, mostly typically via facial expressions, but also verbal expressions. The aptitude to identify and interpret facial emotion recognition (FER), or facial emotion 'affect' does not necessarily develop to the same degree in all children (Lewis & Sullivan, 2014). This appears to be the case for children with neurodevelopmental disorders, specifically Autism Spectrum Disorders (ASD) (Davidson, Hilvert, Misiunaite, Kerby, & Giordano, 2019). According to the fifth edition of the Diagnostic and Statistical Manual of Mental Disorders (DSM-5, APA, 2013), children with Autism Spectrum Disorder (ASD) have persistent deficits in social communication and interaction across several contexts and present restricted, repetitive patterns of behaviour, e.g., stereotyped motor movements or adherence to routines. They find it difficult to make eye contact, to recognize body language, to talk about personal feelings, and to understand other people's emotions (Lord & Bishop, 2015).
The comprehension of emotional expressions (happiness, sadness, anger, fear, surprise and disgust) has mostly been investigated using facial expressions, as the ability to discern emotion from facial expressions is essential for successful social interaction. A study by Widen and Russell (2013) showed that in neurotypical (NT) childhood development, there is a general progression in the development of basic emotion recognition. Specifically, "happy" expressions are typically recognized first, then expressions of "anger" and "sadness", followed by recognition of the more complex emotions of "fear", "surprise", and "disgust".
It is therefore unsurprising that many children with ASD frequently exhibit delays and deviations in their ability to recognize emotions in themselves and others (Harms, Martin, & Wallace, 2010). Moreover, individuals with ASD struggle to recognize complex emotions, have difficulty expressing and regulating their own emotions and show evidence of atypical eye movements when processing emotional faces (APA, 2013;Baron-Cohen, 1997). Indeed, Kanner (1943) originally described autism as a "disorder of affective contact".
The meta-analyses conducted recently by Uljarevic and Hamilton (2013) found that children and adults with ASD had difficulties across all basic emotions except happiness. While in another meta-analysis, the Authors (Lozier, Vanmeter, & Marsh, 2014) showed that "individuals with ASD were less accurate than were controls for all six basic emotions, showing significantly worse performance for anger, fear, and surprise after adjusting for multiple comparisons" (p. 940). Various studies have shown that there may be recognition impairment for expressions of fear and sadness in ASD (Tell, Davidson, & Camras, 2014;Wallace et al., 2011). Furthermore, children with ASD more frequently associated a neutral facial expression with a negative emotion label compared to NT children (Tell et al., 2014).
Researchers (Russo-Ponsaran, Evans-Smith, Johnson, Russo, & McKown, 2016) reported an interesting study of 25 children with ASD conducted over a period of 4-6 weeks using training for emotion recognition. The results showed that the facial emotion training program enabled children and adolescents with ASD to identify feelings in facial expressions more accurately and quickly with stimuli from both the training tool and generalization measures (Russo-Ponsaran et al., 2016). They also demonstrated improved self-expression of facial emotion (Russo-Ponsaran et al., 2016). A tool know as the Social Stories™ is used to develop relationship skills and social understanding. This tool consists of short stories written to help individuals understand the social world and learn how to behave in interpersonal relationships (Scattone, Tingstrom, & Wilczynski, 2006). Researchers have shown Social Stories™ to be successful when applied to a wide variety of problem behaviours including aggression, screaming, grabbing toys, using inappropriate table manners, and crying (Kuoch & Mirenda, 2003;Rowe, 1999;Scattone, Wilczynski, Edwards, & Rabian, 2002). The first group to empirically validate this intervention with ASD was Swaggart and collaborators (Swaggart et al., 1995). The researchers observed a reduction in aggression as well as an increase in appropriate greetings and sharing behaviours for these participants. In another study conducted by Bernad-Ripoll, (2007) the effectiveness of showing videotaped emotions and Social Stories™ to teach a child with ASD to help recognize and understand emotions in himself and to generalize them in other situations was evaluated. The data collected showed an improvement between baseline and intervention in the child's ability to recognize emotions and their occurrence (Bernad-Ripoll, 2007).
Video-modelling is a type of intervention that is effective in limiting problem behaviours and in promoting sensory-emotional regulation (National Autism Center, 2015). The National Standards Project (National Autism Center, 2015) has enabled us to gather evidence of the effectiveness of video-modelling-based interventions that exploit imitation learning mechanisms.
Scientific evidence has shown that children with ASD seem to have a special interest in structured computerized activities, e.g. in clearly defined tasks and that they benefit from the specific attention focus that occurs due to reduced distractions from unnecessary sensory stimuli (Grynszpan, Weiss, Perez-Diaz, & Gal, 2014).
Literature suggests that children with ASD show a preference for robotic devices over non-robotic objects and indeed humans (Simut, Vanderfaeillie, Peca, Van de Perre, & Vanderborght, 2016). Specifically, because robots can display different characteristics of people's social behaviour, this may contribute to creating "a simplified, safe, predictable and reliable environment where the complexity of interaction can be controlled and gradually increased" (Robins, Dautenhahn, Boekhorst, & Billard, 2005, p. 108). Evidence also shows that people who experience physical interactions with robots consider them as more engaging and motivating than interactions with other screen-based information technologies (Matari|, 2017). This is probably because robots evoke emotional reactions potentially leading to specific emotional bonds between human and machine (Eyssel, 2017). Therefore, when the robot held the child's attention and interest in both itself and the tasks, it was possible to extend the therapy sessions for a longer time period (Rudovic, Lee, Mascarell-Maricic, Schuller, & Picard, 2017).
In the light of previous psychological and robotics research, we hypothesize that a robot could support affect recognition training for children with ASD as a long-term intervention. Specifically, the aim was to integrate the robot within the standard treatment, i.e. the TEACCH (Treatment and Education of Autistic and related Communication Handicapped Children) approach (Mesibov, Shea, & Schopler, 2004), and to evaluate the follow-up phase. To this end, we tested the following hypotheses (H1-H3): H1. The child with ASD has difficulties in recognizing basic expressions of emotions (i.e., anger, fear, happiness, sadness) or lack of emotion (i.e., neutral); H2. The child with ASD continues with the affect recognition training in the follow-up phase; H3. The child's abilities to recognize the emotions can be conveyed to the therapist.

Participant characteristics and diagnosis
A male child of 10 years and 4 months was selected from hospitalised patients diagnosed with ASD who were receiving treatment at the Oasi Research Institute-IRCCS of Troina (Italy), a specialized institution for the support and the rehabilitation of neurodevelopmental disorders. The child met the criteria for ASD according to DSM-5. His total score on Autism Diagnostic Observation Schedule (ADOS) was 14, where, based on the categories developed by Gotham, Pickles, and Lord (2009), the cut off >12 indicates autism disorder. He scored 6 on the communication subscale, 8 on the social interaction subscale, 3 on the play subscale, and 2 on the stereotypical behaviours subscale.
To evaluate the severity of autism spectrum symptoms in natural social settings we administered the Social Responsiveness Scale (SRS) where the total T-score was >90 and indicated severe and strong symptoms associated with the clinical diagnosis of Autistic Disorder. The nosological diagnosis indicated: Autism Spectrum Disorder (ICD code 10: F84.0). His language was fluent, and no comorbidity with ID had been diagnosed. The clinical functional diagnosis indicated a clinical condition characterized by difficulties in social communication and mutual interaction. The child showed a limited ability to communicate his emotions, and difficulty in interacting with adults or peers, mainly evident in turn-taking communication. However, sometimes the child only managed to communicate if he was feeling happy or angry, the only emotions he recognized. He didn't show interest in friendships, e.g. he avoided relating to others (adults and peers), and he immediately felt frustrated in case of misunderstanding. The child had considerable difficulty in capturing social signals and interpreting them. He did not always express physically what he said verbally and sometimes it was not very congruous. He had difficulties in perceiving what others thought or felt, even in reference to his own behaviour. However, the child showed a good level of autonomy. During the assessment and intervention task he asked repetitive questions that were not linked to the task and continuously rearranged the cards on the table.
Children at the therapeutic center followed a clinical daily program of training using the TEACCH approach; the core of TEACCH is that structured teaching can effectively benefit children with autism (Mesibov et al., 2004).
In accordance with Oasi Research Institute and Sheffield Hallam University ethics procedures for research with children, the parents provided written consent prior to their child's participation and the child a provided verbal assent before taking part in the study.

NEPSY-II -Affect Recognition subtest
The assessment instrument used in this work is the NEPSY-II scale (Korkman, Kirk, & Kemp, 2007), which was designed to measure neuropsychological functions (NEPSY is short for "neuropsycholo-gy") in children from preschool to school-age. This instrument was chosen because it is child-friendly and it allows ease of administration, portability, and dynamic clinical utility (Sattler & D'Amato, 2002).
In particular, we used the NEPSY-II Affect Recognition subtest (devised for ages 3-16) to evaluate the child's skills, before and after the robot-led training. This subtest, pertaining to the Social Perception sub-domain, includes facial emotion recognition and theory of mind, i.e. the capability to understand others' perspectives, intentions, and beliefs. This subtest included 35 items and assessed the ability to recognize several emotional states (happy, sad, angry, fearful, disgusted, and neutral) from photographs of children's faces in four different tasks: first, the child stated whether or not two pictures depicted faces with the same affect; then, he selected two pictures of faces with the same affect from 3-4 pictures; thirdly, the child selected one of the four faces that depicted the same emotion as a face at the top of the page (e.g. Figure 1); lastly, the child was briefly shown a face and he selected two pictures that he thought represented the same emotion.

Technological Platform
For this study the robot used was the Softbank Robotics NAO (Model H25, version 4), which is a small, toy-like humanoid robot, very popular for child-robot interaction research (Coninx et al., 2016;Conti, Cirasa, Di Nuovo, & Di Nuovo, 2019;Conti, Di Nuovo, & Di Nuovo, 2019). The decision to use a humanoid robot was taken on the basis of recent evidence suggesting that robots resembling a human appearance and with moving limbs are more effective in prompting social responses in children with ASD (Conti, Trubia, Buono, Di Nuovo, & Di Nuovo, 2018;Desideri et al., 2018;Lee, Takehashi, Nagai, Obinata, & Stefanov, 2012;Robins, Dautenhahn, & Dubowski, 2006). The NAO robot is 57.4 cm tall, weights 4.3 kg and can produce very expressive gestures with 25 Degrees of Freedom (DoF) (4 joints for each arm; 2 for each hand; 5 for each leg; 2 for the head and one to control the hips). This robot can detect faces and respond to eye contact moving its head accordingly. It can also vary the colour of LEDs in its eyes' contours to simulate emotions, and it can capture a lot of information about the environment using sensors and microphones. The NAO robot is programmed through a user-friendly graphical programming tool (Figure 2), named Choregraphe (Pot, Monceaux, Gelin, & Maisonnier, 2009), which provides an intuitive way to design complex behaviours, including several interfaces for non-verbal communication, i.e. including gestures, sounds, and LEDs. In this research, the humanoid robot was programmed to implement the emotions of the Affect recognition subscale: happiness, sadness, anger, fear, and disgust. There was also a 'neutral' condition where Figure 2. Example of the Choregraphe platform the robot, did not make movements or change colours and was said to be "neutral".
To improve the children's understanding of the NAO robot, we set the volume at 90/100 and slowed down the language speed to 85/100, agreeing these levels with the clinician. The voice we used was the standard Italian provided by the robot manufacturer.

Environmental Setup
The experiments were carried out in the dedicated therapy room where the child had his TEACCH treatment each day. During all sessions, the robot was deployed on a table, at a distance of 80 centimetres (approximately 31.5 inches) (Figure 3).
Adults in the room included a therapist and an operator. The operator was hidden behind a panel to manage the NAO robot using the Wizard-of-Oz (WoZ) methods (Bainbridge, Hart, Kim, & Scassellati, 2011). On the panel was the visual card provided by the TEACCH method, where the child inserts the image of the activity to be performed. Schedules with pictures of various activities tell the child where he should be and when he should be there and have been proven to aid with transitions and increase predictability (Hagiwara & Smith Myles, 1999). Therefore, our robot-assisted therapy was incorporated into the TEACCH program among the standard activities, which are identified via a specific visual schedule (Mesibov et al., 2004).
Behind the child, an experienced therapist was always present to represent a "secure base" for the children (Bowlby, 2005). During the first six weeks of the study, the role of the therapist was only to support the possible needs of the child. From the seventh week, the therapist took a more active role.
All the sessions were recorded by the video-camera integrated within the NAO robot. A hidden video camera was also placed in the room, behind the robot.

Study design
The study followed an A-B-A' single-case design. Specifically, in an A-B-A' design, data that is gathered in the second baseline (A') indicates whether the effects of the intervention (B) continue when the intervention is no longer in place, or if the removal of the intervention (the NAO robot in our study) results in a return to baseline (A) behaviours.
The benefit of a single-case experimental design is to allow a more intensive investigation of a single participant. Single-case research is the preliminary way to establish generality because researchers can use the single case to identify first the relevant controlling variables for the phenomenon under the study (Sidman, 1960). Generality and external validity are then established inductively, moving from the single case to ever-larger collections of single case experiments with high internal validity (Guala, 2003;Hogarth, 2005). Thorngate (1986) explains the principle in this way: "To find out what people do in general, we must first discover what each person does in particular, then determine what, if anything, these particulars have in common" (p. 75).

Procedure
This section presents the details of the procedure, which is summarised in Table 1.

Phase A -First Baseline
First, we evaluated the child's affect recognition skills, using the NEP-SY-II, before starting the experimental procedure with the robot. This evaluation without the robot constitutes the first baseline (Ex Ante, phase A).
To minimize the novelty effect (Han, Jo, Jones, & Jo, 2008), the robot was preliminarily presented to the child for 10 minutes in a nontherapeutic context, when the child was encouraged to do as much interaction as he wanted without any specific training purpose.

Phase B -Intervention
The intervention training, devised from the TEACCH program, included 12 encounters over four weeks, with one encounter in a day. Encounters lasted about 10 minutes and consisted of a game in which NAO asked the child to say what emotions the robot was displaying.
To further support the game from the visual point of view, we provided the child with simple images of the five emotions included in the training game and one of the neutral condition (Figure 4). The images were 13x18 cm in size, mounted on cardboard measuring 15x21 cm. All the pictures were arranged in front of the child, mostly in a 3x2 arrangement.
Each encounter consisted of 3 sub-sessions, and each sub-session suggested 3 random emotions. The order in which the emotions were proposed was randomized to avoid order effects. A sequence example of emotions proposed by the robot is shown in Figure 5.
After six daily sessions, we removed the two-dimensional images in favour of generalisation and autonomy, by working only on the three-dimensional state of the robot.
The training game protocol was as follows: the robot asked the child if he wanted to play at mimicking emotions that he had to understand. The NAO suggested that it would help the child with three possible answers where only one was correct and that the child could use the images on the table. Also, the robot specified that if for any reason the child wanted to stop the game he could press the robot's head or ask for help. Successively, the robot encouraged him to pay attention to its movement and speech. Then, the robot mimicked an emotion, e.g. happiness, by changing eye colours, making utterances and gestures, moving limbs and torso. After this, the robot gave three suggestions and encouraged the child to respond. The child used the images to say which emotion the robot mimed. When the answer was correct the NAO rewarded the child with happy music, with iridescent eye colour and by saying words like "good, right, super". If incorrect, it emitted an unpleasant sound with red eyes. In the latter case, the robot suggested the child pay more attention to the next emotion. Finally, the therapist put back and rearranged all the images. This condition was repeated three times during each encounter.

Phase A' -Second Baseline
During the second baseline, the clinician used the NEPSY-II to evaluate progress in the child's affect recognition (Ex Post).

Role-Playing session
After a week, to reinforce our evaluation and to investigate the generalisation, we engaged the child in a role-playing activity with the therapist. During this session, the therapist asked the child to play the role of the robot and mimic the emotions to her (Figure 6). The five emotions were proposed by the therapist in random order.
We decided to simulate formative role-playing because it is a valuable training tool, based on the simulation of something that has or could be related to a real situation. The purpose was to generalize the learned behaviour to other contexts for the child (Rogers & Dawson, 2010).

Follow-up
A follow-up phase is essential to evaluate whether the benefits have been maintained in the absence of intervention or after a long time from its suspension. Therefore, we performed a follow-up evaluation three months after the second baseline. During the follow-up, we asked the child to mimic to the therapist the emotions learned in the robot training. As in the training, the meeting took place in a single encounter with three sub-sessions: the first sub-session with the support of images, and the remaining two sub-sessions were without them.

Measures And Analysis
To assess the affect recognition training with the child, we decided to attribute 1 point when the child's answer was correct and 0 points when it was wrong. Therefore, the child had nine opportunities in each session to receive a score: in total, a maximum of nine points could be collected for each session by the child. Table 2 shows a comparison of the administration of the NEPSY-II affect recognition scores, including the percentile ranks. The NEPSY-II scale scores as "Below Expected Level" when scores are not as developed as 90% of their peers. While "At Expected Level" scores are given when results are equal at least to 50% of their peers. Table 2 includes the score and percentile ranking evaluated in the follow-up session after 3 months. In this case, the emotional skills were maintained with an increase in the percentile ranks. Specifically, Figure 7 shows the child's progress during the intervention phase with the robot.

Results
The child had evident difficulties in recognizing some emotions, compared to others. For example, in phase A, the child showed more difficulty in recognizing anger (50%), disgust (38%), and fear (29%) than the happiness, sadness and neutral. While, in phase A', the errors were considerably reduced below 10%, as shown in Figure 8.

Discussion and conclusions
Literature indicates that emotional expressions in facial photographs are less salient to children with autism than, for instance, hats (Jennings, 1973;Weeks & Hobson, 1987) and wigs (Bormann-Kischkel, Hildebrand-Pascher, & Stegbauer, 1990), and has been confirmed by behaviour observations in natural situations. Pre-schoolers with autism were less attentive towards an adult expressing distress, fear or discomfort than children with an intellectual disability or neurotypical developed children. Children with autism also exhibit profound difficulty in assuming another's role (Baron-Cohen, 1997) an ability crucial for role-play.
The results of this case study showed the positive impact of a humanoid robot in supporting the emotion training of a child with severe difficulties in this area.
Concerning our first hypothesis (H1), results in Figure 8 show that the child was better at recognizing basic expressions of emotions (i.e. happy and sad). The lack of emotion (i.e., neutral) with the display of neutral faces was less accurate, probably because children with ASD perceive ambiguous emotional stimuli as more negative than the NT children (Kuusikko et al., 2009).
Regarding H2, we found that the robot could be embedded into the standard TEACCH protocol, where the learning curve was evaluated with a standardized and highly reliable psycho-diagnostic tool, named NEPSY-II. The child with ASD continues with the affect recognition training in the follow-up phase, after 3 months. With regard to our third hypothesis (H3), the longitudinal evaluation shows constant improvement during the robot-led session and importantly, that the child was able to convey to the therapist what he had learned with the robot.
An important strength of this study is that even if it can be considered a long-term intervention (five months) the child kept his interest throughout these sessions, showing a great adherence to the treatment. However, it should be kept in mind that our results are limited to a single child with ASD. In the future, it could be essential to increase the sample of participants. Furthermore, it could be important to define and/or quantify the spontaneous requests for interaction/ communication by the child towards the humanoid robot.