{'fraction_non_alphanumeric': 0.026833724876011485, 'fraction_numerical': 0.018010963194988253, 'mean_word_length': 5.215444516547696, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Clinical perception of effectiveness of virtual appointments and comparison with appointment outcomes at a specialist children's hospital

Introduction A transition from face-to-face to virtual consultations occurred in response to the COVID-19 pandemic. Evaluation of outcome data is essential for future healthcare modelling. Methods Clinicians at a children's hospital evaluated perceptions of face-to-face video and telephone appointments by questionnaire. Responses were compared with operational outcomes from June 2019 and June 2020. Results Ninety-three clinicians responded from 28 subspecialties. Virtual consultations increased from 6% (2019) to 67% (2020). No differences were found between appointment types for recording a medical and social history; a significant difference (p<0.001) was seen for the perceived ability to detect clinical signs, organise investigations and make a diagnosis. The proportion of appointments resulting in discharge compared with face-to-face visits was unchanged. The proportion of patients requiring further contact increased from 35% (32% face-to-face and 3% telephone) to 46% (14% face-to-face; 21% telephone and 11% video; chi-squared 426; p<0.0001). The percentage of patients offered an appointment following two ‘was not brought’ appointments increased from 71% (2019) to 81% (2020) and was most common following telephone appointments (20% face-to-face, 43% telephone and 18% video; chi-squared 474; p<0.0001). Conclusion The perception of clinicians is that virtual appointments enabled continuity of paediatric care with improved clinical assessment capability and attendance during video consultations compared with telephone consultations.


Introduction
Many healthcare providers transitioned from face-to-face to remote virtual consultations in response to the COVID-19 pandemic. 1 The speed of change and need to adapt rapidly, has introduced new challenges and opportunities for clinicians, children and young people (CYP), and carers. Previous research has demonstrated that virtual consultations by video or telephone are likely to be cost effective and have high patient satisfaction when used for monitoring and follow-up of discrete patient groups. This is particularly true of chronic conditions, such as diabetes and mental health follow-up. 2,3 Paediatric outpatient care is more complex and necessitates evaluation of children in many different stages of physical and cognitive development. Appointments usually require the presence of a parent(s) or guardian, and consultations must be adapted to take into account the needs of all those present. In response to COVID-19 restrictions imposed by the government and in order to reduce risk of exposure and spread of the virus for both staff and CYP, our UK-based children's tertiary hospital implemented scheduled video and telephone appointments across the trust in a short time by integrating a video communication platform with our electronic patient record system, supported by our information technology (IT) and information governance teams. To establish whether these virtual appointments were an effective alternative to face-toface appointments in a tertiary paediatric hospital setting, we initially evaluated subjective clinical impressions across a range of specialties regarding the ability of clinicians to provide appropriate diagnostic and follow-up care and compared this with operational outcomes of the outpatient encounters.

Methods
In April 2020, Great Ormond Street Hospital (GOSH) introduced functionality for video visit outpatient appointments that were fully integrated into the electronic patient record. As a result of the COVID-19 pandemic, clinicians were asked to limit face-to-face appointments to essential appointments only and encouraged to convert appointments to video or telephone encounters. In June 2020, during a UK lockdown, a link to a survey questionnaire (supplementary material S1) was sent to clinicians that conducted outpatient appointments (doctors, specialist nurses and allied health professionals) through trust-wide communication asking about their perceptions of face-to-face, video and telephone appointments, and whether they believed that they could assess CYP effectively. The questionnaire was designed by clinicians with quality improvement training. Ten key questions were asked and responses were quantitative using a Likert scale of 'always', 'often', 'sometimes', 'rarely' and 'never'.
For the same month, we compared the operational outcomes of the outpatient encounters (admission request, discharge, active monitoring and 'was not brought' (WNB)) for each of the appointment types. These data were extracted from the electronic patient records. Completion of this field is a mandatory hard stop for all outpatient encounters and provides objective outcome data. The data from June 2019 were compared with data from June 2020 using descriptive statistics and comparison of proportions test in order to assess any change in overall appointment numbers and type.
This project was assessed as being in line with the Health Research Authority guidance as a clinical audit/service evaluation in line with the criteria outlined for determining whether work is audit, service evaluation or research. The project was supported by our transformation team at GOSH and there was due governance to ensure that no burden was placed in staff or patients, and all data was managed according to our information governance procedures.

Clinician responses
Ninety-three clinicians completed the survey from 28/61 (46%) of GOSH subspecialty departments. This included 14 medical, nine surgical and five therapy departments (supplementary material S2). The responders were a mix of 45 (48%) doctors and 48 (52%) nurse practitioner / allied health professionals. There was no difference in responses between role types (Table 1).
When comparing appointment type (face-to-face, video and telephone), clinicians reported little difference for ability to take a medical and social history, but differences between appointment types were evident for ability to pick up non-verbal cues and to ascertain clinical signs (Fig 1). Clinicians reported being most confident about making a diagnosis when reviewing patients faceto-face and least confident with a telephone conversation.
There was no reported difference between groups for time efficiency, although clinicians indicated that telephone appointments were the most flexible with regards to appointment time and face-to-face appointments were the least flexible. Overall, more clinicians reported that they were always or often able to organise investigations, start treatment and train parents/ patients when meeting face-to-face compared with video or telephone appointments (Fig 1).

Objective outcomes following outpatient appointments
In response to COVID-19, there was a significant increase in the percentage of outpatient activity delivered virtually: in June 2020, 10,433 out of 15,542 (67%) outpatient appointments were delivered via video and telephone compared with only 1,073 out of 18,898 (6%) in June 2019 (chi-squared 14,238; p<0.0001). Clinicians recorded their follow-up decision after each outpatient appointment and completion of this field is a mandatory hard stop within the electronic record. When examining the split of outcome options in June 2019 and June 2020, the spread was similar (Fig 2). In particular, the proportion of appointments resulting in discharge with or without treatment was not significantly different (chi-squared 0; p=1.0), despite the significant increase in virtual appointments. In June 2019, 7% of appointments resulted in discharge and all were face-to-face. In June 2020, 7% of appointments resulted in discharge but with a split of appointment type (1% for face-to-face, 2% for video and 4% for telephone).
Between June 2019 and June 2020, there was a small but statistically significant reduction in the proportion of appointments in which the decision was made that CYP do not require any form of treatment currently but should be monitored in secondary care (active monitoring). This was 10% in June 2019 (9% for face-to-face and 1% for telephone) and 6% in June 2020 (1% for face-to-face, 3% for telephone and 2% for video; chi-squared 190; p<0.0001). However, the proportion of CYP who were recorded as requiring further contact significantly increased from 35% (32% face-to-face; 3% telephone) to 46% (14% face-to-face; 21% telephone and 11% video; chi-squared 426; p<0.0001). The latter indicating that the highest proportion of CYP requiring further contact were following a telephone visit.
The age of CYP for whom appointments were scheduled were no different between groups (p=0.7) with a median age of 8 years for face-to-face appointments in 2019 and 2020; a median age of 6 years for telephone appointments in 2019 compared with 8 years in 2020; and a median age of 10 years for video appointments in 2020. The range in each group was from newborn to adulthood.
Did not attend outcomes CYP who did not attend (DNA) outpatient appointments follow a WNB pathway to ensure safeguarding measures are applied before discharge. The proportion of CYP offered a subsequent appointment following two DNA attendances significantly increased from 71% in June 2019 (70% for face-to-face and 1% for telephone) to 81% (20% face-to-face, 43% for telephone and 18% for video) in June 2020 (chi-squared 474; p<0.0001). Similarly, there was a slight decrease in the number of CYP discharged after their first and second DNA from June 2019 (13% face-to-face, 0% telephone and 0% video) to June 2020 (1% faceto-face, 7% telephone and 2% for video).

Discussion
This clinical survey was undertaken at the peak of the first wave of the COVID-19 pandemic in the UK and after a rapid transition in outpatient delivery from predominantly face-to-face appointments to the majority of appointments being delivered via telephone or video. Virtual appointments have previously been shown to be effective when used for selected groups of adult and adolescent patients, such as diabetes, heart failure and mental health follow-up. 4,5 These cohorts of patients often have conditions with discrete measurable outcomes, such as blood glucose or glycated haemoglobin in diabetes. The paediatric setting brings additional telehealth challenges as the clinician may wish to consult with both the child / young person and the carer. Telephone consultations may exclude a younger child from the consultation and video consultations need to take into account the developmental age of the young person and their ability to interact on a virtual platform.
The COVID-19 pandemic led to the adoption of virtual appointments for all specialties and clinical roles across our paediatric trust. Establishing the value of these appointments from subjective clinical opinion and objective operational outcomes of these outpatient appointments was important to ensure that these appointments were not mere 'holding' appointments to 'buy time' until normal outpatient pathways resumed. It was reassuring to find that clinicians reported being able to request admissions, add to waiting lists and discharge CYP when reviewing over the phone or by video. This suggests the clinical perception that clinicians are frequently able to take a history, order investigations, make a diagnosis and start treatment is correct.
It is unsurprising that clinicians reported that they were less likely to pick up non-verbal clues, to be able to assess clinical signs e663    or to make a diagnosis on the telephone, potentially impacting the ability to start treatment. Video appointments were an intermediate option for these aspects of clinical care. This may be particularly relevant in a paediatric setting where the interaction with a younger child may be non-verbal and observational. Clinicians may need to be able to simultaneously consult with the child and the carer. Although clinicians reported that they were less able to organise investigations when consulting through virtual appointments, the outcome data suggest that clinicians continued to add CYP to a waiting list for an admission or further investigation regardless of visit type.

Effectiveness of virtual appointments
However, the proportion of CYP offered further contact after a virtual appointment was just over double that of the patient group seen in a face-to-face appointment. This suggests that the physical review of a patient informs decision making in a significant number of cases and, indeed, this would be surprising if not the case, particularly in a paediatric setting. The majority of respondents indicated that they were always or often able to pick up clinical signs, make a diagnosis and organise investigations when CYP were reviewed face-to-face. This indicates the importance and value of traditional appointment types and clinical teams are best placed to decide if and when appointments are suitable for virtual visits.

Eve Akintomide, Bindi Shah, Shankar Sridharan et al
Clinician confidence in teaching or training CYP/parents was highest for face-to-face, lowest for telephone and intermediate for video. As experience in the use of video consultations increases, it is possible that clinicians may find new ways to physically assess CYP and to offer training. Some areas of medicine lend themselves to virtual assessments more than others, for example, musculoskeletal assessment and intervention. 6 It remains uncertain how much of this difference is related to habit and comfort of previous approaches versus practical difficulties and further research is required.
In this study, it is possible that the clinical decision to convert an appointment to video or telephone could have been based on prior knowledge of the CYP, hence introducing bias such that more of these appointment types resulted in discharge or admission than would occur if the decision to review by virtual appointment had been selected at random. However, in the majority of cases, clinicians were offering face-to-face appointments to CYP by exception and, therefore, for most clinics, the choice of CYP seen by video or telephone appointments was minimal. This may also explain the similar age ranges seen for each clinic type.
When CYP were not brought to the appointment, clinicians were more likely to offer a further appointment in June 2020 than the year before. It is unclear if this increase relates to an increase in the number of virtual appointments or whether clinicians' behaviour changed in response the pandemic as a whole. Discharging a CYP after repeated DNA (WNB) appointments requires ascertaining if there are safeguarding concerns and clinicians may be more cautious than in an adult setting.
It is also important to have clarity that failure to attend is unrelated to issues around equity of access. Technical difficulties have been reported by some patient groups as the primary reason for missing a telehealth appointment. 7 However, the average adult in the UK spends just over 4 hours per day online and there has been a rapid increase in the use of video calls for work and personal use. 8 It is likely that increased familiarity with this form of technology will support the successful use of video visits in healthcare and the age demographic of parents of children under the age of 18 years is likely to reflect a more technologically literate population. 9 Despite this, digital poverty remains a barrier for some families with limited access to Wi-Fi and broadband, which is often costly. Accessing consultations through phones with limited pay as you go credit may also hinder the successful use of phone and video appointments. Overcoming these difficulties may require collaboration between the NHS and network providers to set up free access to NHS consultation platforms.
Clinicians reported that telephone appointments offered the most flexibility with appointment times, and face-to-face appointments the least. While this flexibility may be welcome to clinicians at a time of increased pressure, we do not have data about patient satisfaction in relation to any changes in appointment times. Families were in lockdown and may have been more amenable to schedule changes than at other times. However, of the CYP who did not attend their appointment and were offered follow up, the highest DNA percentage was for booked telephone appointments and lowest for video appointments. It is possible that the forethought required from CYP/carers to set up technology successfully positively influenced the attendance rate.
Understanding the perspective of children, young people, parents and carers following different appointment types is also important and several patient surveys have been published in this area, although mostly in the adult setting. 10 This survey specifically explored clinical experience and outcome at the peak of the first COVID-19 wave. It is possible that time and experience over the past year may alter clinical perceptions, particularly for those users who experience technical issues beyond their control, for instance, poor video or audio connectivity. The concentration required for video consultations is different to face-to-face encounters and it is important that we support clinicians with guidance that allows them to be productive and prevents unnecessary stress. 11,12 An outpatient consultation is more than just the clinician interaction with the CYP. It is a step on an investigative and treatment pathway, which may also include teaching, training, referrals and consideration of psychosocial and safeguarding issues. Understanding how virtual appointments can fit into these pathways may allow opportunities for innovative working and coordination of care. For instance, maximising face-to-face appointments by ensuring that relevant physical examinations or laboratory investigations required by all teams involved takes place in the same visit. Virtual follow-up appointments may then increase in value to patient and clinician.
Although these surveys are subject to response bias, the responses reflected a range of paediatric medical and surgical  Effectiveness of virtual appointments subspecialties and clinical roles within the trust. This suggests that there is a role for telemedicine across paediatric specialties including surgical and therapies. 13,14 The survey was sent as part of trust-wide communication at a time of huge clinical pressure and, therefore, the denominator cannot be determined as it is not possible to measure how many clinicians read the survey information. Despite these limitations, these data suggest that virtual clinics offer alternative appointment options in settings beyond chronic care and that these appointments are reported by clinicians as having value and often result in definitive actions. ■

Supplementary material
Additional supplementary material may be found in the online version of this article at www.rcpjournals.org/fhj: S1 -Telephone and video visits survey questionnaire S2 -Specialty department of survey respondents.