{'fraction_non_alphanumeric': 0.02372219311198971, 'fraction_numerical': 0.005716191111322822, 'mean_word_length': 5.244497323022011, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Effectiveness of virtual and augmented reality for improving knowledge and skills in medical students: protocol for a systematic review

Introduction Virtual reality (VR) and augmented reality (AR) technologies are increasingly being used in undergraduate medical education. We aim to evaluate the effectiveness of VR and AR technologies for improving knowledge and skills in medical students. Methods and analysis Using Best Evidence in Medical Education (BEME) collaboration guidelines, we will search MEDLINE (via PubMed), Education Resources Information Center, PsycINFO, Web of Knowledge, Embase and the Cochrane Central Register of Controlled Trials for English-language records, from January 1990 to March 2021. Randomised trials that studied the use of VR or AR devices for teaching medical students will be included. Studies that assessed other healthcare professionals, or did not have a comparator group, will be excluded. The primary outcome measures relate to medical students’ knowledge and clinical skills. Two reviewers will independently screen studies and assess eligibility based on our prespecified eligibility criteria, and then extract data from each eligible study using a modified BEME coding form. Any disagreements will be resolved by discussion or, if necessary, the involvement of a third reviewer. The BEME Quality Indicators checklist and the Cochrane Risk of Bias Tool will be used to assess the quality of the body of evidence. Where data are of sufficient homogeneity, a meta-analysis using a random-effects model will be conducted. Otherwise, a narrative synthesis approach will be taken and studies will be evaluated based on Kirkpatrick’s levels of educational outcomes and the Synthesis Without Meta-analysis guidelines. Ethics and dissemination Ethical approval is not required for this systematic review as no primary data are being collected. We will disseminate the findings of this review through scientific conferences and through publication in a peer-reviewed journal.


INTRODUCTION
The acquisition of clinical and examination skills is a vital component of medical education. 1 Traditionally, the acquisition of such skills has been via mentorship models from senior clinicians in clinical environments. However, due to increased clinical loads and reduced working hours, it is often difficult for trainees to have the same exposure to practical skills as their predecessors. 2 Furthermore, due to increasing infection control risks and costs associated with training practical skills, the need to develop alternative teaching environments is pressing. 3 4 One solution is the use of immersive technologies for education, which place students in a virtual environment whereby they can perform practical procedures as well as interact with a variety of simulated clinical scenarios. This can be useful in the development of physical examination skills (through using sound, tactile and visual examination findings), development of basic life support skills, surgical training and human anatomy. [5][6][7] Some advantages of such technologies include that they allow for spaced repetition, reduce time burden on senior clinicians and permit students to practice in their own time. The use of simulators also provide real-time feedback, and record data from different attempts, to compare to those of different levels of expertise. 8 In medical education, the three main types of technology used to construct these simulated environments are: virtual reality Strengths and limitations of this study ► First systematic review assessing the effectiveness of virtual reality and augmented reality technologies for improving knowledge and skills outcomes in medical students. ► A rigorous search strategy will be employed, which will involve searching multiple databases, grey literature and for non-indexed trials. ► The quality of evidence will be assessed using the Best Evidence in Medical Education Quality Indicators checklist and the Cochrane Risk of Bias Tool. ► Limitations include the likely heterogeneity of interventions and outcomes investigated.
Open access (VR), augmented reality (AR) and mixed reality (MR) (as defined by Moro et al, 9 in table 1). VR is in essence a method that 'provides a three-dimensional (3D) and dynamic view of structures and the ability of the user to interact with them' via a digital interface. 10 AR is a composite view produced by the superimposing of a computer generated image over a user's view on the real world (eg, using your phone to project an image onto a table). MR is an advancement of AR in that it is multisensory and generally involves the use of a computer headset, earphones and tactile gloves to enhance sensory feedback from the clinical scenario and make it as realistic as possible; however, unlike VR it does not take place within a wholly virtual environment, with certain aspects (ie, the background) still remaining 'real'.

E-Learning and current literature
The use of e-learning in medical education has been explored to varying degrees in the literature. A recent review exploring the barriers and solutions to online learning in medical education identified the need for institutional support and a clear strategy on how e-tools and programmes should be applied, as their application was often undirected and unsuccessful. 8 They also identified the necessity of interfaculty collaboration to create a cohesive e-learning system. Further barriers to successful e-learning included poor technical skills, educational skills and training, which could be addressed by improving the training of educators in computational literacy. 11 Immersive technologies, which constitute one modality of e-learning, are being increasingly used to improve knowledge and skills outcomes. Further research into their role and effectiveness would build the required evidence base for industry and medical education bodies to enhance the development and uptake of e-learning. There have been a number of reviews exploring the use of VR tools for healthcare professionals in different stages of their training, and the vast majority have called for further research in the area. The reviews have primarily focused on healthcare professionals acquiring specific surgical skills (eg, laparoscopy, ear, nose and throat (ENT), ophthalmology) or learning a particular clinical procedure. [12][13][14] A recent comprehensive review evaluated the effectiveness of VR in educating health professionals, and found it to improve knowledge and skills outcomes. However, it focused on all healthcare professionals, and therefore, may have included clinical skills which are not relevant for medical students. 15 While the review attempted to extract outcome measures relating to postintervention knowledge, skills, attitudes, satisfaction and change in behaviour, almost all of the included studies captured either knowledge or skills outcomes. Although extensive, the review only searched for articles up to 2017, since which there have been several studies published. 5 9 16-21 Furthermore, it did not describe the types of intervention used and only included information on their effectiveness. A Cochrane systematic review in 2015 18 aimed to explore the evidence base for introducing VR into ENT training programmes. The studies assessed the technical skills of both surgical trainees and medical students in controlled environments, for either endoscopic sinus or temporal bone surgery. Nine studies were analysed, of which four included medical students. These studies suggested that VR simulation could be a useful supplementary learning tool for medical students, showing improved anatomical identification scores (MD 4.3, 95% CI 2.31 to 6.29) after use of VR simulators. A systematic review by Tang et al 22 investigated the current state and role of AR technologies in medical education. It showed current uses for AR include surgical training, anatomy teaching and as a component of the blended learning approach. Crucially, however, it identified the lack of a review which explores the effectiveness of AR technologies in medical education, and of a standardised assessment tool.
Given the current state of the literature and the burgeoning interest in using immersive technologies to teach medical students, a review is warranted to provide educators as well as researchers with a better understanding of the uses of these tools as documented in the literature, and with information on their effectiveness compared with standard practice. As such, this review aims to evaluate the effectiveness of VR and AR technologies for improving postintervention knowledge and skills outcomes in medical students. It will also describe the characteristics and objectives of the identified devices, as indicated in the literature. Our findings will inform  9

) Abbreviation
Undergraduate medical education Medical students are described as any student undertaking a course of study at a medical school to reach a primary qualification in medicine, enabling them to practice as doctors Virtual Reality (VR) Immersion of a user in a synthetic environment, experienced through their senses (sight, hearing and motion), which mimics properties of the real world through head mounted displays, stereo headphones and motion tracking systems.
Augmented Reality (AR) Superimposition of digital models on to the real world, using a camera and a screen (eg, smartphone or tablet). Users can then interact with the virtual and real-world elements of their environment. Mixed Reality (MR) A significant advancement of AR, whereby virtual objects can be mapped onto the real world. MR headset devices present holographic renderings of images in their display, which can be interacted with. AR, augmented reality; MR, mixed reality; VR, virtual reality.
Open access medical schools and educational bodies on the current state and effectiveness of immersive technologies in undergraduate medical education and will help shape the design of future curricula.

METHODS AND ANALYSIS
The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Protocols guidelines were used to facilitate development of this protocol, see online supplemental appendix 1. We will use the Best Evidence in Medical Education (BEME) collaboration guidelines to conduct our systematic review to answer the following research questions: 1. Are VR technologies effective at improving knowledge and skills outcomes in medical students? 2. Are AR technologies effective at improving knowledge and skills outcomes in medical students? BEME guidelines were chosen as the systematic review framework due to their specificity to medical education methodology. Any significant amendments made to our protocol will be documented and published with the results of our systematic review. Reporting of the review will be informed by the PRISMA guidance and the Synthesis Without Meta-analysis guidelines. 23 The review has been submitted to and is awaiting registration on the International Prospective Register of Systematic Reviews (ID 250531).

Inclusion criteria
The inclusion and exclusion criteria are summarised in table 2.

Population
Only studies including medical students in undergraduate medical education will be included in this review; those focused only on medical professionals or postgraduate learners will be excluded. Studies with both undergraduate and postgraduate learners will be included if data relating to undergraduate students can be separately analysed.

Intervention
Due to the varied nature of VR simulators and the environment being simulated, it is difficult to define what constitutes immersive (VR, AR and MR) technologies. However, we have provided our definitions of these (table 1), and will include all VR/AR delivery devices and levels of immersion. In order to include all relevant studies, all spatially immersive visualisation technologies which simulate an interactive 3D environment will be included, whether or not they include tactile feedback or other sensory feedback.

Study design and comparators
Randomised trials which provide primary data for our outcomes will be included. We will evaluate and interpret data and results carefully in the context of their design and will assess and report the quality of each study type.
We will compare primary outcomes against traditional teaching methods or techniques (which do not use VR/ AR).

Outcome measures
As there is significant heterogeneity in the tasks conducted in VR/AR environments as well as the type of VR/AR used, outlining one specific outcome measure to be extracted is not achievable. 24 Instead, primary outcome measures relating to medical students' knowledge and skills will be extracted.
Knowledge will be defined as students' factual or conceptual understanding, and measured using any validated or non-validated instrument to obtain pretest and post-test scores. In the case of multiple post-test results, the difference between the pretest and first post-test will be calculated to include in our analysis.
Skills will be defined as students' ability to demonstrate a technique or procedure, and measured using validated Open access or non-validated instruments (eg, to obtain pretest and post-test scores, task performance/errors, observed assessment scores).

Search strategy and sources
We will identify studies by searching the following electronic databases: MEDLINE (via PubMed), Education Resources Information Center, PsycINFO, Web of Knowledge, Embase and the Cochrane Central Register of Controlled Trials. Databases will be searched for English-language records, from January 1990 to March 2021. Prior to 1990, the use of digital technologies was limited to basic tasks. Additional relevant studies will be identified based on expert knowledge of the reviewers. Grey literature will be identified by emailing primary authors of included studies and searching conference abstracts from relevant meetings.
We will develop our search strategy using a combination of Medical Subject Headings and keywords related to the following concepts: VR, AR and undergraduate medical education (medical school/medical students). A full electronic search strategy for PubMed is included in online supplemental appendix 2.

Patient and public involvement
No patient involved.

Data collection
Two reviewers (NA and NN) will identify studies which fit our inclusion/exclusion criteria independently using titles and abstracts. If there is any disagreement, then a discussion will take place between the two reviewers using the full text of the article in question. If there is still disagreement, then a third reviewer (AK) will be consulted. Cohen's kappa, a measure that identifies the level of agreement between our reviewers, will be calculated as well. 25 A web-based app called Rayyan will be used to track studies that have been identified. 26 As per PRISMA guidelines, a flow chart will be created once this has been completed to identify the number of studies found using our search terms, the final number of studies and the reason for exclusion of studies not included. 27 Data extraction and management All data extraction from relevant studies will be carried out by two independent reviewers (NA and NN), using a modified version of the BEME coding form which will also include data on several measures, as seen in table 3. Both authors will go through a process of orientation to the tool, before full extraction, to ensure inter-rater reliability to a kappa of at least 0.80 agreement. If needed, a third reviewer will be consulted to resolve disagreements (AK).

Quality assessment
Two independent reviewers (NN and NA) will conduct a risk of bias assessment, and similar to our study selection, if there are any points of disagreement which cannot be resolved then a third reviewer (DMN) will be consulted. We will conduct a formal risk of bias assessment for randomised trials using The Cochrane Risk of Bias Tool. 28 The BEME Quality indicators checklist, developed by Buckley et al 29 will be used to evaluate internal validity. The checklist consists of 11 criteria, listed in table 4, which are designed specifically for studies in medical education as per the recommendations of the BEME collaboration. Each criterion can be listed as 'met, 'unmet' or 'unclear' and a study is required to meet a minimum of seven indicators in order to be deemed high quality.

Synthesis of extracted evidence
First, we will describe all studies that met the inclusion criteria, based on the information extracted using table 3. Information on the type of VR or AR device used, their characteristics, and on the task given or skill being developed (eg, basic scientific knowledge or clinical skill), will provide insights into the role/use of each of the VR or AR technologies being used.
We will take an iterative and responsive approach as we proceed through data extraction and evaluate studies. Where we find sufficient homogeneity to combine, we will follow standard methods for meta-analysis in the Cochrane Handbook, using the random-effects model. 28 In the case of heterogeneous data, we will conduct a comprehensive narrative synthesis of the evidence, and will group and report studies according to a modification of Kirkpatrick's levels of educational outcomes (hierarchy). 30 Narrative synthesis is often criticised for the lack of transparency in the methods used; to aid transparency, Open access we will use the recent Synthesis Without Meta-analysis guidelines. 23

DISCUSSION
The findings of this review will inform educators and institutions around the world on the effectiveness of VR and AR devices for teaching medical students and on how they are being used. This will provide useful information for organising bodies and medical schools when designing future curricula and developing educational interventions.
The outcomes for educational practice include improving the delivery of digital education, and ensuring that investment in digital technologies, including VR and AR, for medical schools, is justified. Institutions will be better informed when exploring such devices, and will be able to ensure they are tailored to and have been tested for, the needs of their students In future, this review could also be used to develop a set of outcome criteria for testing and deploying immersive technologies.

ETHICS AND DISSEMINATION
Ethical approval is not required for this systematic review as no primary data are being collected. We will disseminate the findings of this review at scientific conferences and through publication in a peer-reviewed journal. Twitter Zain Hussain @zhuss95, Debbie Aitken @debbieaitken and Aziz Sheikh @ drazizsheikh Acknowledgements The authors would like to acknowledge Ruth Jenkins, Academic Support Librarian at the University of Edinburgh, for helping develop ofthe search strategy and identify relevant databases. They would also like to acknowledge the BEME Collaboration for their insightful feedback on their Topic and Protocol registration.
Contributors ZH and DMN contributed equally to this paper. ZH and DMN conceived and planned the review, with supervision from AS, DA and AH. ZH, DMN and NA wrote the initial draft, with critical intellectual contributions from all the coauthors (ZS, NN, AK, AH, DA, AS). All authors approved the final version.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.

Patient consent for publication Not required.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.
Open access This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http:// creativecommons. org/ licenses/ by-nc/ 4. 0/. Were the results supported by data from more than one source? BEME, Best Evidence in Medical Education.