{'fraction_non_alphanumeric': 0.03513414016842299, 'fraction_numerical': 0.014840550365836824, 'mean_word_length': 5.298072743080713, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Qualitative Assessment of Studies Evaluating the Classification Accuracy of Personnel Using START in Disaster Triage: A Scoping Review

Background Mass casualty incidents (MCIs) can occur as a consequence of a wide variety of events and often require overwhelming prehospital and emergency support and coordinated emergency response. A variety of disaster triage systems have been developed to assist health care providers in making difficult choices with regards to prioritization of victim treatment. The simple triage and rapid treatment (START) triage system is one of the most widely used triage algorithms; however, the research literature addressing real-world or simulation studies documenting the classification accuracy of personnel using START is lacking. Aims and Objectives To explore the existing literature related to the current state of knowledge about studies assessing the classification accuracy of the START triage system. Design Scoping review based on Arksey and O'Malley's methodological framework and narrative synthesis based on methods described by Popay and colleagues were performed. Results The literature search identified 1,820 citations, of which 32 studies met the inclusion criteria. Thirty were peer-reviewed articles and 28 published in the last 10 years (i.e., 2010 and onward). Primary research studies originated in 13 countries and included 3,706 participants conducting triaging assessments involving 2,950 victims. Included studies consisted of five randomized controlled trials, 17 non-randomized controlled studies, eight descriptive studies, and two mixed-method studies. Simulation techniques, mode of delivery, contextual features, and participants' required skills varied among studies. Overall, there was no consistent reporting of outcomes across studies and results were heterogeneous. Data were extracted from the included studies and categorized into two themes: (1) typology of simulations and (2) START system in MCIs simulations. Each theme contains sub-themes regarding the development of simulation employing START as a system for improving individuals' preparedness. These include types of simulation training, settings, and technologies. Other sub-themes include outcome measures and reference standards. Conclusion This review demonstrates a variety of factors impacting the development and implementation of simulation to assess characteristics of the START system. To further improve simulation-based assessment of triage systems, we recommend the use of reporting guidelines specifically designed for health care simulation research. In particular, reporting of reference standards and test characteristics need to improve in future studies.


INTRODUCTION
Mass casualty incidents (MCIs) can occur as a consequence of a wide variety of events, such as those resulting from emergencies, disasters, or pandemics, and often require enhanced prehospital and emergency supports and coordinated emergency response. When MCIs cause the demand for medical care to exceed capacity, prioritization of patients shifts from treatment of the most severe casualties to an attempt to provide the best care for the highest number of victims. In these situations, medical professionals allocate priority to those who are most likely to benefit from the available resources and have the best chance of survival and recovery (1).
Created in the 1980s, the Simple Triage and Rapid Treatment (START) triage system was developed to be used in the event of a MCI (2), allowing responders to triage a patient in fewer than 60 seconds (s). (3). It has since become widely adopted (4,5), especially in the United States, Canada, Australia and the Israeli-occupied territories (6). Its main goal is to appraise and identify conditions that can lead to death if not treated within 1 h by prioritizing clinical markers of respiration, perfusion, and mental status to identify impaired breathing, severe hemorrhage, and head injury. Responders employing START evaluate victims assigning them to one of four triage categories: deceased/expectant (black), immediate (red), delayed (yellow), and walking wounded/minor (green). Inaccuracies in correctly evaluating victims to a START triage category can result in either under-triage (not recognizing that victims could likely benefit from urgent medical intervention) or overtriage (in which valuable resources are used prematurely or unnecessarily). An effective triage tool should have a high sensitivity to minimize the occurrence of under-triage, but should not undermine specificity to prevent the occurrence of overtriage. Sensitivity and specificity can be determined using the rate of appropriately assigned clinical priority levels for victims of a MCI against a reference standard.
The highly stochastic nature of MCIs, as well as the complexity of subsystem interactions, makes simulation one of the best strategies for preparing individuals and health systems to develop the most efficient procedures. START is often utilized in simulation studies employing a variety of MCI scenarios assessing, for example, the impact of educational interventions, the effect of different simulation technologies, or its performance in comparison to other triage systems (7)(8)(9). A common element in these studies is the evaluation of the ability of participants to apply START in view of various outcome measures of classification accuracy. This is done to assess whether victims are being triaged to the appropriate triage category. Thus, observing simulation strategies employed in different studies and whether participants/trainees are triaging appropriately using one of the most adopted triage systems is an important step to advance studies using simulation in the field of disaster medicine.
Despite the widespread utilization of START across the literature, there was just one published synthesis of the classification accuracy of START. In this recently published systematic review it was found that the accuracy of START is insufficient to serve as a reliable disaster triage tool (10); however, it was noted that the included studies varied considerably in terms of the use of true vs. simulated MCIs, the implementation and conduct of the simulations, as well as the assessors applying the START triage system. While beyond the scope of the systematic review (10), a description of the characteristics of the simulations in which START accuracy is assessed is essential for several reasons (11)(12)(13)(14)(15). First, it can reveal nuances of the interaction of both (simulation techniques and triage systems) and recommend adaptations (if necessary). Second, reproducibility of findings can also be considered. Thus, the research question directing this scoping review is: What is known about simulation studies of MCIs assessing the classification accuracy of the START triage system? The purpose of this scoping review is two-fold: first, to explore the existing literature related to the current state of knowledge about simulation strategies of studies assessing the classification accuracy of the START triage system; second, to consider implications for further research.

METHODS
This scoping review was conducted following the methodological framework described by Arksey and O'Malley (16) including: identifying the design and search question; searching for relevant studies; selection of studies; charting the data; and finally, collating, summarizing and reporting the results. The methods of this study were enhanced by the recommendations of Levac, Colquhoun and and O'Brien (17), which include connecting the research question to the purpose, ensuring that practicality does not limit the findings of the study, and identifying practical implications of the review. We did not engage in the optional stage 6-consultation with the community-in this current study, although such consultation may form a part of future knowledge translation. This scoping review followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses for Scoping Reviews (PRISMA-ScR) (18) (see Supplementary Material 1).

Search Terms and Strategies
Following an initial search to identify publications on the topic, a health sciences librarian (SC) developed a search of nine electronic databases including OVID Medline, OVID EMBASE, OVID Global Health, EBSCO CINAHL, Compendex (Engineering Village), SCOPUS, Proquest Dissertations and Theses Global, Cochrane Library, and PROSPERO. The search strings for each database was adjusted appropriately for different databases and included controlled vocabulary and keywords for three concepts: (1) START, (2)  Search results were exported to RefWorks citation management system (ProQuest, LLC, Ann Arbor, USA) and the Covidence systematic review program (Veritas Health Innovation Ltd, Melbourne, Australia).
To identify additional studies, a search of the gray literature was conducted in May 2020 which included Google Scholar, Controlled-trials.com, a forward search of the included studies using Web of Science SCOPUS, and a search of the references of included studies and relevant reviews. In addition, recent conference abstracts (2017-2020) from Canadian Journal of Emergency Medicine, Academic Emergency Medicine, and Annals of Emergency Medicine were searched. Non-English language papers were translated first via native speaker, or using Google Translate if a native speaker was not available.

Study Screening and Selection
Following the removal of duplicates, the title and abstract of all articles identified in the search were reviewed by two independent reviewers (UDW and SWK) to identify potentially eligible studies based on the inclusion criteria. Once identified, the full-text of all studies classified as potentially eligible were reviewed by two reviewers (UDW and SWK) in duplicate. Decisions of inclusion or exclusion were made independently based on pre-defined inclusion criteria.
To be eligible for inclusion in the current scoping review, studies had to utilize the START triage system either in a true or simulated MCI scenario for the triage of adult victims. Studies that strictly used a modified version of START were not eligible. In addition, studies had to report outcomes related to the classification accuracy of START (i.e., accuracy, over-triage, under-triage, sensitivity, specificity) to be included. Studies were required to consist of a single cohort or multiple groups as long as at least one of the study cohorts were triaged using the START triage system. Non-experimental studies including case-reports, case-series, reviews, and editorials/opinion pieces were excluded.
Reasons for exclusion were documented. Multiple reports of the same study were collated so that each study, rather than each report, was the unit of review. Disagreements regarding study inclusion were resolved via a third-party adjudication (JMF). The results of the search, screening, and selection are reported in full in a PRISMA flow diagram (19).

Charting, Collating, and Reporting the Results
For studies included in the review, pre-specified outcomes were extracted onto standardized forms in Microsoft excel. Data were extracted independently by at least two of three reviewers (JMF, SWK, UDW). Disagreements were settled via discussion between the reviewers and any conflicts that could not be settled were mediated via third party adjudication (BHR, JMF). The primary outcome of interest was the summary of the methods employed to develop the MCI real or simulation study in which START was applied. As such, information regarding the nature of the simulated MCI, how the simulation was implemented, who conducted the assessments, education/training of assessors, and the triage process was collected. Additional extracted outcomes included study characteristics, reporting of classification accuracy outcomes, and details regarding the reference standard. Definition of type of MCI was based on standard definitions (20).

Study Analysis
The heterogeneity in study methods and reported findings required a narrative approach to synthesis. Findings were grouped into themes after careful reading of the final selected publications by two reviewers (SWK, UDW). These groupings were determined in relation to the research question, and in consideration of logical presentation of the findings to a diverse audience of stakeholder readers (researchers, policy developers, educators, etc.). Face validity of the themes was established by a physician specialized in emergency and disaster medicine (JMF) and a physician specialized in emergency medicine and research synthesis (BHR). This process resulted in themes that were derived from the intended scope of the study, and included the reviewers' interpretation of the data. Thematic analysis was developed using the Lancaster University Guidance on the Conduct of Narrative Synthesis in Systematic Reviews (21). Variable labels included in the studies were extracted as "themes" in the same way as conceptual themes are extracted from qualitative research (21). Development of themes was influenced by the theoretical and disciplinary lenses of emergency medicine.

RESULTS
After removing duplicates, the literature search yielded 1,820 citations. Following the screening of titles and abstracts, 349 publications were identified as potentially relevant. Ultimately, full-text screening resulted in the inclusion of 32 studies involving 37 cases/simulations in the review. The PRISMA flow chart of study selection is presented in Figure 1.

Descriptive Summary of the Studies
From the 32 included studies, 30 were peer-reviewed articles, one was a conference abstract (22), and one was a master's thesis (23). The included studies were published between the years 2005 and 2019, with 28 published in the last 10 years (i.e., 2010 and onward). Studies originated from 13 countries; the United States of America (n = 12), Italy (n = 5) and Canada (n = 4) accounted for the majority of them. Most studies were published in English, with the exception of two (24,25).
Together, these studies involved 3,706 participants conducting triaging assessments involving 2,950 victims. Participants conducting the triage assessment were nurses, physicians, pharmacists, emergency medical technicians, paramedics, first responders, firefighters, non-medical personnel, as well as students from different medical areas, such as paramedic, nursing, medical and various levels of training. The majority of the studies (n = 25) did not specify whether the participants conducting the triage assessment had prior experience with real or simulated disaster events. Tables 1, 2 presents a descriptive summary of included studies that align with the objective of the scoping review.

Narrative Summary of the Studies
Thematic analysis of the charted findings led to the identification of two themes: (1) typology of simulations and (2) START system in MCIs simulations. Each theme contains sub-themes regarding the development of simulation employing START as a system for improving individuals' preparedness.

Theme 1: Typology of Simulations
This theme explores the common types and characteristics of simulations employed in the studies. Sub-themes include simulation technologies, simulation settings, disaster types, assessors and their training/experiences in MCI (see Table 3).  Determine the e?ciency in the execution of the START triage, comparing virtual reality to clinical simulation in a mass-casualty incident.
• Quantitative non-randomized comparative study • Assessors: emergency and special care nursing master's students (n = 67) • Assessors prior experience with MCI: not reported • Victims: actors (3rd year students from the superior school of dramatic arts) + virtual reality cases (n = 20) •

(Continued)
Frontiers in Public Health | www.frontiersin.org Develop a core curriculum of disaster medicine centered on blended learning and simulation tools • Quantitative non-randomized comparative study • Assessors: medical students (n = 524) • Assessors prior experience with MCI: yes (n = 37) • Victims: computer-based cases (n = 30) • The blended approach and the use of simulation tools were appreciated by all participants and successfully increased participants' knowledge of disaster medicine and basic competencies in performing mass-casualty triage.
Ingrassia et al. (27), Italy Explore the ability of virtual reality simulation, compared with live simulation, to test mass casualty triage skills, in terms of triage accuracy, intervention correctness, and speed to complete triage, of naive medical students using the START triage algorithm in a simulated mass-casualty incident scenario and to detect the increase in this expertise after a brief learning session on mass casualty triage.
• Quantitative randomized controlled trial • Assessors: medical students (n = 56) • Assessors prior experience with MCI: no (n = 0) • Victims: actors (3rd year medical students, n = 10) • No significant differences in START triage accuracy when comparing virtual reality and live simulation. • Training could improve the ability to correctly categorize patients.

Izumida et al. (39), Japan
Propose a triage training system in which the expression of information changes according to the skill level of each trainee.
• Quantitative non-randomized comparative study • Assessors: university students and graduated school students (n = 12) • Assessors prior experience with MCI: no (n = 0) • Victims: virtual reality cases (n = 10) • The results revealed the system was e?ective to implement triage quickly and accurately.

Jain et al. (28), Canada
Compare unmanned aerial vehicle technology (UAV) to standard practice in triaging casualties at a mass-casualty incident • Quantitative randomized controlled trial • Assessors: second-year primary care paramedic students (n = 20) + advance care paramedic students (n = 20) • Assessors prior experience with MCI: not reported • Victims: actors (n = 10) • No significant differences in START triage accuracy when comparing UAV technology and standard practice. • One-hundred-percent accuracy was noted between both groups. • A non-clinical statistical difference in the time to completion with UAV groups was noted.
Kahn et al. (46), USA Analyzed whether START is accurate in assigning acuity levels to victims of a real train crash.
• Quantitative descriptive study • Assessors: paramedics (n = not reported) • Assessors prior experience with MCI: not reported • Victims: victims of a real mass-casualty incident (n = 265) • No triage level met both the 90% sensitivity and 90% specificity requirement set forth in the hypothesis. • START ensured acceptable levels of red under-triage: 100% sensitive (95% CI 16% to 100%). • START ensured acceptable levels of green under-triage: 89.3% specific (95% CI 72% to 98%). • START incorporated a substantial amount of over-triage. • The Obuchowski statistic was 0.81, meaning that victims from a higher-acuity outcome group had an 81% chance of assignment to a higher-acuity triage category. • This analysis demonstrates poor agreement between triage levels assigned by START at a train crash and a priori outcomes criteria for each level.
Khan (29), Qatar Evaluate the mass-casualty incident triage skills of the medical staff like doctors and nurses at Hamad General Hospital Emergency Department.
• Quantitative randomized controlled trial • Assessors: physicians (n = 50) + nurses (n = 50) • Assessors prior experience with MCI: not reported • Victims: paper-based cases + computer bases cases (n = 40) • The study results report 90% triage accuracy in the intervention group and 70% in control group with a difference of 20-30%. • The over and under triaging were 5% for both in the intervention side but 20%, 10% respectively in the control side. • The reliability also improved in the intervention group due to repeated training.

(Continued)
Frontiers in Public Health | www.frontiersin.org  Assess the effectiveness of written and moulage scenarios using video instruction for mass-casualty triage by evaluating skill retention at six months post intervention.
• Quantitative non-randomized comparative study • Assessors: emergency medical technician + emergency medical technician paramedics (n = 45) • Assessors prior experience with MCI: not reported • Victims: actors (n = 12) + paper-based cases (n = 12) • No significant differences between written and moulage testing results at either initial testing or at six months. • Prior skill level did not in?uence test performance on the type of testing conducted or long-term retention of triage skills. • There was a significant decrease in performance between initial and six-month testing, indicating skill decay and loss of retention of triage skills after an extended nonuse period.
Riza'I et al. (41), Indonesia Evaluate the accuracy of triage decisions made by first-year medical students after receiving two intervention methods.
• Quantitative non-randomized comparative study • Assessors: first-year medical students (n = 54) • Assessors prior experience with MCI: not reported • Victims: paper-based cases (n = 10) • The mean of method 2 (8.03 ± 0.72) was significantly improved for correct triage compared with the mean of method 1 (6.33± 1.63) for 54 students (P < 0.001 • Overall triage accuracy rate was 78%, exceeding data suggesting that the triage accuracy rates using different triage strategy algorithms are approximately 45% to 55%. • Contrary to expectations, the triage to transport times for the green-, yellow-, and red-tag patients were similar. Silvestri et al. (9), USA Compare the START and SALT classifications of patients to a published reference standard category, and evaluated the accuracy of the START method applied by emergency medical services personnel in a field simulation.
• Quantitative non-randomized comparative study • Assessors: EMS personnel (n = not reported) • Assessors prior experience with MCI: not reported • Victims: actors + mannequin (n = 82) • SALT triage system was overall more accurate triage method than START at classifying patients, specifically in the delayed and immediate categories. • In the field exercise, paramedic use of the START methodology yielded a higher rate of under-triage compared to the SALT classification.

(Continued)
Frontiers in Public Health | www.frontiersin.org  • The trainees' scores increased significantly after the training (P < 0.001). • Improvement (post-test score minus pre-test score) was not significantly different among the occupational groups. • Medical (physicians, nurses, and EMTs) and non-medical groups displayed similar improvement, but post-training scores were significantly lower in the non-medical participants (P < 0.001). • Trainees with prior triage training had higher pre-training scores (P < 0.05), but no significant improvement was evident in the non-medical personnel with prior triage training. • The level of performance of triage by non-medical personnel was less than optimal (post training score = 9.32), but the ability to divide casualties into minor (green) and major (yellow and red) groups was reliable.

Simulation Technologies
The technology employed in the delivery of simulations varied considerably across the literature (see Table 3). In a few studies, victims from MCI were re-assessed retrospectively using real mass casualty incident data (23,34,46) or data from a previous simulation exercise (24). In some studies, paperbased simulations were employed in which a scenario was described involving victims of a MCI and participants were asked to review and apply START (8,26,29,32,37,41,43,47). Other studies employed computer-based simulations, which generally involved a multimedia-facilitated activity (28-30, 35, 36, 40, 49). Computer-based simulations varied from use of latent images to more complex software in which a series of victims of a disaster or MCI arrive to an ED or other hospital setting requiring participates to triage presenting victims via START. The majority of the studies required participants to partake in a live simulation exercise, of which participants are at the scene of a simulated MCI and are required to apply START to actors or manikins representing the victims (8,9,23,25,27,28,31,33,38,42,44,45,49,50).
Within the last 6 years, studies started utilizing virtual reality, where participants usually wear a head-mounted display allowing them to have a 360 • visual of images and videos (27,33,39,50). Virtual reality was also used by live broadcasting a MCI scenario to participants; however, instead of wearing a head-mounted display, participants guided a person via video call (7). The guide at the scene would verbalize information needed for participants, so that they could evaluate each victim and assign them the appropriate triage category (7).
It should be noted that some of these studies applied a mixed technology approach when implementing their simulations (8, 23, 27-29, 33, 49, 50). For example, one study employed the use of unmanned aerial vehicles to allow paramedical students to survey a simulated multi-vehicular accident with live actors with moulage playing the victims (28). Other studies compared different technologies for implementing simulations such as virtual reality-based simulation vs. live simulation with actors (27,33,50). Two studies did not report the technology employed to perform simulation exercises (22,48), while another study reported using moulage without specifying whether manikins or live actors were used (8). • Ordering of different triage modalities may have impacted triage time.
• Simulation conducted differently between groups.
• Unintentionally ambiguous scenarios made triage level determination difficult.
Challen and Walter (34) Stated • There was a paucity of available documentation.
• Data collection challenges since staff at the incident scenes were using their own tags as well as official supplies.
• There was missing data within the medical records.
Crews (23) Not stated • Lack of previous studies.
• Confinement of geographical region studied.
• Simulation was done with paper-based assessment tool, which is an oversimplification of actual triage.
• It only includes adult victims. • Although similar, the two scenarios were not identical since there were slight differences with regard to the resources available to each group.
• The evaluation of performance indicators could be observer biased.
• Since it was necessary to set a time limit, it is clear that the overall evaluation of the hospital response to the simulations is potentially biased by shorter simulation time. • Possibly over-triage bias as researchers did observe that some of the assigned triage levels differed from what strict application of the START algorithm would have mandated.
• The black, or "deceased," category was not examined.
• Using only one tool or system of triage (START).
• Issues during data collection.

(Continued)
Frontiers in Public Health | www.frontiersin.org • Displacement of the victims from the triage area to the canvases for care during simulation.
• Place of collection and the limitation of the material used in the simulation to care for the victims were not well-defined for the participants as well.
• Pictures only showed one victim at a time, which isn't realistic for an MCI.
• This study failed to show significance for its secondary objective of improvement in triage accuracy.
McCoy et al. (7) Stated • Voluntary enrolment in the course, thus sample may not be representative of all professions.
• Not designed as an observational-analytical study so not powered to detect differences between groups.
• Heterogeneous group of "other" participants. • Exercises assumed unlimited transport and treatment resources.
• The use of mannequins slightly impacted the study.
• The impact of the familiarity of the scene is unknown.
• STM triage and resource management software was not tested. • The same written test was given before and after the training session, which may rise the concern of improvement comes from short-term practice but not learning.
Reported. Not reported. *Potential conflict of interest.

Assessors
Studies employed a variety of medical professionals to assess the classification accuracy of START across the literature (see Table 1). First responders/paramedics were most commonly recruited to participate in studies requiring to apply START (8,9,22,23,25,31,38,44,46,49), with two studies specifically recruiting firefighters (24,35). Students of various professions, including a variety of college-level (36,39), medical (27,32,40,41), nursing (45), and paramedic students (28,50) were the second most common participants recruited to apply START. Other professionals including nurses and physicians were also recruited; however, studies tended to assess the ability of a mix of health professionals to accurately apply START (7, 29-31, 37, 42, 43, 47). Few studies compared the differences in the accuracy of START among different healthcare professionals (7,25).

Experience and Training in Disaster Medicine and START
Seven studies specifically reported participants had previous experience with the START system (9,30,35,37,38,44,46) and 11 studies specified whether or not participants had any prior experience with MCI (27,30,32,36,37,39,40,42,43,47,50). Seven of the 21 studies that did not report participants' prior MCI experience also did not involve any MCI education intervention or reported whether participants were trained in MCI triage for the specific study (22-24, 34, 35, 48, 49).

Theme 2: START System in MCIs Simulations
This theme explores how the classification accuracy of START triage system was assessed across the different studies (see Table 4).

Diagnostic Properties
A summary of the various diagnostic outcomes assessed across the studies are provided in Table 4. As per the inclusion criteria,  • Over triage (green sub-group, yellow sub-group, and black sub-group) • START with virtual reality on day 3 • Under triage (green sub-group, yellow sub-group, and red sub-group) • START with live simulation on day 1 • START with live simulation on day 3  Lee and Franc (30) • Accuracy (total and all sub-groups, with exception of black) • START (one-step triage) Expert opinion • Over-triage (total and all subgroups, with the exception of two-steps red sub-group, and one-and two-step black sub-groups) • START and CTAS (two-step triage) • Under-triage (total and all subgroups, with the exception of two-steps red sub-group, and one-and two-step black sub-groups) • Under-triage (red classified as black) • Under-triage (red classified as yellow)   all of the studies reported at least one outcome related to the classification accuracy of START. All but two studies (34,46) assessed the accuracy of START by comparing participants' performance (correctly matching of triage levels to a reference standard). With the exception of two studies (9,48), all studies measuring classification accuracy of participants performance reported the overall accuracy for all victims. In addition, some studies also reported the accuracy of participants' performance based on the triage subgroups of START (i.e., black, red, yellow, and green) (8,25,26,30,31,35,42,44,47,48). Still within accuracy of participants performance, some studies teased out the proportion of patients over and under-triaged within the START triage subgroups (8, 9, 23-27, 29, 30, 32, 35, 37, 38, 41, 42, 44, 46, 47, 49). Only two studies reported on outcomes related to START diagnostic properties, such as specificity, sensitivity, positive and negative predictive values, or likelihood ratios (34,46).
Lastly, the vast majority of included studies (n = 22) did not specify which prerequisite they used to measure classification accuracy (i.e., a reference standard). When specified, the reference standard was most commonly described as expert opinions (9, 22, 23, 30-33, 37, 42) followed by the Baxt and Upeniek criticality (34), and the modified Baxt criteria (46). From the nine studies using experts' opinions as the reference standard, five studies did not specify the background of the experts or how this consensus was determined (22,23,30,31,33).

DISCUSSION
Given the widespread use of START for the triage of victims in real-world MCI's, training simulations, as well as assessing educational interventions, this scoping review aimed at exploring and summarizing the existing literature related to the current state of knowledge regarding studies assessing the classification accuracy of START. Gaining a better understanding of the literature helped us to identify gaps in reporting that may hold implications for future studies. Through an extensive and systematic search of the literature, 32 studies assessing the classification accuracy of START were identified. These studies were conducted around the world, with the majority of the studies published in the last 10 years, indicating that knowledge about simulation strategies using START for triage is a global concern and growing field of research.
Over the years, the methods used for simulations has changed as technological advancements occurred. For example, computer simulations replaced the early text-based paper exercises, and live simulations with actors have more recently been replaced by virtual reality technology. Studies included in our review employed different types of simulation technologies and, despite technological advancements, some of the most recently published studies employed technologies ranging from basic text-based exercises to the more advanced ones. This may be attributable to the high cost of using more advanced technologies during simulations, and the paucity of funding opportunities for disaster research within the research ecosystem. Although simulation can be effective at preparing individuals and systems to effectively deal with MCIs, it comes at a price. Different types of simulation technologies have different costs aggregated to them including training, equipment and systems, technicians, laboratory setup, maintenance and so on. In fact, the elevated costs of many simulation technologies has been a key criticism of medical training using simulation (51, 52). Therefore, it is reasonable that researchers developing MCI studies using simulation consider their population needs, available resources and return on investment to determine which type of technology they will study and adopt.
Other common themes arose when reviewing the articles, one of which was the reporting and implementation of the simulation. For the most part, studies provided satisfactory details regarding how the simulation exercises were conducted; however, the establishment of more systematic reporting is warranted. As discussed below, many studies lacked information that should be included in articles involving MCI simulation for them to be transparent, reproducible, and usable (53)(54)(55).
This review found that some important details regarding the methodologies of the studies and classification accuracy assessment were inconsistently reported across the literature. Approximately a third of the studies assessing the classification accuracy of START failed to report the type of MCI from which the victims were being triaged. Almost half of the studies did not specify the source of disaster scenarios-whether or not the MCI was based on a real event or created by the research staff, healthcare professionals, or disaster medicine experts. In many studies using live simulation, it was unclear if the mock victims had previous training on how to simulate clinical conditions or how these mock victims were prepared (e.g., use of make-up). At this time, it unclear whether the complexity of the disaster or MCI affects the classification accuracy of disaster triage, but this might be worth exploring in future studies.
Another common theme explored in this study was the reporting regarding the assessors of START and their experiences. It was not surprising that the majority of studies assessed the classification accuracy of paramedic/EMS providers to apply START; however, it was perhaps a little surprising that students (including paramedical, nursing, and medical) were the second most common assessors of START across the literature. It is not clear why this is the case. It could be that studies assessing novel technologies for simulations or triage methods may see students as a population of participants more available, willing and able to embrace novel technologies. In addition, students are more likely to lack any prior experience in disaster triage or START, allowing researchers to assess the impact of training or educational interventions on START classification accuracy.
A fundamental methodological bias associated with this literature is a lack of transparency which impacts the trustworthiness of the science. More than a third of the studies did not state if there was any potential conflict of interest. Over two-thirds did not state if there was any funding source. In addition, several studies did not acknowledge any limitations to the study, and the ones acknowledging them overlooked or reduced to simplistic and minimally relevant themes (e.g., single institution study or small sample size) (56). With respect to the assessment of the classification accuracy of START, while the majority of the studies reported overall accuracy, a third of them did not report under-and over-triage. It is vital for studies assessing triage accuracy to provide a full assessment of the classification accuracy of START. Beneficial triage decisions direct victims to the most appropriate hospitals, resulting in lower mortality and better resource allocation (57).
Yet, one of the most concerning issues we found in this review exploring the current state of knowledge of studies assessing the classification accuracy of the START system was that two-thirds of the studies completely lacked details regarding the reference standard to which START was being compared. When a reference standard was reported, the most common was expert opinion, although details regarding the credentials of the experts were not provided. The traditional classification accuracy paradigm is based on studies that compare the results of the system under evaluation (index system) with the results of a reference standard, and it is regarded as the soundest method to determine the classification accuracy of the system or measure participants' performance. To appraise the classification accuracy of the index test, its results are compared with the results of the reference standard; subsequently indicators of accuracy can be determined. The reference standard is therefore an important determinant of the classification accuracy. From a theoretical perspective the use of an appropriate reference standard is critical and the lack of information regarding it impacts the confidence readers have in research findings.

Strengths and Limitations
We aimed at using precise and transparent review methods when conducting (16,17) and reporting this scoping review (18). A comprehensive approach using several appropriate databases without language restrictions improved the rigor of the review. Consistent with the purpose of a scoping review, we expanded the literature search from January 1983 until March 2020, so that more literature sources could be identified, and findings could truly reflect the state of knowledge. The search words were selected by the researchers and refined by an expert health librarian. In addition, the reference lists of the included articles were forward searched. To reduce the risk of selection bias, this review utilized two independent reviewers to assess and identify potential eligible studies. Lastly, the use of Refworks and Covidence software supported meticulous documentation of screening decisions.
There were, however, some limitations of this scoping review. First, since this review did not pursue quality appraisal, we were not able to speak of the quality of the studies in the field assessing the classification accuracy of START, which could have resulted in inclusion of studies with comprised research quality and incomplete synthesis. Therefore, it is recommended that the findings should be used with caution and applied in research and practice after careful scrutiny. Second, 87.5% (n = 28) of the reviewed studies originated from developed countries which limits the extrapolation of findings to low-and middle-income countries. Third, the results of this scoping review may have been impacted by selective reporting within the included studies. While contacting the study authors could have helped clarify aspects of the simulation, triage assessment, or accuracy outcomes that were unclear or not reported, the objective of this review was to provide an assessment of studies assessing START accuracy based on what is reported in the available literature. Lastly, as with any review, there is a risk of publication bias, particularly among studies assessing the impact of novel interventions on triage classification accuracy.

CONCLUSION
Studies included in this scoping review provided satisfactory details on how their simulations were conducted. However, we found there is room for improvement in view of insufficient information regarding location where simulation exercises were performed, the type of disaster they were simulating, the source of the MCI event, the characteristics of the victims, whether or not participants had any prior experience with MCI triage, and potential source of bias. To further improve simulation-based assessment of triage systems, it is important that stakeholders are mindful of the complexity of subsystem interactions. It is recommended that if simulations are used for assessment purposes, they should be based in a systematic appreciation of the whole system. Future research could be more explicit about the knowledge upon which simulation training is based to allow for description of core theoretical and operational definitions, identification of the function of each component, promotion of similar construct measurement, reporting of findings in a common language, as well as replication and comparison of findings across studies. We recommend the use of reporting guidelines such as the "reporting guidelines for health care simulation research: extensions to the CONSORT and STROBE statements" (11). In particular, incomplete reporting of the reference standards and accuracy needs to be addressed and reported in future studies.
We recommend the development of a systematic review with meta-synthesis to assess overall accuracy, rate of undertriage, and rate of over-triage using the START method, as well as to obtain specific rates of accuracy for each of the four START categories: red, yellow, green, and black. A systematic review with meta-synthesis will allow the combination of results ensuring reliability across a number of studies, while assessing and minimizing bias. As a result, reliable and scientifically derived findings can be obtained for research and clinical practice.

DATA AVAILABILITY STATEMENT
The original contributions presented in the study are included in the article/Supplementary Material, further inquiries can be directed to the corresponding author.