{'fraction_non_alphanumeric': 0.01943507061617298, 'fraction_numerical': 0.006686664166979128, 'mean_word_length': 5.375697211155378, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Metrics Development for Minimal Invasive Unilateral Laminotomy for Bilateral Decompression of Lumbar Spinal Stenosis With and Without Spondylolisthesis by an International Expert Panel

Study Design: Prospective study. Objectives: To develop, operationally define, and seek consensus from procedure experts on the metrics that best characterize a reference approach to the performance of a minimally invasive unilateral laminotomy for bilateral decompression (ULBD) for lumbar spinal stenosis. Methods: A Metrics Group consisting of 3 experienced spine surgeons (2 neurosurgeons, 1 orthopedic surgeon), each with over 25 years of clinical practice, and an educational expert formed the Metrics Group that characterized a lumbar decompression surgery for spinal stenosis as a “reference” procedure. In a modified Delphi panel, 26 spine surgeons from 14 countries critiqued these metrics and their operational definitions before reaching consensus. Results: Performance metrics consisting of 6 phases with 42 steps, 21 errors, and 17 sentinel errors were identified that characterize the procedure. During the peer review, these were evaluated, modified, and agreed. Conclusions: Surgical procedures can be broken down into elemental tasks necessary for the safe and effective completion of a reference approach to a specified surgical procedure. Spinal experts from 16 countries reached consensus on performance metrics for the procedure. This metric-based characterization can be used in a training curriculum and also for assessment of training and performance in clinical practice.


Introduction
Neurosurgical and orthopedic spine surgery programs rely on cadaveric and operating room training as their standard teaching methods. As these training forms can be limited by either institutional infrastructure and/or biohazard regulations with operating room time even more limited by work time regulations, legal issues, and high costs. 1,2 Furthermore, the surgical learning curve is associated with an increased risk of complication and higher morbidity and mortality for the patient. 3,4 Given that cadavers and animals tend to have different characteristics when compared to a living patient training options of specific procedure steps on cadavers or animal models have limitations and new options would be valuable.
Surgical training, at both trainee and master levels, intends to enable a surgeon to acquire the theoretical and practical knowledge necessary to perform a designated surgical procedure skillfully, reliably, and safely. Many experienced surgeons who expertly perform a specific procedure are able to identify and agree on the essential "steps" to be completed and "errors" to be avoided, whereas they rarely think about the procedures they perform with the level of detail needed to identify those key features. 5 By psychologist definition those key features have to be subjected to a detailed task analysis specifying the order, duration, and result of a specific action and determine them to be unambiguously measurable to be called metrics. 6 The metrics do not have to capture every aspect of performance, but should allow differentiation in the way developed by Dreyfus and Dreyfus 7 to serve as a tool to evaluate effectiveness with a quantitative score.
The systematic implementation of a simulation curriculum in neurosurgery or orthopedic training programs has been proven to be feasible, is favorably regarded, and has a positive impact on trainees of all levels. 8 Therefore, a few academic centers around the world have focused on the role of highly realistic spine surgery simulation models to train spine surgeons 9-14 and even though simulation training is not a substitute for operating room experience, it may decrease the time teaching intraoperatively and reduce the morbidity secondary to resident surgical errors.
Simulation-based medical education with deliberate practice was proven to work even better in technical skill acquisition and maintenance for several clinical skills than traditional apprenticeship-styled clinical education. 15 The aim of this study was (1) to develop and objectively define performance metrics that comprehensively characterize a minimal invasive unilateral laminotomy for bilateral decompression of lumbar spinal stenosis with and without spondylolisthesis (Figures 1 and 2), (2) to see if the metrics are assessible on a simulator, and (3) to seek consensus on the completeness and appropriateness of the metrics from a panel of experienced orthopedic and neurological surgeons.

Metric Definitions
As described before metrics are composed of different parts, such as phases, steps, errors and sentinel errors. 16 Metric: A standard of measurement of quantitative assessments used for objective evaluations to make comparisons or to track performance. Procedure phase: A group or series of integrally related events or actions that, when combined with other phases, make up or constitute a complete operative procedure.
Step: A component task, the series aggregate of which constitutes the completion of a specific procedure. Error: A deviation from optimal performance. Sentinel error: An occurrence involving a serious deviation from optimal performance of a procedure that exposes the patient to serious risk but falls short of being a critical error (ie, life threatening)

Metrics Development
Three experienced spine surgeons (2 neurosurgeons, 1 orthopedic surgeon), each with over 25 years of clinical practice, and an educational expert formed the Metrics Group that characterized a lumbar decompression surgery for spinal stenosis as a "reference" procedure. In the first step, all 3 experts individually described their classic lumbar decompression procedure.
As it became evident that this resulted in a nondetailed description with just few steps identified, the next task was to explain it as they would explain it to a resident or fellow. Besides seeking consensus in the description and reviewing the current literature, 17-20 a series of 9 videos of lumbar decompressions  available online (Google, YouTube, VuMedi, PubMed) were reviewed in detail. Based on a detailed task analysis and deconstruction process 6 as well as decades of experience practicing and teaching spinal surgery the units of performance that are integral to skilled task performance were identified. During individual face-to-face meetings, phone calls, and four 1.5-to 2-hour online conferences along with countless email exchanges metric units were identified and then unambiguously defined so that they could be subsequently scored as either occurring or not occurring with a high degree of reliability. For the online sessions, the use of Skype videoconferencing (Microsoft, Redmond, WA; available at www.skype.com) enabled the investigators who reside in different geographic locations to look at videos simultaneously and discuss results facing the same screens. The metrics included the phases in general order as well as the specific steps and the instruments used. The metrics also described for each procedural step what should not be done (errors) thus characterizing behavior that deviated from optimal performance. 21 The intent again, was to create unambiguous operational definitions (rather than descriptions) for each performance unit namely procedure steps, errors, and sentinel errors. The goal was to characterize a "reference" minimal invasive decompression procedure without unusual or complex pathology and subsequent unexpected obstacles.

Metric Stress Testing and Reliability of Identification
With the Metrics Group content with characterization of the procedure, the metrics were "stress tested" by subjecting them to an analysis of reliability assessing how independently scored in blinded fashion. Eight video recordings of complete minimal invasive decompression procedures that were performed by surgeons using either tubular or spatula retractors were independently reviewed and scored in binary fashion as either occurring or not. Differences in the scoring of each metric by the reviewers were discussed and if necessary, definitions of steps and errors were clarified, with the items modified, or dropped and new ones added until the Metrics Group was satisfied that the metrics accurately and unambiguously characterized the specifics of a minimal invasive decompression procedure. The extent of agreement between 2 raters ranged between 0 ¼ no agreement and 1 ¼ complete agreement.

Experts Perform the Procedure to Evaluate the Metrics
To test the metrics further each expert performed 2 procedures on a simulator while being scored individually by the other 2 using the metrics. The procedures were recorded, and the records reviewed by all experts conjointly. Differences in the results and scores once again were discussed with steps missing or unnecessary deleted or modified in 3 phases (Table 1).

Modified Delphi Panel to Evaluate the Metrics
The metrics were then sent for appraisal to a group of surgeons who were also expert in the performance of the procedure. A questionnaire was developed by the Metrics Group seeking consensus for each of the procedural steps and potential errors that needed to be evaluated individually. The questionnaire was then sent by the educational specialist via SurveyMonkey (Dublin, Ireland; available at http://surverymonkey.com) to every surgeon who agreed to participate. While the metrics developed by the Metrics Group for a reference procedure for unilateral approach to a bilateral decompression for lumbar stenosis might not depict the exact techniques used by the individual panelists, it was agreed that the operative steps accurately reflect the essential components of the procedure.
An affirmative vote by a participant indicated that the metric definition presented was accurate and acceptable as written but not necessarily that it was the manner in which that particular panelist might have chosen to complete the step. Therefore, panel members were encouraged to comment on the metrics even though they might not perform the steps in the exact same way. "Consensus" meant that panel members voted unanimity and that a particular metric definition was "not wrong or inappropriate."

Subjects
Twenty-six experienced spine surgeons from 14 countries ( Table 2) were asked to take part in an expert panel. Participants were selected (to ensure regional representation) from the international and regional faculty of AOSpine as well as Society for Minimally Invasive Spine Surgery (SMISS) and by their experience in minimally invasive spine surgery providing an equal distribution of orthopedic surgeons and neurosurgeons.

Procedure Phases
The procedural metrics resulting from task deconstruction were grouped into 6 phases. Each phase (eg, "Identification of ipsilateral hemilamina ipsilateral hemilaminotomy, and identification of ligamentum flavum" or "Contralateral decompression") contains a series of related, unambiguously defined, observable procedure steps with specific beginning and ending points (Table 3). By processing the information from the expert operations and the Delphi panel survey all phases were accepted with changes, but no extra phase was proposed or was a phase deleted (Table 4) while phase V was renamed. Phases I and VI were agreed on with just linguistic changes, while wording and content was altered within the other phases (II-V)

Procedure Steps
Within the reference procedure, 44 steps were identified by the Metrics Group. After the expert operations and reevaluation of the steps, 1 was added, 1 was deleted, and 3 were modified. These were distributed unequally within the phases with the most occurring in the fifth phase ("contralateral decompression"). After discussion of the feedback from the expert panel, 2 steps were deleted and 11 were modified. (Table 5).

Procedure Errors and Sentinel Errors
All potential errors identified had been noted to occur during the stress testing of the metrics. Identical errors and sentinel errors could occur during different phases of the procedure (eg, "dural tear with CSF leak"), but were named within the phases individually. Overall, 18 errors and 20 sentinel errors were defined by the Metrics Group and affirmed. No errors and sentinel errors were added, but 1 sentinel error was deleted and 1 sentinel error occurring in 2 phases was changed to an error instead of a sentinel error. Furthermore, changes were made in 3 errors adding information or changing wording. In the end, 21 errors and 17 sentinel errors were defined (Tables 6 and 7).   After affirmation of the panel suggestions the metrics were agreed as an accurate characterization of a minimal invasive unilateral laminotomy for bilateral decompression for lumbar spinal stenosis. An example of the complete metrics and steps for a phase is shown in Figure 3.

Discussion
Assessment of medical education and competence is always difficult and becomes even more challenging within postgraduate training. Ideally, the assessment of competence should provide insight into actual performance as well as the capacity to find and generate new knowledge and improve overall performance. Competence is contextual, reflecting the relationship between a person's abilities and the tasks he or she is required to perform in a particular situation in the real world.
Medical training and its assessment have to meet certain criteria. The first criterion, reliability, refers to the reproducibility, consistency, and generality of the results over time. The second criterion, feasibility, reasons that, in order to be accepted, a new tool should be cost-effective, easy to execute, and not timeconsuming. The third criterion, validity, asks if the tool assesses what it is supposed to assess. Validity is divided into 5 subcategories: predictive validity, which asks if the tool can predict future performance; face validity, which asks if the tool reflects real life; construct validity, which asks if the tool measures what it is designed to measure; content validity, which asks if the domain that we are assessing is assessed by the tool; and criterion-related validity, which asks if the result assessed by a tool correlates with that measured by the current gold standard tool.
Metric development is therefore crucial as the metrics build the fundament of good medical training. Metrics define how the training should be characterized and the procedure   performed by the trainee as well as grant the opportunity for assessment of the trainee's performance and progress. In contrast to other, mostly quantitative approaches, metrics provide for an individual not associated with the initial development to use them to score performance reliably as they provide precise definitions of performance and a binary scoring system. 6,22,23 Especially training in spine surgery varies a lot due to regional and national differences as well as differences between neurosurgical and orthopedic approaches to surgical procedures. Furthermore, with duty hour limitations and regulatory pressure for enhanced quality and outcomes limitations occur in trainees' access to fundamental skills training. In addition, a cultural shift is seen from competence to mastery learning. 24 Fundamental skills training as part of a curriculum might help mitigate this challenge to surgical education. 12,25 This is what we aim to achieve.

Conclusion
With a total of 29 surgical experts from 14 countries achieving consensus on the metric-based definition of a reference procedure, this study validates that definition and exact characterization of the phases and steps depicting a reference procedure for a minimal invasive unilateral laminotomy for bilateral decompression of lumbar spinal stenosis is possible. Furthermore, face and content validity for the resulting steps and error metrics can be obtained by simulator-based procedures by experts and through review by a panel of experts in a modified Delphi panel.