{'fraction_non_alphanumeric': 0.012588512981904013, 'fraction_numerical': 0.0, 'mean_word_length': 5.659685863874346, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Assessing Sources of Score Variability in a Multisite Medical Performance Assessment: An Application of Hierarchial Line Modeling

Standardized patient (SP) examinations are widely used by medical schools and testing and certification organizations to evaluate clinical and interpersonal skills not readily measurable with written multiple-choice examinations. Albeit valuable, SP examinations bear limitations, mainly decreased reliability of examinee scores attributable to the limited number of cases seen by the student and variations in SP recording, rating, and portrayal accuracy. Regardless of whether SP exams are being used by medical schools for teaching and learning purposes or by medical testing organizations for licensure or certification, it is critical that scores accurately reflect the appropriate clinical skill levels of the examinees. Threats to reliability may increase when exams are administered on a large scale and it becomes necessary to train multiple SPs to portray the same case across multiple testing sites. Much research has focused on quantifying sources of variability in SP exams because any type of unwanted variation could have a deleterious impact on pass/fail decisions. The conclusions of these studies are not easily discerned.