{'fraction_non_alphanumeric': 0.033426183844011144, 'fraction_numerical': 0.01288300835654596, 'mean_word_length': 4.923711340206186, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Achieving inter‐rater reliability in evaluation of written documentation

these stations. Why the idea was necessary Wide variation in examiners’ scores for these paired stations has been observed. What is unknown is whether the variation in scores reflects examiner leniency or stringency, or issues related to the assessment instrument. Consistency of examiner ratings is necessary to improve reliability of assessment. The researcher chose and modified mini-clinical examination (miniCEX) scoring, firstly by using five of the seven domains to standardise the assessment instrument for summative purposes (i.e. history taking, communication skills, clinical judgement, professionalism and management planning) and, secondly, by training the examiners to provide feedback to students on multiple occasions as part of formative assessment during the 7-week family medicine rotation. What was done A five-step study was carried out as follows. Six OSCE SP stations were video-recorded. Each video was shown to four examiners (identified as examiners 1a, 1b, 2a and 2b) with responsibility for these stations, who were asked to score them independently using the current assessment instrument. The same videos were subsequently scored by the four examiners using the adapted mini-CEX form. The researcher then trained the examiners in how to use the adapted mini-CEX form by anchoring the scores in each domain to expected levels of competence. After training, the same videos were scored again by the examiners using the adapted mini-CEX form. At the conclusion of the study, a focus group discussion with the examiners took place about their experiences using the adapted mini-CEX form for assessment. Evaluation of results and impact With the current assessment instrument, agreement between examiners 1a and 1b was very good (j = 1.00), but agreement between examiners 2a and 2b was poor (j = 0.18). Using the adapted mini-CEX form before training, agreement of scores between paired examiners 1a and 1b, and 2a and 2b, was poor (j = 0.02 and j = 0.18, respectively). After training, the level of agreement remained poor between examiners 1a and 1b (j = 0.00), but improved between examiners 2a and 2b (j = 0.67). The examiners’ focus group discussions elicited the following themes: it is difficult to assess ‘professionalism’ and ‘communication skills’ (examiners 1a and 1b); the mini-CEX form can be adapted as a comprehensive, holistic instrument that allows for self-reflection, and there is a need for more training on the use of the mini-CEX form. Study limitations include the small sample size and the limited number of SP stations; these issues will be addressed in future applications. The adaptation and utilisation of the mini-CEX for both summative and formative purposes may prove to be a useful way to apply mini-CEX scoring and concepts in our setting.