{'fraction_non_alphanumeric': 0.030455562334756967, 'fraction_numerical': 0.0055419971527354076, 'mean_word_length': 4.951285930408472, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Workplace-based assessment of family medicine competencies using “field note tool” – A pilot study

Background: Department of Family Medicine in a medical college in South India introduced “field note” (FN) as a tool for Work-Place Based Assessment in postgraduate training. FN collects “open-ended” feedback from both resident and faculty and helps them to arrive at an action plan. This study describes our experience of implementing FN and perceptions of learners and faculty. Methods: While precepting the residents in Family Medicine service areas, faculty documented their observations of the resident's clinical work using FN and provided an action plan. Faculty and residents described their experience and provided feedback. Focus group discussions were conducted for faculty and residents. Data were coded and grouped into themes. Results: Four residents and seven faculties participated in the study during 12 weeks period using 17 consultations. Clinical expert (13/17) and communicator (6/17) are the most commonly assessed competencies followed by professionalism (2/17) and collaborator (2/17). Faculty and residents agreed that “FN” was a useful tool and it helped the faculty to give feedback and guide the learner. Residents and faculty arrived at an action plan in 70% of the consultations. Three of four residents perceived the change in their behaviour positively after the use of FN. Both resident and faculty found the rating of the learner using Dreyfus scale as a barrier. Conclusion: FN could be one of the important tools in our “Toolbox of Assessment Methods” for family medicine specialty. There is a need for sensitizing the learners to feedback process and training the faculty in assessment and feedback.

Introduction "Assessment drives student learning" is a known axiom in medical education. Even though assessments use more resources, it improves motivation of the learner and helps the faculty and learner to achieve educational outcomes. [1,2] While summative assessment is effective in motivating students to improve their performance toward certification, it cannot be used to provide feedback and guide the learner when a deficiency is identified. [3] Work-Place Based Assessment (WPBA) assesses the "does" level of competence in Miller's pyramid. It is done in an authentic environment and in the context where the resident is going to work in future. It is not merely intended to assign a grade to the learner but to provide feedback and guide the learner. This should be an ongoing activity, based on direct observation by the faculty. It clarifies the goals with the learner and helps faculty to diagnose and correct specific mistakes in learning and behavior. [4] Competency-based education is being introduced in family medicine postgraduate training across the world. [5] Workplace-based assessments are considered more authentic for assessment of competence and performance in real life. There are many tools available for WPBA like Mini-clinical evaluation exercise, video-review, case-based discussion, multi-source feedback, and portfolio. [6,7] Field note (FN) is a new tool that has been described by Ross et al. They used FN as the primary assessment tool in the Competency-Based Achievement System in the University of Alberta, to provide feedback to the residents. [8] FN is a paper or electronic document in which both resident and faculty reflect on the consultations done by the resident [ Table 1]. Open-ended feedback is collected about the competencies demonstrated by the resident in the clinical encounter. FN helps the resident to arrive at an action plan in discussion with the faculty. In FN, the learner needs to reflect first followed by the faculty and this is documented and stored for later review.
This study is aimed at introducing "field note" as a tool for workplace-based assessment in family medicine and collecting the perceptions of the learners and faculty.

Method
After obtaining ethics committee approval and informed consent, four residents and seven faculties were sensitized to the FN tool and the definitions of competencies in family medicine. Over 12-week study period (March to June 2016), the faculty observed and documented the clinical consultations by the resident in the FN and rated the competencies using "Dreyfus five stage model of adult skill acquisition." [9] The seven competencies that were assessed are the clinical expert, communicator, professional, collaborator, system-based practitioner, scholar, and leader. These competencies were defined by the faculty attending the National Faculty Development Workshop organized by National Board of Examinations in Christian Medical College, Vellore in February 2016 and were adopted from ACGME Family Medicine Milestone project [ Table 2]. The ratings used were (ascending from lower to higher level of performance) novice, advanced beginner, competent, proficient, and expert where novice represents the learner without any knowledge about the competency and expert represents the learner who has demonstrated the maximum level of performance in that competency. In our study, we used a paper-based two-page FN document that was simple to use and could be filed for future reference.
Faculty provided feedback about the selected few competencies relevant for each consultation and helped the learner to arrive at an action plan to progress in their learning curve. Faculty and learners were asked to describe their experience of using the FN and rate the usefulness of FN for assessing the learner and for giving appropriate feedback using the Likert scale.
Both faculty and residents suggested changes for improvement and mentioned about action plan they provided or received. Focus group discussions were conducted for faculty and residents to collect their perceptions and challenges in using the FN. The data were coded and grouped into themes. The following questions were asked to focus group: describe your feelings and experiences on using FN, any challenges or barriers you faced while using FN, and any suggestions to make it a better tool.
"Advanced beginner" was the most common rating given by the faculty in 7 of 17 consultations, followed by competent (5/17) and novice (2/17). Residents were never rated as "proficient" or "expert" in any of the consultations. In 3 of 17 encounters, there was no rating mentioned at all. In about 70% of (12 of 17) consultations, residents were given action plan. Three of four residents perceived a positive change in their behavior in achieving the family medicine competencies.
All faculty and learners agreed that the FN is a useful tool for providing feedback and guiding the learner. Overall, FN was found to be a useful tool for assessment of the learner's competence. Almost half of the residents were "hesitant" to agree on the usefulness of the FN [ Figures 1-3]. Tables 3 and 4 describe the selected quotes from the faculty and Table 1: Reflective questions in the Field note tool; (created based on the Competency-Based Achievement System, used by University of Alberta in Edmonton [8] Question asked in Field note Answered by Describe your educational interaction today; write the positive aspects first. What went on well? Resident description of educational interaction: (patient problem, not patient name) Supervisor description of educational interaction: (patient problem, not patient name)

Resident first, followed by faculty
Comments on what could have been done differently. Write the negative experience if any. Resident (what could be done differently) Supervisor (what could be done differently)

Resident first, followed by faculty
Select the rating of the resident's performance today by the faculty in selected competency (Novice, Advanced Beginner, Competent, Proficient and Expert)

Faculty
Plan for resident's next steps (Action plan) Resident first, followed by faculty Volume 7 : Issue 6 : November-December 2018 learners, respectively. The quotes are grouped into strengths and challenges/barriers.

Suggestions from the faculty and resident
The following are few suggestions from the faculty; • "There is a need for entry level assessment of the competencies in the PG student" • "Systems are needed for behavioral change in resident when a problem is identified" • "Action plan needs to be followed up" • "If we could make the documentation online and review periodically, it would be better." The following are the suggestions from the resident: • "Give time to reflect with the resident"   The ability of the physician to provide comprehensive, continuous, clinical care for a defined population for a wide variety of clinical problems across ages, genders, both acute and chronic including early undifferentiated illnesses, in a person-centered, personalized, family focused and community-based process. Communicator The ability that helps the physician to perform doctor-patient consultations to foster therapeutic relationship and trust. It helps to communicate with family members, health professionals, and the community.

Collaborator/ Coordinator
The ability of the family physicians to work with patients, families, healthcare teams, other health professionals, government agencies, and communities to achieve optimal patient care and education in a multi-professional environment.

Professional
The ability of the physician to display the behaviours consistent with the expectations of the program, the profession and with society's expectations of the profession. The ability of the physician to maintain the well-being of the person of the physician. Scholar The ability of a physician to demonstrate a life-long commitment to excellence in practice through continuous self-directed reflective learning, the teaching of others, the evaluation of evidence and other resources, and contributions to scholarship in the creation, dissemination, application, and translation of medical knowledge.

Systems based practitioner
An ability of the physician to demonstrate an awareness of the larger context of the person, the deeper systems within the person and systems of health care and appropriately respond to them. Ability to call on system resources to provide care for the person/family that is of optimal value. Leader The ability of the physician to develop a personal vision and a collective vision in collaboration with other health care leaders, of a high-quality health care system and take responsibility for effecting change to move the system toward the achievement of that vision in whatever context they are working.

*Defined by the faculty who attended first National Faculty Development Workshop organized by National
Board of Examinations in Christian Medical College, Vellore in February 2016. Adopted from ACGME Family Medicine Milestone Project. [12,13] Volume 7 : Issue 6 : November-December 2018 • "No need of rating of the student" • "Field note should be done for the same condition/problem over few weeks if possible to assess whether the learner has improved based on the feedback by the faculty" • "A learner needs to be followed up. If a learner is Novice in one topic, he could be followed up with multiple field notes till he is competent on that topic" • "Remove the Novice and advanced beginner. Rather use stage 1, 2, 3, and 4" • "Is it possible to counter check the action plan? Whether the action plan is achieved?" • "One faculty could see all the field notes according to the subject and according to the competencies and provide overall feedback to the resident."

Discussion
Any assessment method should be aligned with the content of the training program and all the desired competencies of the specialty. Assessment is also expected to have an educational impact. [10] In postgraduate education in India, more emphasis is given for summative assessment using theory examination and practical clinical case discussions. In these high-stakes summative examinations, only a few competencies such as clinical expertise and communication are assessed. But there is less opportunity to assess important competencies like the collaborator, leader, scholar, and system-based practitioner which are needed for the practice of medicine. "Students seem to act. I am not sure whether we are capturing the reality". "User-friendly across multiple contexts." "Field notes documents objective evidence of learner's performance. Action plan is the best part of the field note" "We are more focused on the deficiencies of the learner, that's good. But we need to look at both positive and negative sides of the learner…" "Excellent tool for formative assessment and regular review of the milestones of the learner". "FN made the learner address all competencies in each patient encounter".
"Rating the resident's performance on the various family medicine roles into one category was difficult. Needs more time…" "Brings closeness with residents… It helps us to 'understand' and 'know' the learner. I could give feedback to the learner when I found a problem in the resident's consultation". "The faculty is forced (obliged) to observe, listen to the resident and document the observations" It is difficult to rate the learner. "I was afraid how the learner would take it if I write Novice?" "Rating of the learner in the first year can be removed. It is important to rate the learner in the second year".
"Immediate documentation of learner's performance is vital for reflection between learner and teacher." "This tool is relatively easy to use and teacher friendly" "I do not like this process but it helped the resident. It helped me to guide them appropriately."

Strengths of Field note perceived by residents
Challenges & Barriers perceived by the residents "By writing down the positive points, we are able to remember what we did. The positive thing, (feedback) always stays in our mind". "By reflecting upon the patient, I could understand the patient better". "In a busy day, it is usually otherwise a one-way communication happens (with the faculty). But the reflection and the discussion of the reflection with the field note is more helpful (to make it a two-way communication)." "We are not perfect. We need improvement, day by day. We have to learn from our mistakes. Sometimes I don't want to list the weaknesses." "Multiple observers and multiple conditions are there…. Both are the problems…" "It helps us to know where we are lacking. When we see a similar patient, we could apply this to the next patient as well." "Our weaknesses are our strengths. By realizing a mistake, that makes us a better doctor." "Precepting with faculty itself is a good method to improve learning. Field note gave a backbone or better understanding to this learning method." Faculty never used it in some rotations. The consultant said 'it is more difficult to write this down than teaching you'. "It needs extra effort from both resident and faculty". "It is a time-consuming process" "The feedback depends upon who is doing and how it goes" "One and one basis discussion, a time together with the consultant is more real and make it more relevant." "It helps me in a way that preceptor tells me 'you are good at this point'" "There is no reflection or action plan on a particular patient without field note".
"Are we always Novices…? We will be always novices, never reach the expert level…even after the end of the course…?" "It feels that we will remain forever in 'Novice' phase considering the unlimited spectrum of family medicine. What can be done for this?" "Every faculty has different thoughts and different ways of managing the same condition…" "Was able to know (realize) what went wrong in our management". "Was able to know what was missing in my consultation." "Some points get clarified due to field note." "We need to see how much the action plan is helping the resident". "One week we are given feedback on a particular condition that is never being assessed again. Next week we are given a different condition. The progress depending upon the feedback is not assessed." Volume 7 : Issue 6 : November-December 2018 This pilot study found that "FN" can be used to assess most of the competencies in family medicine. Leader as a competency was never assessed in our study. That could be explained by the short duration of the study. Leader is a complex competency and it needs multiple observations to make a reliable judgement. Formative assessment using FN is a way forward in implementing competency-based medical education.
FN was accepted as a tool for workplace-based assessment both by learners and faculty, and it is cost-effective. But the learners had difficulty in accepting the negative feedback and the rating as novice or advanced beginner. They had received more negative feedback than positive. This could be one of the reasons for their hesitation in agreeing on the usefulness of FN [ Figures 1-3]. The residents have felt the differences in the ways they are assessed by different faculties [ Table 4]. There is a need for developing definitions of milestones of competencies and faculty development in assessment and feedback to reduce the observer bias.
Educational impact of the FN was perceived by the residents during the focus group discussion [Tables 3 and 4]. The utility of any assessment tool depends on its validity, reliability, educational impact, acceptability, and cost. We did not measure the "validity" of the tool in this study. Validity refers to whether the instrument measures what it is expected to measure. According to Dr. Van der Vleuten, validity depends on the authenticity of measurement and integration of competencies during measurement. [11] FN was used to assess all competencies, in any clinical context: being outpatient, in-patient care, emergency care, and home care. FN can be used to assess the consultation skills, procedural skills, ward rounds, counselling skills, and academic presentations. The fact that FN is used in the authentic workplace and the whole task is assessed without trivialization of the competencies, FN can fulfil the criteria for validity.
"Reliability" is the reproducibility of the scores achieved in assessment. [11] One can argue that assessment by FN has the subjective bias as it is open-ended without objective scores and this may negatively affect reliability. Van der Vleuten says the reliability of an instrument depends on careful sampling across the content and not based on objectivity or standardization. To reduce the observer bias and improve the reliability, we decided to collect the FN of each resident given by different faculties at different context and file it for review by the course coordinator once in 4 months to assess the resident's progress. A feedback will be given to the resident about the performance in the previous 4 months and also suggestions will be given to achieve the desired milestones in the next 4 months. [8] Limitations This is a pilot study done over a short period in one institution in India. Findings of this study need to be validated in a larger multicentric study done in multiple settings with many institutions training family medicine residents in India.

Conclusion
FN could be one of the important tools in our "Toolbox of Assessment Methods" for family medicine specialty when we introduce competency-based postgraduate medical education in India. There are few challenges observed in using the FN. Rating the learner using Dreyfus levels was the most important barrier that needs to be addressed during implementation. There is a need for sensitizing the learners to feedback process and training the faculty in assessment and feedback.