{'fraction_non_alphanumeric': 0.023614234687127925, 'fraction_numerical': 0.007937557878026195, 'mean_word_length': 5.361666316010941, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Feedback on feedback: a two-way street between residents and preceptors

Background Workplace-based assessment (WBA), foundational to competency-based medical education, relies on preceptors providing feedback to residents. Preceptors however get little timely, formative, specific, actionable feedback on the effectiveness of that feedback. Our study aimed to identify useful qualities of feedback for family medicine residents and to inform improving feedback-giving skills for preceptors in PGME training program. Methods This study employed a two-phase exploratory design. Phase 1 collected qualitative data from preceptor feedback given to residents through Field Notes (FNs) and quantitative data from residents who provided feedback to preceptor about the quality of the feedback given. Phase 2 employed focus groups to explore ways in which residents are willing to provide preceptors with constructive feedback about the quality of the feedback they receive. Descriptive statistics and a thematic approach were used for data analysis. Findings We collected 22 FNs identified by residents as being impactful to their learning; analysis of these FNs resulted in five themes. Functionality was then added to the electronic FNs allowing residents to indicate impactful feedback with a “Thumbs Up” icon. Over one year, 895 out of 8,496 FNs (11%) had a “Thumbs up” added, divided into reasons of: confirmation of learning (28.6%), practice improvement (21.2%), new learning (18.8%), motivation (17.7%), and evoking reflection (13.7%). Two focus groups (12 residents, convenience sampling) explored residents’ perception of constructive feedback and willingness to also provide constructive feedback to preceptors. Conclusion Adding constructive feedback to existing positive feedback choices will provide preceptors with holistic information about the impact of their feedback on learners, which, in turn, should allow them to provide more effective feedback to learners. However, power differential, relationship impact, and institutional support were concerns for residents that would need to be addressed for this to be optimally operationalized.


Introduction
There is a rapidly expanding international uptake of competency-based medical education (CBME). [1][2][3][4][5] One key principle within CBME is the importance of providing residents with quality feedback on their clinical performance, with the underlying premise that this is critical for residents' development. [6][7][8][9][10][11][12][13][14][15] Quality feedback has been defined as criterion-referenced, timely, behaviourally-specific, actionable, context-specific and tailored to individual needs. [16][17][18][19][20] It has been shown to improve performance. 21 With the onus on preceptors to provide feedback that impacts learning, it is critical that preceptors continually hone this important skill. Preceptors stand to benefit from feedback on their feedback and residents are well placed to provide it.
There are many resources available for preceptors to learn how to provide effective feedback. Articles, learning modules, and faculty development workshops all provide helpful foundational information about feedback provision. 22,23 There are concerns, however, that such generic offerings are not effective in changing behaviour. [11][12][13] Providing personalized specific feedback about assessment performance does appear to improve assessment practices. 13 A multipronged approach to faculty development aimed at providing generic knowledge about feedback-giving, layered with personalized feedback about this skill, is likely a reasonable strategy.
Such personalized feedback to preceptors can happen in a variety of ways. A common approach is to provide periodic anonymized resident-generated summative assessments of performance. For anonymity reasons these are often held until enough assessments can be collated to mask the assessors, 24 often delaying receipt of that feedback. Anonymity and delay can render feedback less effective. For anonymity to be maintained, comments necessarily lack specificity. Just as feedback to residents is best done in a timely and specific way to be optimally effective, feedback to preceptors should be the same. The concept of dialogue in feedback has been highlighted in the literature. Feedback is ideally not a unidirectional linear process but an interactive conversation promoting self-reflection and opportunities for clarification and exchange between participants. 25,26 Herein lies a dilemma: dialogue, either face-to-face or electronic, compromises anonymity and anonymity compromises meaningful dialogue.
Are residents ready to engage in providing this nonanonymous, upward feedback to their preceptors?
Preceptors are responsible for the assessment of residents which will impact residents' advancement and overall success in their training program. This inherent power differential can inhibit resident's willingness to provide non-anonymous feedback. [27][28][29] Despite this power differential, fruitful efforts have been made to engage residents in non-anonymous feedback with their preceptors. Fluit and colleagues reported on a collated survey derived from multiple residents' anonymous input that resident representatives then reviewed with the preceptor and moderator. 30 Dudek and colleagues reported on a carefully designed system in which residents provided both formative and summative feedback to their clinical supervisors. 27 This was carried out in a small program characterized by longer term relationships between faculty and residents. Feedbackgiving skills were taught before residents engaged in providing feedback to their preceptors. This study reported residents' willingness to engage in this non-anonymous upward feedback however it was unclear if the study was reporting on the formative or summative feedback. Prolonged contact between residents and faculty, may increase the possibility of building more trusting relationships, which may or may not increase the likelihood of residents feeling comfortable giving more difficult constructive feedback to their preceptors. Both these studies involved residents giving summative feedback to preceptors with whom they had prolonged contact.
This study aims to add to the existing literature looking at residents providing non-anonymous, timely, formative feedback to their preceptors, granting preceptors an immediate way to gauge their performance. There are gaps in the literature. We have not found any articles focussed on providing preceptors with behaviourally-specific formative feedback addressing feedback-giving skills. We are unaware of studies where residents in large programs, with less preceptor continuity, provide non-anonymous feedback to their preceptors. Research is required in large programs to determine how residents can be supported in giving formative feedback to preceptors. Our study looks at providing feedback to preceptors in three new ways to address these gaps; 1) residents giving preceptors timely, behaviourally specific formative feedback on feedbackgiving skills, 2) within a large program, and 3) with less resident-preceptor continuity. In addition, we also describe an electronic means for providing such timely feedback, critical in a large program for feasibility and practicality reasons. Constructive feedback is used throughout this paper to replace the term negative feedback as feedback focusing on changing behaviour (formerly called negative), if given in an effective way, is often very constructive.
The purpose of this study was to determine, from the resident's perspective, (a) the most useful qualities of preceptor's formative narrative feedback to promote resident's learning, and (b) to determine willingness and ways to facilitate residents giving both positive and constructive formative behaviourally specific feedback to their preceptors about the feedback they receive. This is done in the context of a large postgraduate training program using an electronic assessment system.

Methods
Our research is an exploratory study which informs improving feedback-giving skills for preceptors in any PGME training program. It delves into ways to facilitate residents providing both positive and constructive formative feedback to their preceptors about their preceptor's feedback-giving skills using an electronic assessment system, in a large family medicine program (140 residents), with hundreds of preceptors, and often limited contact time between residents and preceptors. We adopted a two-phase exploratory design for this study. In Phase 1, we conducted a mixed methods exploratory sequential design where we began with collecting qualitative data from preceptor feedback given to residents through narrative comments on Field Notes (FNs; see context section for a description of FNs). We then collected quantitative data that identified the number of FNs where residents provided feedback to their preceptor about the quality of the feedback given. Informed by Phase 1, Phase 2 employed a qualitative exploratory approach through focus groups to further identify ways in which residents are willing to provide preceptors with constructive feedback about the quality of the formative assessments they receive. 31

Context
This department embarked on a CBME approach in 2010. As part of facilitating the provision of more daily formative feedback to residents, the department developed electronic FNs. FNs are brief documents capturing the verbal feedback given by preceptors to residents. 32,33 In our context, FNs capture assessment of small aspects of performance, tied to the critical competencies needed for a developing family physician. 34 Within the FN, there is a qualitative narrative component wherein preceptors can reinforce what a resident has done well, and identify steps for further improvement. A dialogue between preceptor and resident about what is written in the FN is possible using the 'comments box' which is part of the FN interface. A complete description of our competency-based assessment practices is available. 35 Preceptors in the department have had access to numerous faculty development sessions and electronic modules focussed on feedback-giving. Academic Advisors in our system refers to one of the resident's primary preceptors, who has the added role of meeting regularly with the resident throughout their training and is an integral part of making summative competency decisions.

Data collection and analysis
Phase 1 The first phase of this study identified helpful preceptor feedback-giving skills and generated an electronic means for residents to provide positive feedback to preceptors. In 2015, we held five FN competitions (with a $20 gift card as a prize), each involving 20 residents who were involved in their family medicine rotation. Over a two-week period, residents submitted any of their daily FNs that they found particularly helpful and described the reasons why. These friendly competitions were part of our change management strategy to enhance the amount and quality of feedback provision that our faculty were providing, a necessary part of our CBME implementation. Competition publicity served to highlight the value for residents of receiving quality feedback, and to reinforce qualities of good feedback, and so became informal opportunistic continuing medical education (CME)-type events. Through those competitions 22 FNs were submitted by residents as being most helpful for learning. Using an inductive, emergent design all submitted FNs underwent a thematic analysis. 36 We identified five themes of helpful feedback (see Table 1 and Appendix B). Subsequent FN competitions (held 2016-2019) and analysis of the submitted FNs confirmed these five themes; no additional themes emerged.
In 2016, we added a 'Thumbs-Up' (TU) icon to the electronic assessment system. Residents, on reflecting about their electronically submitted FN, could opt to add a TU to their FN if they felt it was beneficial to their learning. Once they selected the TU icon, they could then choose from a drop-down menu one of the five behaviourallyspecific themes to identify why this FN was particularly helpful. This was a first step in getting residents comfortable in providing feedback in a specific timely way to their preceptors. They could also add free-text narrative comments if they desired (see Figure 1). Figure 2 depicts feedback that a resident viewed as helpful for promoting reflection ( Figure 2).

Figure 2. An example of a FN with a TU that indicates the theme 'promoting reflection'
Preceptors receive information about their FNs in a timely way. We developed a reporting function within our electronic platform that allows each preceptor, the Program Director and Department Head access to a realtime collated report for that preceptor on the number of FNs they have done across competency domains, as well as the number of FNs given a TU with its associated theme. This positive feedback is done non-anonymously and so is available in real-time to preceptors.
From June 2016 to June 2017, a total of 8,496 FNs on our 140 residents were submitted. Descriptive statistical analysis was performed to identify the frequency with which residents gave their FNs a TU and which of the five themes they chose.

Phase 2:
In the second phase of the study, we explored residents' perceptions of providing constructive feedback for improving preceptors' feedback-giving skills. Using convenience sampling, we recruited 12 end-of-year PGY-1 residents (4 male; 8 female) to participate in two semistructured focus groups (n = 7, n = 5). Two researchers who are experts in qualitative research methods and not associated with the residents or their success in their training program conducted the focus groups. One researcher facilitated the discussion and the other wrote notes and summarized discussion points. Each focus group took place at a location convenient for participants, was one hour in length and audio-recorded.
First, we explored residents' concerns about providing preceptors with constructive feedback. Second, by providing residents with sample constructive feedback statements, we explored their comfort with using these statements, revising those based on their input. Appendix A includes the questions asked in the focus group and the sample phrases of constructive feedback for preceptors. Finally, residents provided us with recommendations that would support residents providing preceptors with constructive feedback.
Both focus groups were completed using the same guide and the data from both were analyzed at the same time.
No differences in the findings were identified between the two groups. The focus group interviews were transcribed verbatim. Data were analyzed through an inductive, thematic analysis approach.
Two researchers independently open-coded both transcripts in NVivo 11,37 and then met to compare codes to ensure intercoder reliability. 36 They discussed any discrepancies in coding and reached consensus to ensure shared meaning. Once a codebook was developed, the research team met multiple times to further discuss the codes and group them into similar categories and emerging themes. This process allowed our research team to mitigate personal biases and tensions, thereby addressing reflexivity and ensuring rigor in our process. [38][39][40] We verified all data to ensure consistency of our findings through member checking. We summarized the emergent themes at the end of the focus group for all participants to confirm.

Findings
We have organized the findings into two distinct parts. Part 1 describes the emergence of five themes from the analysis of the 22 FNs identified in the FN competitions. Part 2 identifies two overarching themes: (1) residents' suggested wording and phrases for constructive feedback statements, and (2) Table 1 and Appendix A). By June 2017, 8,496 FNs were collected, of which 895 (11%) received a TU as feedback to preceptors. Distribution of these themes is illustrated in Table 1 (right column).

b) Residents' concerns about and recommendations for providing preceptors with constructive feedback
Focus group participants agreed that constructive feedback to preceptors would help preceptors make FNs more valuable for their learning. They, however, expressed concerns in providing this feedback. Those concerns were grouped into three themes: power differential between preceptor and learner, the impact on their relationship with their preceptor/Academic Advisor, and a lack of institutional expectations for preceptors to receive constructive feedback from residents. They also suggested recommendations to mitigate concerns (see Table 3).

I. Preceptor-resident power differential:
Participants acknowledged the power differential between residents and preceptors in residency training. In a learner position they felt uncomfortable telling their preceptors that their feedback was not very useful, concerned that constructive feedback to preceptors may impact their progression in the program adversely. Also, they acknowledged that having a   Participants suggested that anonymity would be helpful to mitigate this concern and would encourage residents to provide constructive feedback to preceptors:

II. Preceptor/Academic Advisor -resident relationship:
Participants also expressed concerns that provision of constructive feedback may negatively affect their relationship with preceptors, particularly when the preceptor was also their Academic Advisor.

Discussion
This study has shown that residents in a large program, with varying amounts of continuity with their supervisors, are willing to provide formative positive behaviourallyspecific feedback to their preceptors about the quality of the narrative feedback their preceptor provided. This was done using a non-anonymous electronic system. Previous studies have reported on non-anonymous summative resident-derived feedback. 13,30 The study of timely nonanonymous formative feedback within a large program is a new addition to the literature. Given the evidence that providing personalized specific feedback does appear to improve assessment practices, 13 this has potential to be a powerful form of faculty development. The timely formative nature of the TU feedback quickly identifies for preceptors the feedback residents find helpful and should reinforce these helpful feedback-giving behaviours.
Concerns have been raised that preceptors may not give constructive feedback for fear of a negative reaction or adverse preceptor assessment by the resident 24,41 and that residents may not attend to constructive feedback. Residents most often indicated liking feedback relating to performance assessed as being done well. Our study shows that residents were also appreciative (as witnessed by a TU designation) of constructive feedback that "teaches them something new" and "changes their performance". This provides some reassurance that residents are not just choosing the TU as a reward for positive things said about their performance. It also reinforces that "negative" feedback is indeed constructive feedback and residents do appreciate it. The fact that only certain FNs are awarded a TU and that there is a good spread across the five themes shows that residents are discerning in the feedback they provide to preceptors about what they perceive to be helpful.
When asked to look at the flip side of feedback, that of providing constructive feedback to their preceptors aimed at improving their feedback-giving skills, residents endorsed, through their editing, wording that was behaviourally-specific and actionable and, importantly, that they would be willing to give to their preceptors. However, despite the professional nature of the wording agreed upon, residents still expressed concern about the mechanism for providing such constructive feedback. This discomfort primarily ties to their relationship with their preceptors and Academic Advisors (the power differential in the relationship, not wanting to impact their relationship with their Academic Advisor negatively). Residents felt that the effect of a power differential may be softened with a more established longer-term relationship between resident and preceptor. Practical suggestions were provided by residents to mitigate these concerns.
Residents want a way to remain anonymous if providing constructive feedback. This option for anonymity reinforces what other programs 12,13 report, with their findings also identifying that the desire for anonymity is often hypothetical, as some residents are comfortable with non-anonymous, timely constructive feedback. Anonymity prevents the immediate timeliness of constructive feedback but appears to be a necessary compromise for some residents, to optimize their constructive feedback to preceptors. When feedback is given anonymously the risk of unhelpful or hurtful comments may arise. 30 Having agreed upon statements to provide constructive feedback may reduce this risk. Perhaps the most important drawback of maintaining anonymity and delaying feedback is the prevention of a meaningful two-way dialogue between resident and preceptor which we know is an important aspect of assimilating and applying feedback. 25 This new process offering the option of adding constructive feedback to FNs is a first step towards the idea of open dialogue between residents and preceptors to improve preceptor feedback skills. The comments box is a way to further promote that dialogue.
Residents also identified the importance of institutional support to foster a culture where preceptors are open to constructive feedback from residents. They also believe that there needs to be transparent program expectations for preceptors to provide regular feedback. Without a culture that supports regular quality feedback, residents feel guilty about asking for preceptor feedback fearing that it will increase preceptor's workload. Although residents are vital stakeholders in changing assessment culture, the prime responsibility lies with the institution to set assessment expectations, especially given the reality of the power differential between the residents and program. 42 Both this study and others identify the importance of an explicit institutional expectation for feedback for both learners and preceptors. 27,43 The electronic platform used within our department made it logistically easy to deliver to preceptors two important elements of effective feedback on their feedback: timely and behaviourally-specific information. Our platform was initially created out of necessity to collect and collate the massive amounts of assessment data for our 140 residents. Now, through the TU option and report-generation function, the electronic platform can also be leveraged to provide meaningful feedback to preceptors about their feedback-giving skills. The electronic platform can be easily programmed such that feedback by residents can be either anonymous or non-anonymous.

Conclusion
This study shows that resident-generated, nonanonymous, timely, behaviourally specific, formative, positive feedback for preceptors is possible. This has the potential to improve preceptors' feedback giving skills as well as promote residents' feedback-giving behaviour and reinforce helpful feedback giving practices. These habits can highlight the importance of feedback within a teaching environment changing the assessment culture in the program. Ideally this would also be true for constructive feedback, however, at this time residents want an option for anonymity for this. In addition to adding to the literature a way to provide formative feedback to preceptors, it also demonstrates that residents within a large program are willing to provide this to preceptors that they may not have a long-term relationship with.
Practical take away points 1. Residents are willing to identify, in a non-anonymous timely way, preceptor's feedback that they find helpful for their learning.
2. Residents are discriminating in which feedback they identify as being helpful.
3. Residents want to also be able to give constructive feedback to preceptors to help improve feedback and have identified wording that they would be comfortable using with preceptors to do that.
4. An electronic assessment system can be leveraged to provide holistic (both positive and constructive) feedback to preceptors in a way that addresses residents' concerns with providing constructive feedback to preceptors.

Future work
Of all FNs, 11% were awarded a TU designation. We do not know what drives residents to award a TU although we do have evidence it is not entirely done as a "reward." Monitoring the percentage over time and exploring this further will tease out the reasons why residents use this function. Based on this study we have added a second drop-down menu for constructive feedback using the statements endorsed by the resident focus group. We chose a "light-bulb ah-ha" icon to go with this constructive feedback drop-down menu (the idea of a Thumbs Down icon being too negative). This is a new feature of our FNs found alongside the existing TU feature identifying positive feedback skills, with a new option for the feedback (positive or constructive) to be anonymous. Uptake and use of these constructive themes and anonymous option will be evaluated. Monitoring the use of the anonymous feature over time will allow us to see if this need for anonymity declines over time as feedback becomes a more entrenched practice. Knowing that there are a number of factors impacting recipients' receptivity to feedback and that not all feedback is acted on, 43 a critical next piece will be understanding and optimizing the receptivity of, and use by, preceptors of both the positive and constructive feedback they are given by residents to see if it improves their feedback giving skills. It will also be interesting to see if this bidirectional feedback pathway positively impacts the culture of assessment within our department, increasing both our resident's and preceptor's feedback seeking and accepting behaviours.

Limitations of this study
This study is confined to a small sample size in a single program in one discipline. It does not explore the perceived value by residents of the quality of verbal feedback they are given, however the value of written feedback, which was explored, is important in reinforcing verbal feedback and documenting evidence of residents' performance. It does not include the perspective of preceptors, which will be an important aspect to understand if feedback is to be used to change behaviour. Highlighted application of existing knowledge in a clinical situations (very often this is a nuance that 'comes with experience' -preceptor is sharing his/her experience) "My takeaway from this note was that it really is OK not to cram every issue that comes up into one appointment."

Conflicts of Interest: None
"Give a tip for improving my management of this case so that next time I can go beyond the standard of care." 2. Imparted new knowledge a) Observed preceptors to gain new knowledge "Highlighted a situation which is not one I have encountered that frequently, and the issues that come along with this." "But what I find very useful [is] listening to his conversation with the patient and gaining insight into how to systematically approach the topic that was individualized to her goals for pain management." b) Direct teaching by preceptors addressing gaps in knowledge "Addressed areas of the interaction that I was specifically uncertain about (in this case, the necessary investigations) and helped outline an approach in how to address complex, unusual symptoms such as the one that the patient presented with." 3. Provided motivation to learn a) Offered support and encouragement "Provides an appropriate amount of support through encouragement." "[It] was also very encouraging and supportive." b) Generated specific learning objectives "I also had very specific learning objectives for reading." 4. Confirmed skills a) Confirmed and/or acknowledged learning and/or competence in specific skills "I found this field note particularly helpful as it reflected back that I did indeed handle a difficult discussion well." "Acknowledged my attention to non-pharmacologic management…solidified how important it is to provide comprehensive care for my patients." "Provided a specific example of a skill that we were recently taught and how I was using it well. In the future I will continue to help patients find small attainable goals that they can work towards." 5. Promoted reflection a) Reminded to consider unspoken agenda and biases "Helped me to remember that not only is it a good thing to advocate for your patients, but when needed it is ok to advocate forcefully." "Helpful reminder to review your diagnosis when the initial treatment does not work." b) Reflected on improving difficult-to master skills "Reflect on the communication aspect of this particularly difficult encounter, it made me think about the way in which I'd communicated, and how, in future encounters, I can communicate in different ways to ensure patient understanding." c) Timeliness of feedback facilitated reflection as the clinical encounter was easily recalled "[The preceptor] submitted this field note right after the occurrence so upon looking at my portfolio a few days later I was able reflect on this experience again soon after the event and look back on it while the experience was fresh in my mind." "Constructive feedback was even more helpful in that Dr. X completed the Field Note immediately after the encounter so that it was fresh in my mind when I read the Field Note." d) Reminded of the more in-depth teaching provided by the preceptor around the clinical case "The FN serves as a reminder of a more in-depth discussion." "[The FN feedback] brought to light something that I know I should be doing but did not realize I wasn't doing. Exploring protective factors not just for suicidal patients but also for patients with violent thoughts is something I will be more conscious of doing next time."