{'fraction_non_alphanumeric': 0.025703383789676716, 'fraction_numerical': 0.00691852907878737, 'mean_word_length': 5.241821512693012, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Design Principles and Clinician Preferences for Pharmacogenomic Clinical Decision Support Alerts

OBJECTIVE: To better understand clinician needs and preferences for the display of pharmacogenomic (PGx) information in clinical decision support (CDS) tools. MATERIALS AND METHODS: We developed a semi-structured interview to collect feedback and preferences in six key areas of PGx CDS design, from clinicians who had prior experience with live PGx CDS tools. Eight clinicians from Northwestern Medicine's (NM) General Internal Medicine clinic participated in the study. RESULTS: Clinicians expressed preference for interruptive pop-up alerts during order entry, brief descriptions of relevant drug-gene interactions, and a clear and specific recommended alternative course of action when a medication is contraindicated. They did not wish to see detailed genetic data, preferring phenotypic information predicted from the genotype. Nor did they wish to be interrupted when genetic test results do not indicate a change in treatment plan. Clinicians reported little familiarity with Clinical Pharmacogenetic Implementation Consortium prescribing recommendations but reported trusting recommendations of their professional societies and resources like UpToDate. Analysis of unstructured comments concurred with structured results, indicating a general uncertainty among participants around how to interpret and apply PGx information in practice. DISCUSSION: Results point to several underlying principles that can inform future PGx CDS alert designs: Be Specific and Actionable; Be Brief; Display Phenotypes not Genotypes; Rely on Sources Clinicians Already Trust; and, Be Adaptable to Learning Effects. CONCLUSION: This study is part of a broader socio-technical design approach to PGx CDS design underway at NM and provides a baseline for future PGx CDS development. Designs based on these results have the potential to improve clinician education and adherence levels, and to improve patient outcomes.

CONCLUSION: This study is part of a broader socio-technical design approach to PGx CDS design underway at NM and provides a baseline for future PGx CDS development. Designs based on these results have the potential to improve clinician education and adherence levels, and to improve patient outcomes.

Background and Significance
Genomic medicine continues to show promise for improving patient outcomes in myriad ways, with new discoveries of links between genetics and health regularly appearing in the scientific literature. Certain specific genetic tests have garnered broad public awareness (e.g. the BRCA genes [1]) and direct-to-consumer genetic testing is reaching the mainstream. [2] Notwithstanding, clinician comfort levels with genetic data and the use of genetic testing remains relatively low. [3] [4] Integration of genetic-and genomic-based factors into standard clinical practices must be done in a way that benefits both the clinician and the patient, allowing the best course of action to be followed with minimal burden and confusion.
Pharmacogenomics (PGx) is among the most promising areas of genomic medicine. It is regularly cited as a means to reduce adverse drug events, avoid ineffective drugs, and optimize drug and dose selection more reliably and with less trial-and-error. [5,6] Clinical recommendations for the use of PGx in drug prescribing exist but are not yet widely implemented. [7] Clinical decision support (CDS) that delivers PGx information and prescribing recommendations at the point of care is one promising way of disseminating this knowledge and improving patient care. [8][9][10][11] Some PGx CDS implementations have been described in the literature. Most sites in the Electronic Medical Records and Genomics (eMERGE) Network have implemented PGx CDS systems. [12,13] Several other organizations have also succeeded in implementing firstgeneration PGx CDS systems, both in the United States [14][15][16] and other countries. [17] Institutions implementing PGx CDS tools have taken widely varying approachesthe University of Chicago's Genomic Prescribing System uses a "traffic light" model in a standalone system, [18,19] while St. Jude's PG4KDS system uses traditional pop-up alerts integrated into the EHR. [20] Even within the eMERGE network, where institutions collaborate on genomic medicine projects, significant variation in PGx CDS alert design made it difficult to aggregate results for cross-site analyses. [13] There is little information regarding which of these designs may be the most effective at conveying PGx knowledge to providers and improving care, and similarly, little consensus around optimal PGx CDS design.
Some studies have begun to examine PGx CDS alerts in an effort to optimize their designs, but this work has been too preliminary to drive a consensus on best practices for incorporating PGx knowledge into clinician workflows, or for displaying it in a way that is easy to understand and use by clinicians. [13,[20][21][22] Due to the complexities of genomic medicine, and low levels of clinician education and comfort in the field, simpler approaches to PGx CDS development-such as design-by-committee processes-may not be effective. [23] Instead, more formal socio-technical design approaches that iterate on designs, deeply incorporate user workflow needs, and respond to user feedback at multiple phases of development may be more effective in creating genuinely useful PGx CDS tools.

Objective
This study aimed to apply socio-technical design methods to determine actual clinician preferences for PGx information at Northwestern Medicine, as a first step towards designing better-optimized PGx CDS alerts. This was accomplished through a series of semi-structured interviews conducted with clinicians who have real-world experience with a live PGx CDS system. The results of this study can be used to inform future improvements on PGx CDS designs. This study emphasized matching the user's workflow and information needs through a focus on appropriate timing and location for PGx data display, type of information conveyed, level of detail, specificity of recommendations, and trusted sources of PGx recommendations.

Materials and Methods
We developed a semi-structured interview that focused on areas of interest we identified after previous phases of NU's eMERGE-PGx project. [11] We interviewed eight clinicians from Northwestern Medicine's General Internal Medicine (GIM) outpatient clinic. Each of the participants had previous experience with our existing PGx CDS tools. Existing PGx CDS tools were built using the Best Practice Advisories functionality of the clinic's EHR (Epic Systems Corp., Verona, WI), based on genomic indicators received from an ancillary genomic system. [24] As described in detail below, interviews were transcribed and systematically analyzed to identify themes and attitudes that could inform the design of improved alerts. The Northwestern IRB approved all aspects of this study.
Building upon our previous research, [23] we identified six areas of interest for which we believed a better understanding could lead to improved alert designs and more positive clinician interactions. We developed an interview guide with a series of structured and open-ended questions to probe these areas. The areas of interest and methods for assessing them are listed in Table 1. All interviews began with a series of questions assessing basic demographics and PGx comfort level. Participants were provided the opportunity to elaborate upon their answers for all structured questions, if desired. The complete interview guide is available in Supplement 1. were analyzed by two independent coders to identify themes and attitudes that were not captured by the structured responses. TMH performed the first coding pass and created a codebook with 97 individual codes roughly organized in parallel with the interview guide. This codebook, and un-coded copies of the transcripts, was provided to TAN, who performed a second coding pass without knowledge of TMH's prior coding decisions. TMH and TAN met in a series of sessions to compare coding choices and reconcile any discrepancies via one-on-one discussion. Through these meetings, 100% consensus was reached without need for third-party intervention.
To identify the themes and axes that emerged from the coding process, we performed a two-pass analysis of the codebook. TMH created physical cards individually printed with each code and description from the codebook and performed a card sort by grouping codes into related groups. TAN and JBS collaboratively performed a second card sort, without knowledge of TMH's decisions. The codes and categories for each pass were entered into a spreadsheet.
All three authors jointly reviewed the results of the two passes and reconciled differences to finalize the overall themes used for the unstructured data analysis.

Basic Demographics
The eight participants averaged 21.5 years of clinical experience. Seven (87.5%) were board certified in only internal medicine. One participant (12.5%) was certified in both internal medicine and cardiology. On average, participants were scheduled to see patients 17 hours per week.

Understanding Workflow
When presented with our model diagram representing a typical prescribing workflow, seven of the eight participants (87.5%) reported that they review medications earlier in their workflow than in the diagram. There were no other major discrepancies between our diagram and actual clinician workflow. The updated version, representing typical clinician workflow as reported by the participants, is presented in Figure 1.

Feedback on Existing Tools
When asked about their thoughts on the existing PGx CDS tools, participants indicated a general ambivalence towards the tools, with clear room for improvement and no clear hostility towards the use of PGx CDS. All eight participants (100%) reported that they recalled seeing PGx CDS alerts at some point in their practice. When asked whether they felt the alerts were useful for them in making a prescribing decision, responses were overall neutral, with four (50%) finding them helpful and four (50%) not finding them helpful. Six (75%) of the physicians had suggestions for improvements and two (25%) did not.
Participants also made suggestions for improvement. The most common suggestions were related to making it easier to act on the alerts. Possible improvements included providing clearer guidance and specific alternative recommended medications. This topic is explored in additional depth in the results of the unstructured data analysis, below.
When asked to speculate on why real-world PGx CDS response rates for the existing tools are relatively low (see previous publications [13,23]), the most common response was that the recipient of the alert may not be the right target audience. Six participants (75%) reported some variation on this idea; for instance, a different specialty may be more appropriate, or they may not have been the one to initiate the drug and do not want to modify it. Five (62.5%) believed that it could be due to patients that are already successfully on a medication and a lack of desire to change what is working.

Appropriate Locations for PGx Information
When asked about potential locations for presenting PGx information in the EHR, two of the seven options had zero participants rate them negatively on the seven-point Likert scale (ranging from Must Avoid to Must Have). "As an alert during order entry, when relevant" was the most approved-of option, with five participants (62.5%) rating it Very Helpful and three (37.5%) rating it a Must Have. "As an entry on the Problem List" also had no negative responses, with five participants rating it at least Helpful and three rating it Neutral. All response rates are shown in Figure 2.

Level of Detail and Interruption for Alerts
When asked about the level of detail and interruption an alert should provide, there was a general trend towards a preference for brevity in the information presented. Additionally, clinicians did not want to be interrupted for alerts that were primarily informational in nature, although they did prefer interruption when an action was required. The overall ranking for each display option was determined by the mode value. Each queried scenario and the aggregated clinician-reported rankings are listed in Table 2. Conversely, "A complete, detailed description of the drug-gene interaction, and how prescribing is affected, taken from the academic literature" was not well-received, with three participants (37.5%) rating it as Very Unhelpful, three (37.5%) rating it as Neutral, and two (25%) rating it as Helpful. The option "Your patient's actual genetic data in the format returned by the testing laboratory (e.g., CYP2C19 *1/*17)," was also rated negatively, with two participants (25%) rating it as Very Unhelpful, one (12.5%) rating it as Unhelpful, two (25%) rating it as Neutral, and three (37.5%) rating it as Helpful. All response rates are shown in Figure 3.  and common codes are presented in Table 3. participants expressing that PGx was less helpful for the particular drugs in this study (generally because they do not use those drugs frequently, or they are declining in popularity), 5 participants expressing some form of the idea that they were not the right audience for the alert, 5 participants relating the usefulness of PGx to its timeliness and relevance, and 5 participants expressing that PGx alerts aren't helpful if a patient is already successful on the medication in question.
Seven participants expressed that they review a patient's medication list early in their encounter process. Four stated that they were unlikely to go to a different screen to look up genetic information. Additionally, five participants expressed in some way that lists, such as problem lists and medication lists, are poorly maintained or inconsistent. Though participants were never asked about it explicitly, they frequently raised the issue of financial concerns. Drug and genetic testing affordability were frequently cited under Attitudes and Issues Affecting PGx Use. Additionally, five participants expressed a desire to see cost information included in alert text. In particular, they were interested in seeing information about the costs of the recommended alternative drug.
Overall, the themes that emerged during this qualitative analysis paint a clear and coherent narrative. Clinicians are receptive to the idea of using PGx in their practice, but they are currently uncomfortable with PGx and unclear and uneducated on how to use it. They feel a sense of burden about adding it to their workflows. They lack time to think about it, and often find it irrelevant. Therefore, any interventions based on PGx must be highly actionable with clear, concise, and explicitly stated recommendations. Implementers must be selective in which medications they build PGx alerts for and in which clinicians are exposed to them. These results are concordant with the findings from the structured analysis reported above.

Discussion
The high level of concordance we found in clinician preferences and attitudes allows us to establish a set of baseline principles upon which to build more effective PGx CDS at NM.
Several of the findings are consistent with previous research on non-PGx CDS and the general concepts of being timely and relevant. [25][26][27][28] However, others are especially relevant, or even novel, to the use of PGx information. We present these principles in Table 4. Make a clear recommendation to the user with a specific alternative medication and dosage, pre-calculated based off of the patient's genetics. Avoid recommendations to simply "consider alternatives." Do not interrupt the user with informational alerts about a patient's genetic profile if those test results are not immediately relevant. Ensure the user is the right person to see the information. Ensure alerts are built for the right medications.

Be Brief
Most physicians are not interested in reading in-depth information on genetics at the point of care, so keep text brief and simple. Make educational materials available through a link for those that want more background. Display Phenotypes not Genotypes Because most physicians are generally not well trained on genetics, they do not find raw genetic data useful in an alert. Avoid values like "CYP2C19 *2/*2," and instead provide a phenotypic interpretation such as "Clopidogrel Poor Metabolizer." Rely on Sources Clinicians Already Trust Because genetics-based prescribing is novel to many clinicians, lean on sources they already trust, whenever possible. Use information from UpToDate, established professional societies, or emphasize that your institution has vetted the recommendations coming from unfamiliar sources. Be Adaptable to Learning Effects As physicians learn and become more comfortable with PGx, their information needs may change over time. Review and revise alert designs on a regular basis or design them to be adaptable in the content they display.
In Table 5 Of relevance to PGx CDS, they concur with our recommendations around brevity and trust.
Their analysis is silent on the topic of genotype and phenotype, as well as on potential learning effects. While they touch on the concepts of specificity and actionability, they draw somewhat different conclusions than us. They recommend CDS that is more "advisory," because "some clinicians may perceive overly prescriptive advice [as] infringing on their sense of professional autonomy." Though we do not recommend PGx CDS alerts be wholly inflexible and commanding in their design, we do find that, likely due to lack of education on the topic, participants in this study preferred stronger, more specific recommendations and showed distaste for more ambiguous or unspecific guidance. Based on the results of this study, the issue of passive PGx alerting (i.e., optional to view, as opposed to interruptive) deserves particular attention. Of the seven location choices provided, "in an alert on a separate Best Practice Advisories tab in the patient's chart" was the least popular option among clinicians, due to the fact that many users do not regularly check that location in the EHR. This result suggests that passive alerting may not be the most effective approach. However, previous work at NM shows that even infrequent use leads to a significant number of actual alert interactions, due to sheer volume. Passive alerts should not be discounted because their default location is unpopular, but it is a good idea to reconsider where such alerts should appear. Our results suggest that another location that is already part of common workflows may be more appropriate, such as the Problem List or Medications List. In the future, as genetic data use becomes more common, a Genetic Information section may also be appropriate. These solutions will require PGx data standards and EHR support that either do not currently exist or are only in the earliest stages of functionality.
Two particular findings of this study concur with previous PGx literature. The general sentiments of uncertainty and doubt expressed by the clinicians we interviewed are consistent with previous publications documenting the low level of physician education in pharmacogenomics. [3] Additionally, our finding that clinicians hold a generally neutral view of PGx's helpfulness, with some receptivity to its use in the future, is consistent with our previous findings. [23] Given the time difference between when those attitudes were assessed (2015 vs. 2019), it does not appear that those attitudes are changing quickly at NM; nor have those attitudes been altered with further exposure to the first-generation PGx CDS tools at NM.
Our finding that PGx CDS recommendations need to be specific and actionable is at odds with the current state of PGx guidelines. Currently, the closest the field has to a "gold standard" for PGx CDS is the genetically-based prescribing recommendations published by CPIC. [7] These recommendations are very useful for identifying when a drug-gene interaction is relevant and when to interrupt the clinician, but they do not go so far as to provide a specific recommended alternative. Therefore, PGx CDS that directly translates CPIC recommendations in their published form is unlikely to be well received by clinicians. Organizations such as CPIC should strive to provide more specific recommendations in the future, but that will take significant time to achieve, due to the complexities of the decision-making process. Genetics are only one piece of the equation, with other patient-specific clinical factors being highly relevant.
Until the field develops comprehensive algorithms for calculating recommended medications and dosages based off of both genetics and traditional clinical variables, it will likely fall upon individual institutions to create and implement their own rules and recommendations. This will require substantial institutional commitment and will likely lead to further heterogeneity of practice.
In light of these results, we reiterate the need for standardized PGx interpretations and vocabularies, which we, and others, have called for in previous publications. [13,32,33] Phenotypic interpretations are clearly a critical component of conveying PGx knowledge to clinicians and standardized terms would significantly ease future implementations and EHR support. Additionally, computable vocabularies for representing this information in EHRs will be necessary for integrating PGx information deeply into the clinician workflow outside of just pop-up alerts (e.g., Problem and Medication List support, and Genetic Information screens).
Finally, in recognition of the fact that most currently-practicing clinicians have not been trained in PGx, [3] we would call upon the sources that clinicians most trust, such as professional societies like the American College of Physicians and American College of Cardiologists, and reference providers like UpToDate, to seriously examine and expand upon the use of PGx data in their guidelines and best practices, when appropriate. Particularly in light of recently issued FDA guidance for CDS, it is critical that there be established clinical guidelines that PGx CDS systems can cite in their recommendations. [34] Conclusion Through the use of socio-technical principles, this study demonstrates areas in which PGx CDS is different from traditional CDS and provides direction for future PGx CDS designs.
Unlike traditional CDS, PGx CDS requires abstraction of clinical data through the use of phenotypes rather than genotypes. Additionally, lack of training means that recommendations must be clear and strong and that there are likely learning effects to account for. Clinicians showed a preference for brief alerts that are clear about the expected action with specific alternative recommended medications and dosages. They also frequently expressed feelings of uncertainty and doubt around the use of PGx, so future designs must focus on simplicity and specificity while referencing well-trusted sources. Designs following these principles are more likely to encourage adherence than those that simply provide information for the clinician to consider on their own. Therefore, current CPIC PGx recommendations do not directly translate to successful CDS because they do not provide specific recommendations. Until specific recommendations become more standardized, any institution interested in using PGx CDS must commit to establishing recommended alternatives. Likewise, the field as a whole must work to develop standardized phenotypic interpretations and vocabularies for clinically relevant PGx results.