{'fraction_non_alphanumeric': 0.031204057630108792, 'fraction_numerical': 0.012496324610408704, 'mean_word_length': 5.26069949378739, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Revising the Tool for Assessing Cultural Competence Training (TACCT) for curriculum evaluation: Findings derived from seven US schools and expert consensus

Background: The 67-item TACCT currently used for needs assessment has potential for evaluating evolving cultural competence (CC) curricula. Purpose: To validate a shortened, more practical TACCT measure. Methods: The 67-item TACCT was administered to students and course directors at US schools. Course directors and students reported which of 67 TACCT items were taught. Intraclass correlation coefficients (ICC) examined faculty-student agreement. Under-addressed content was identified. A new and shortened TACCT configuration was proposed and validated with expert educator input. Results: Across-school faculty and student response rates ranged from 75% to 100%. Aggregate ICC was 0.90 (95% CI: 0.84, 0.94) for the 67-item TACCT, demonstrating faculty-student agreement. Experts agreed on reduction from 67 to 42 items and domain revision from five to six domains to match under-addressed content. Item analysis showed high internal consistency for all 6 new domains and the total revised 42-item TACCT. Conclusions: A shorter, more practical TACCT measure is valid and reliable and focuses on under-addressed CC content. Use for curricular evaluation is suggested.

the value and impact of studies reporting the effectiveness of specific educational interventions and asserted that attention should be paid to the proper design, evaluation, and reporting of such training programs and courses. 8 Another systematic review examined a wide array of tools for assessing learner attitudes and CC curricula and noted little standardization for use across medical schools. 9 For example, a survey of 19 US medical schools in 2001 10 identified 8 important content areas in cross-cultural education for medical students and suggested a standard nomenclature for measuring 'the success of cross-cultural education curricula'. In this paper we focus on the need to identify core content that addresses knowledge, skills and attitudes leading to cultural proficiency and competence promoting improved healthcare outcomes in the context of medical encounters with diverse patients. The primary term we choose to use for this purpose is 'cultural competence' to broadly cover an array of terms used in the literature ranging from 'cultural humility' to 'diversity' to 'cross-cultural communication skills'.
The accreditation standards of the Liaison Committee on Medical Education (LCME) 11 specify two cultural competence guidelines. First, 'Medical students should learn to recognize and appropriately address gender and cultural biases in health care delivery, while considering first the health of the patient.' Second, 'The faculty and students must demonstrate an understanding of the manner in which people of diverse cultures and belief systems perceive health and illness and respond to various symptoms, diseases, and treatments.' Based on the LCME guidelines, the Association of American Medical Colleges (AAMC) developed the TACCT for use as a needs assessment tool. The TACCT was designed by a consensus panel of experts. Its intended use was to measure the degree to which the various content elements of CC occur throughout the curricula of US and Canadian medical schools from the perspective of teaching faculty. The measure has five domains comprising of 67 CC content-specific items (or learning objectives) representing knowledge, skills, and attitudes, mirroring a prior AAMC curriculum assessment measure for palliative care. 14 Since 2004, the National Institutes of Health (NIH) has funded 18 US medical schools to design, implement, and disseminate model CC curricula. 15 Separately, in 2005 the AAMC also supported four California medical schools to develop and implement model CC curricula. Two schools received both awards, for a total of 20 schools in the 2 consortia. Investigators at one funded school, the University of California, Irvine (UCI), initially administered the TACCT to both faculty and students and found high congruence (intraclass correlation coefficient =0.89) between faculty and student perceptions of whether CC content, as expressed in the TACCT items, was presented in the extant curriculum. Furthermore, students were significantly more likely to identify content as being covered compared to course directors overall. 16 Faculty and students also agreed on content least likely to be addressed which fell into three broad content areas: health disparities, bias and stereotyping, and community strategies. Subsequently, the TACCT was used by six additional schools among the NIH and AAMC awardees to conduct a baseline curriculum needs assessment.
However, in its original form the TACCT poses a number of challenges for routine use (e.g., annual or repeated administration) in curriculum evaluation. Mainly, the number of items or learning objectives (n=67) is daunting at first glance to potential respondents. Also, the wording of some items appears to overlap and requires respondents to make fine discriminations in intended meaning (e.g., 'identify physician biases that affect clinical care,' 'value the need to address personal bias,' 'recognize how physician biases impact care'). Furthermore, distinctions between generic communication objectives (e.g., 'value curiosity, empathy and respect') and CC-specific communication objectives (e.g., 'respect patient's cultural beliefs') are not clearly made. Finally, the professional behavior of self-reflection in relation to CC is not explicitly included but is instead embedded within other domains.

Purpose
Our objectives were, first, to identify areas of leastaddressed content and second, to apply expert judgment and statistical principles to develop a shorter TACCT.

Methods
Study Sample -Seven US schools divided between the east and west coast regions (three and four schools, respectively) participated in the survey; three were state institutions. The self-selected schools belonged to two consortia of collaborative institutions awarded grants to implement cultural competency curricula by the NIH (six schools) and the AAMC (two schools). One school belonged to both groups, for a total of 7 participating schools. The respective institutional review boards at the seven schools approved the study. Respondents were both medical students in the clinical phase of training (third year medical students [MS3] or fourth year medical students [MS4], n=662), who had completed at least the required core curriculum, and course and clerkship directors (n=144) of the core required medical school courses at the respective institutions. Reported institutional data are de-identified.
TACCT Administration and Data Collection -Five schools (schools 1, 3, 5, 6 and 7) administered the 67-item TACCT 12 to students as a written questionnaire either during a Clinical Practice Examination or during a required class. Two schools (schools 2 and 4) first administered the TACCT to students online using email and a web link. A subsequent face-to-face administration was used with non-responders. Faculty respondents were surveyed by email listserv solicitation followed by a face-to-face administration at school 1 and by written questionnaire only at the other schools. The method of administration (online vs. face-to-face pen and pencil) was determined by each school based on past experience with survey response rates for their particular respondents, with the purpose of maximizing response rate.
Students and faculty were instructed to check all items (scored as 'yes' = 1) that they felt had been adequately addressed in the curriculum medical students (MS) or in their own courses (faculty). Items unchecked were scored as 'not addressed' = 0. Detailed explanation of individual TACCT item content was not provided.
Respondents were asked to interpret each item at face value as they understood it and to not check items they did not understand or that were not taught in the required curriculum.
TACCT administration was completed within a 12-week timeframe by both students and faculty in all schools. Average time for completion of the TACCT was 15 to 20 minutes.
Data Analysis -The frequency of 'yes' responses to each TACCT item was tabulated separately for students and faculty. Mean percentage item scores were computed (sum of 'yes' responses divided by number of respondents). Domain scores were computed from summing items that belonged to each of the five conceptual TACCT domains. Finally, within each domain, knowledge, skill, and attitude scale scores were computed from summing the pertinent items belonging to each respective category. The authors also examined the distributions of faculty and student TACCT item scores to identify clusters of items denoting content areas defined as 'not adequately addressed' in the curriculum (i.e., in the lowest quartile of responses). The intraclass correlation coefficient (ICC) was calculated to evaluate the degree of concordance among faculty and student responses.
Because of concerns about faculty and student responses being skewed in opposite directions, nonparametric Exact Mann-Whitney Tests compared faculty and student responses on each TACCT item and with the corresponding domain and scale scores. One-way analysis of variance (ANOVA) examined potential between-school differences on both domain and scale scores. Significant F-tests were followed by pair-wise mean comparisons by either Neuman-Keuls (homogenous group variances) or Games-Howell tests (heterogeneous variances). The researchers used the nominal, two-sided α <0.05 for testing statistical significance. Because we computed multiple comparisons, the conservative Bonferroni correction (i.e., α divided by number of contrasts) was applied to each family of comparisons (e.g., student versus faculty within each school). All analyses were performed using SPSS 14.0 for Windows (SPSS Inc., Chicago, Illinois).
TACCT Review Process -The expert survey occurred within eight months of completing the 7-school survey. Over a three-month period, experts defined as educators with experience administering and using the TACCT were identified and recruited from among the NIH Consortium and the AAMC awardees. They first were asked (by telephone, email, and in person) if they had used the TACCT for educational needs assessment at their institutions. They then received the 67-item TACCT and findings from both the prior UC-Irvine study 16 and the recently completed 7-school study. A diversity education representative from the AAMC, who was involved in the construction and dissemination of the TACCT, was included at this stage of the review process. The experts were asked to independently (a) review the 67-item TACT, (b) suggest alternative clusters of items and reconfigured domains for curriculum evaluation based on either the studies' findings or their own experiences with the TACCT, and (c) add their individual suggestions, explanations and comments about revising and improving the measure. Based on the composite suggestions, the TACT was restructured and then sent back to the experts for further review and to achieve consensus agreement about changes made to the measure. At a joint meeting of both the NIH Consortium and AAMC awardees convened by the AAMC Cultural Competency Education committee in September 2007 and attended by all 20 schools' representatives, the restructured TACCT was subjected to final peer review and scrutiny.
Reliability Analysis of Restructured TACCT -Using the survey responses from the seven schools, item analysis was performed using conventional methods. That is, internal consistency reliabilities were assessed by computing Cronbach's coefficient alpha (α) for the set of pertinent items comprising each newly configured domains. Within each domain, we computed the point-biserial correlation coefficients of each individual item score with the total domain score in which the items clustered. Also, for within-domain analyses, α was recomputed by deleting each item score from the total domain score to assess the relative contribution of each individual item to its respective broader domain score.

Results
Response Rates and Student Demographics -Student response rates varied from 75% to 90%; faculty rates ranged from 95% to 100%. Ethnic/racial profiles of students represented significant diversity among the student body within each school (Table 1). Ranges within schools included 6% to 75% non-Hispanic white, 2% to 75% African-American, 1% to 28% Hispanic/Mexican-American, and 4% to 40% Asian-American. Percentage of males ranged from 45% to 55%. The IRB status of the research protocol precluded identifiers to correlate individual responses with respondent.
Item-Level Findings -Students consistently responded 'yes' at a higher rate than faculty to every one of the 67 TACCT items ( Table 2, Appendix I). Using a conservative criterion for statistical significance (p<.001), students provided a statistically higher mean percentage of 'yes' responses to individual TACCT items on 47% of items within schools (range 3% to 90%) and on 100% of items in the aggregate seven schools. Although students responded affirmatively more often than faculty, the two groups generally agreed about what specific elements of cultural competence instruction did or did not occur in the first three years of the curriculum. That is, students and faculty rank ordered in similar fashion the relative occurrence (or non-occurrence) of the cultural competence content represented in the 67 separate TACCT items ( Table 2). The range of intraclass correlation coefficients (ICC) within individual schools was 0.70 to 0.89. For the aggregate seven schools the ICC was 0.90 (95% CI: 0.84, 0.94) ( Table 3). Three were no statistically significant differences by pattern of student demographics in the level of concordance seen between faculty and student respondents (data not shown).

Domain and Scale-Level Findings (67-item TAC-CT) -Examination of the possible domain and scale
4 Lie DA, Boker J, Crandall S, DeGannes CN, Elliott D, Henderson P, Kodjio C, Seng L. Revising the tool for assessing cultural competence training (TACCT) for curriculum evaluation... scores produced a similar pattern to that found with individual TACCT items (Table 4). Students, compared to faculty, yielded statistically significantly higher mean scores on 71% (range 4% to 100%) of domain and scale scores within schools and 100% of the time in the aggregate. Schools 2, 3 and 5 were notable in their magnitudes of differences between mean total TACCT scores from faculty and students (44%, 42%, and 47% respectively). The smallest faculty-student TACCT total score difference was 20% in School 4 (Table 4). Considering only students' responses, significant (p<.0005) betweenschool differences occurred on every domain and scale score, except for Domain V Knowledge ('Cross Cultural Clinical Skills'), where all schools produced similar mean scores (78%, range 69% -84%). In contrast, no significant between-school differences emerged from analysis of faculty domain and scale scores. Students from Schools 2, 3, and 6 consistently responded 'yes' at rates greater than 80% to TACCT items categorized as knowledge, skill, and attitude. Consequently, these three schools had significantly higher total TACCT scale scores than their four counterparts (Table 4).
Under-Addressed Content Areas -Among the seven schools, using the 67-item TACCT, we identified 19 TACCT items for students and faculty that fell within the lowest quartile of 'yes' responses and portrayed, by definition, under-covered curriculum content (see Table 2 ranking and shaded area). Among this item set, 14 were identical ( Table 2, shaded and in bold) for the two groups (students and faculty) and clustered into three broad content areas. The first broad content area was Community Strategies. It was represented by the following TACCT items: 'describe community-based elements', 'describe methods to identify community leaders', 'propose a community-based health intervention', 'describe community partnering strategies' and 'collaborate with communities to address needs'. The second broad content area was Health Disparities. This area was framed by items that include: 'critically appraise literature on health disparities', 'gather and use data as in HP (Healthy People) 2010', 'describe social cognitive factors', and 'concretize the epidemiology of disparities'. The third under-addressed content area was labeled Bias/Stereotyping and contained the following cluster of items: 'show strategies to reduce bias in others', 'value the historical impact of racism', 'describe historical models of health beliefs', and 'strategize ways to counteract bias'.
Expert Consensus Results -Nine experts and the AAMC representative returned suggestions for revising the TACCT. They agreed that the under-addressed curricular areas (Health Disparities, Community Strategies and Bias/Stereotyping) should each occupy a separate domain for ongoing curriculum evaluation to ensure that they were distinctly tracked as new CC curricula were introduced. They also agreed that individual TACCT items (or objectives) should be preserved in their original form of wording and not reworded. Thus, no new TACCT items were introduced in the restructuring process. All 9 experts agreed that there was over-representation of objectives addressing bias and stereotyping in the original TACCT and suggested reducing the number of knowledge, attitude, and skill items in the bias/ stereotyping content from 17 items in the 67-item TACCT to 6 items (see Table 5, Appendix II, Domain II). The experts agreed that distinct domains representing crosscultural communication skills, interpreter use skills, and self-reflection in the context of the culture of medicine should be included (see Table 3, Domains IV, V and VI). Removing redundant objectives from the original TACCT in total reduced the measure by 25 items, arriving at a final number of 42 items (or objectives). The final restructured TACCT (Table 5 Appendex II) comprised six renamed domains, each with no more than 10 objectives: health disparities, community strategies bias/stereotyping, cross-cultural communication skills, use of interpreters and self-reflection/culture of medicine. The revised 42item measure was sent back to the experts for review with full consensus reached on the new domains and domain items. At a meeting of both consortia in September 2007, the 42-item TACCT was reviewed by representatives of 20 schools and no further revisions were advocated by the group.
Item Analysis of Restructured TACCT -Using the 7-school data on the 67-item TACCT, the inter-rater agreement as measured by ICC between medical student and faculty responses to the 42 items of the reconfigured TACCT was .905 (95% CI, .816, .947). Reducing the number of objectives from 67 to 42 thus did not affect medical student-faculty agreement. Conventional item analysis including Cronbach's coefficient alpha showed solid results in support of the restructuring of knowledge, skill and attitude domains in the newly configured TACCT. Cronbach's α for each of the six new domains ranged from .803 to .875. Overall, the α coefficients for all reconfigured knowledge, skill, and attitude objectives were .914, .923, and .857, respectively. The total new TACCT with 42 items had α = .964 (see Table 5). When α was recalculated by deleting each constituent item score from the new domain scores to which they clustered, the resulting coefficient always was lower then when the item was included, suggesting that each item made a positive contribution to the variance in the total domain score (data not shown). The item score-total score correlation coefficients in content and knowledge, skill and attitude domains were consistently moderate to high by conventional definition (i.e., never < .40).

Discussion and Conclusions
The present study with seven geographically dispersed schools replicated and further demonstrated the reliability and concordance of student and faculty responses shown in a prior single-school study. Likewise, three under-addressed content areas identified in the single-school study were confirmed by administering the original 67-item, 5-domain TACCT. An additional study arm using expert review and consensus yielded a restructured TACCT that ostensibly improved its utility for curriculum evaluation. Finally, examination of the psychometric characteristics of the restructured 42-item TACCT showed that reducing the length of the measure did not detract from its internal structure and reliability.
Since students responded according to their instructional exposure across at least three years of the curriculum, whereas faculty course directors responded in the more limited context of instruction provided only within their own formal courses, it perhaps was not surprising that students systematically checked more TACCT items. One reason for these differences may be that students included experiences with cultural competence content in the 'informal or hidden curriculum' in responding. Concomitantly, observed variability in schools may reflect a combination of real differences in the informal and formal curricula and perhaps differential recall of both experiences. Content least likely to be addressed was similar for each school and overall. The three content areas found least likely to be addressed in all schools were Community Strategies, Health Disparities, and Bias/Stereotyping. This remarkable symmetry underscores the importance of developing curriculum that can be used across schools to address these content areas.
Lie DA, Boker J, Crandall S, DeGannes CN, Elliott D, Henderson P, Kodjio C, Seng L. Revising the tool for assessing cultural competence training (TACCT) for curriculum evaluation... It was reassuring that restructuring the TACCT into six renamed domains that included the least addressed areas identified in the 7-school study with a reduction by 25 items (or objectives) did not reduce the internal consistency reliability of the TACCT overall, each domain, and the separate components of knowledge, skill and attitude. The total new TACCT Cronbach's alpha and individual alphas for the new domains support the future use of the restructured 42-item TACCT as a curriculum evaluation measure. The introduction of a new domain (VI) of 'self-reflection, culture of medicine' echoes the a priori expert judgment solicited from educators and this judgment was affirmed by the high internal consistency reliability (Cronbach's α = .803) of this 5-item domain.

Med Educ
Individual schools have different curricular orientations, local community needs, and diverse geographic, cultural, and ethnic backgrounds among students and faculty. There may be differences with respect to the role and importance of cultural competence education for health professionals and with respect to the manner in which this education should be offered. By providing a revised TACCT based on results obtained from multiple schools, our study contributes to the need to address measuring the effectiveness of curricular change in CC education. The restructured 42-item TACCT is more practical and user-friendly than the 67-item TACCT and specifically recognizes key areas of curricular content currently under-addressed in most schools. As such, it is a feasible alternative to the longer original 67-item TACCT for schools undergoing curricular change in CC instruction.
The strengths of the current study are that the seven schools were diverse in respective student demographics, but the TACCT administration method was relatively uniform across the schools, data collection was conducted within a short timeframe, and there were high response rates from both students and faculty. The use of peer review by experts from two consortia representing 20 US schools addressing CC education as a common goal is another strength. Limitations of the study are both the relatively small number of schools represented and potential variability in interpretations of individual TACCT items by respondents within and across schools. The current study does not address the content of the informal curriculum that student responses may have included, and we believe that this aspect of the CC curriculum may best be addressed in greater depth using qualitative methods. This current study was not intended to prescribe particular curricula to address the efficacy of teaching in particular content areas such as community strategies, health disparities or specific cross-cultural communication skills, as such curricula are well described and available in the literature. [20][21][22][23][24][25] The applicability of the TACCT to non-US and non-Canadian schools not guided by LCME standards is uncertain. Despite the shortcomings described, concordance among faculty and students in our multi-school needs assessment was remarkably high and consistent within each school. In addition, agreement among the peer experts familiar with CC curricula and the TACCT was consistently high, and their findings were in turn supported by the feedback from experts from all 20 NIH and AAMC funded schools. Further testing and use to examine the validity of the restructured 42-item TACCT as a curriculum evaluation measure is needed.
In summary, the TACCT in its restructured format is practical, has face and content validity, and is a reliable instrument to administer with straightforward instructions to medical students and faculty. However, despite its comprehensive coverage of CC learning objectives, the original or revised TACCT is not intended to be prescriptive, in that it does not identify specific or best teaching for meeting the objectives or for evaluating learners. That particular task of curriculum implementation, in our opinion, is best achieved at each school for its curriculum by its own educators, because, like other curricular areas, there is not a 'one size fits all' approach for CC education as the literature reporting myriad CC curricula suggests. Students may have greater exposure to cultural competence in the entire formal and informal curriculum experience, in comparison to faculty whose contact is inherently limited to parts of the formal curriculum. Thus, we advocate that both student and faculty viewpoints should be considered in planning CC curricula. Furthermore, the diversity of the population residing in the geographic location surrounding the institution may also influence responses to the TACCT. Future studies should attempt to separate the latter effects from that of the medical school curriculum. Combining an externally validated objective measure of curricular coverage in cultural competence for each school (such as an external review of syllabus and teaching materials and in-depth interview of students and course directors or focus group studies, for example) with the current results may allow confirmation of this observation and help distinguish the informal from the formal curriculum.