{'fraction_non_alphanumeric': 0.02553424196533737, 'fraction_numerical': 0.008244994110718492, 'mean_word_length': 5.592623405435385, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Assessing noncognitive domains of respiratory therapy applicants: Messick’s framework appraisal of the multiple mini-interview

Educators who assess incoming applicants into a health professional training program are looking for a wide array of cognitive and noncognitive skills that best predict success in the program and as a future practicing professional. While aptitude tests generally measure cognitive skills, noncognitive constructs are more difficult to measure appropriately. The traditional method of measuring noncognitive constructs has been the panel interview. Panel interviews have been described as inconsistent in measuring noncognitive domains and consistently reported as unreliable and susceptible to bias. An alternate interview method used in many health professions schools is the multiple mini-interview (MMI) that was specifically designed to assess noncognitive domains in health professions education. This paper discusses the purpose of using the MMI, how the MMI is conducted, specific domains of focus for the MMI, and the feasibility of creating an MMI. Finally, the paper uses Messick’s framework on validity to guide the consideration of the MMI.


INTRODUCTION
The level of education of applicants to respiratory therapy programs can range from basic high school diplomas to international medical degrees. Initially, the applicants are screened based on prerequisite foundation courses and ranked by estimates of academic success (i.e., grade point average) that have been associated with academic success [1,2]. The top-ranked applicants are expected to present themselves to the educational institution and take part in a panel interview. The panel interview is the long-standing assessment method used at most traditional health professions schools to assess candidates' personality characteristics [3]. In respiratory therapy, the characteristics or constructs the educators are assessing include many noncognitive traits, personality traits, and affective skills such as adaptability, social skills, professionalism, and empathy. Such personality characteristics are important in determining who will succeed in training, they have been associated with professional development and career satisfaction, and are also linked to improved patient care outcomes [4,5]. Such characteristics outline the basic expectations of entry-to-practice Registered Respiratory Therapists (RRTs) [6].
The typical panel interviews in health professions education admissions are comprised of practicing health professionals, faculty members, students, and members of the community. Panel interviewers engage in an unstructured or semi-structured dialogue with a candidate for varied amounts of time, typically ranging from 15 to 60 min [7,8]. There have been many inconsistencies reported in academic literature regarding the use of a panel interview to measure noncognitive domains; it has been reported to be consistently unreliable and susceptible to bias [9][10][11]. The question then arises, how can we improve the selection process to best assess applicants' noncognitive characteristics to choose candidates who will likely best contribute to the medical society and flourish as health professionals?
An alternative method to the traditional panel interview that has been adopted by many health professions is the multiple mini-interview (MMI), created at McMaster University, Hamilton, Ontario, in 2002 [12]. The MMI generally consists of 10 stations, with one interviewer at each station. The process is as follows: each interviewee has 2 min to read a question, followed by the interviewee providing their response for the remaining 6-8 min. MMIs thus allow applicants the opportunity to interact with multiple interviewers [9]. Available systematic reviews of the MMI process in medical schools have reported on its reliability, acceptability, and feasibility of implementation [10] along with the ability to predict clinical performance of successful applicants [13].

METHODOLOGY
Below we consider the MMI as an alternative to traditional panel interviews and summarize the domains the MMI assesses. Messick's framework on validity [14] will guide consideration of MMIs. This will include examination of prior works that pertained to reliability, validity, efforts to minimize bias and utility, and the feasibility of integrating the MMIs into the application process of a respiratory therapy training program.
Morris [15] highlighted many shortcomings with using the panel style of interview. Weaknesses of the panel interview include the incurred costs because of the substantial time commitment required of staff and other interviewers as well as the expense and inconvenience to the applicants from the travelling, accommodations, and time lost from school or work [16]. The high subjectivity of interviewers' scores, the varying structure of interviews, the lack of interviewing training and the accompanying biases have a major influence on the validity and reliability of traditional interviews to assess noncognitive features of applicants accurately. The MMI was developed by McMaster's School of Medicine in 2002 to address the concern that panel interviews did not adequately predict the noncognitive performances of medical school applicants. Eva et al. [17] developed the MMI process in an attempt to dilute the impact of individual examiners and allow for a more standardized and generalizable performance rating to be assigned to each applicant [9].

Procedure
The logistics of MMIs are affected by the educational resources available (i.e., space, cost). The most common method of conducting MMIs consists of candidates rotating through multiple stations. Each interview station assesses a single, predetermined construct domain that reflects the objectives of the institution. Each station is kept short (8 min. or less) and is conducted simultaneously in a circuit. With each toll of the bell, the participant moves onto the next interview session in the circuit [18]. Typically, a single interviewer per interview station is sufficient to assess a candidate on the construct [17]. Each interviewer stays in the same interview room throughout the day, as candidates rotate through. The interviewer then scores each candidate based upon the same interview scenario throughout the interview process ( Figure 1).

Domains
The noncognitive domains chosen to become the focus of McMaster's pilot MMI included communication skills, ethical decision-making, critical thinking, collaboration, and knowledge of the health care system. These domains were selected merely based on the preferences of the research team [19]. Other educational institutions chose domains that best represented the school's mission statement or cultural context [20,21]. As the MMI became formally adopted in McMaster's medical school application process, there was an increased motivation by McMaster to outline and validate the desired noncognitive domains systematically. Reiter et al. [19] chose to blueprint the desired noncognitive domains against three prominent medical education frameworks that emphasized the essential competencies of a physician. The Accreditation Council for Graduate Medical Education, the CanMEDS 2000, and the Core Committee of the Institute for International Medical Education all provided outlines, which included competencies that overlapped significantly with one another. These roles and competence domains derived from a much larger number of cognitive and personal qualities cited in the literature as desirable when choosing medical students [19]. Reiter then narrowed down the competencies to a series of seven characteristics and surveyed key stakeholders to assess which were most relevant for McMaster's medical students [19]. The conclusion supported that stakeholders ranked ethical nature, communication, and intelligence as the three most desirable domains in health professionals. The authors then used these domains as a base to build MMI scenarios for McMaster's undergraduate medical program. The authors cautioned that the domains outlined in their study are best used as a guide to developing admissions protocols.
The objective of the MMI is to reveal and assess noncognitive domains that are particularly crucial to the educational institution or health profession in question [19]. The RRT curriculum is founded on many domains of competency that are considered essential components of practice. Domains such as professional behaviour, collaborating with the interdisciplinary team, and effective communication are skills that professionals use in daily practice and significantly overlap with those outlined by the three prominent medical education frameworks listed prior, particularly CanMEDS [6,22]. To use the MMI to assess future RRT candidates appropriately, key stakeholders (i.e., practicing RRTs, educators, managers etc.) would have to blueprint the MMI stations to mirror the outlined, desirable, noncognitive characteristic of entry-topractice RRTs [6].

Validity
Validity is a dynamic process whereby we construct arguments with either weak or strong evidence to support our assessment decisions [23]. Similar to hypothesis testing, validity evidence is collected to support or refute the validity hypothesis [24]. There is conflicting evidence surrounding the use of panel interviews as a valid measure of assessing noncognitive constructs in health professions education [1]. While some investigators report positive correlations between pre-admission interview ratings and in-course academic performance, others conclude that the interview is not predictive of success [15]. Salvatori [14] provided a comprehensive review of a range of admission tools used by health education professionals and found that unstructured interviews provided conflicting predictive validity of outcomes. When considering the exuberant cost involved in training today's health professionals and the public's expectation of professional accountability [25,26], admission committees must include evidence-based selection criteria to ensure that the best-qualified candidates are selected. Qualified candidates must encompass both academic strength and clinically relevant constructs. Examples of these constructs include the previously outlined noncognitive domains, personality traits, and affective skills. While panel interviews had shown a slight improvement in inter-rater reliability and validity scores when interviewers were scripted and had more structure [27,28], the MMI further improves validity scores because of their format. The MMI format was specifically designed to assess intangible constructs. To argue for evidence of the validity of the MMI, Messick's [14] framework will be used to analyze the construct validity of the MMIs. Messick's [14] framework for construct validity includes content validity, response validity, internal structure validity, relationship to other variables, and consequential validity.
Content evidence of validity refers to the logical-empirical relationship of content tested to the achievement domain [29]. MMI stations that are constructed around the academic institution's "blueprints" or desired competencies have a higher content validity. Blueprinting ensures there is an optimal match between the MMI testing stations and the desired characteristics to be tested from the candidates [9]. To ensure content validity for selecting RRTs, the MMI stations would have to be constructed alongside our professional blueprint that specifically outlines the desired core competencies of entry-to-practice RRTs [6].
Response process validity is defined as evidence of data integrity. All sources of error associated with the test administration should be controlled or eliminated as much as possible [29]. Examples of areas of response process include quality control of assessors (i.e., how to assess individuals and what is the purpose of assessment) [30] and ensuring student familiarity with the format [31]. Stakeholders who are involved in the RRT interviews must be made aware of what each MMI scenario is attempting to assess. Additionally, incoming applicants must also be made aware of the MMI interview structure, so they may appropriately prepare for the interview.
Internal structure, as a source of validity evidence, relates to the statistical or psychometric characteristics of the examination questions or performance prompts, the scale properties (i.e., reproducibility and generalizability), and the psychometric model used to score and scale the assessment. Reliability is defined as the reproducibility of assessment data or scores, over time or occasions [29]. Reliability does not lay in the test itself, but rather, the outcome of the test. When we look at the reliability of the MMI, we are assessing if the MMI was able to consistently differentiate the constructs that were tested in each station [32]. Some examples of estimating reliability include test-retest reliability and interrater reliability.
Test-retest reliability assesses the degree to which test scores are consistent from one test administration from the next. Simply put, a candidate's response should score the same on the same question if given at different time intervals. Although test-retest is fundamental in reliability estimates, providing a test to candidates and then retesting them with a similar test is logistically challenging to carry out [32]. The MMI has the ability to test the same applicant on two parallel forms of a test for a particular construct indicating good test-retest reliability [9]. Gafni et al.'s [33] research on failed applicants who retook the same MMI in the same institution also reported good test-retest correlation, suggesting that MMI performance does not vary between attempts [33].
The inter-rater reliability is another measure of validity most useful when calculating scores of behaviours by rater observation. Inter-rater reliability assesses the degree to which test scores are consistent when measurements are measured by different raters with the same, standardized method [32]. Unstructured, nonstandardized panel interviews are the most common method chosen to conduct candidate interviews. The panel interview is void of any standardization, and inter-rater reliability is grossly inconsistent [34]. The existent literature on unstructured interviews indicates ambiguous results [1]. The panel interview inter-rater reliability consistently reports lower than acceptable averages, and inconsistencies may be a result of potential interview bias. The literature indicates that interviewers consistently scored candidates higher when personality preferences matched [35]. In an attempt to control for interviewer bias, it is best to employ the individual opinions of multiple observers.
The inter-rater reliability increases when raters were carefully selected, they received interview training, there were multiple observers, and interview questions were blueprinted against the medical school's goals or objectives [36]. These strategies are the foundational pillars of the MMI format, which assisted in the widespread acceptability of the MMIs. The early MMIs inter-rater reliability was initially called into question because of the single interviewer per station format, which may still be at risk for interviewer bias [1]. Eva et al. [13] recognized the bias and used a statistical theory for evaluating reliability of behavioural measurements (Generalizability Statistical Theory) [37] to conclude that increasing both the number of raters and the number of stations significantly improves the MMI's reliability. Multiple-rater MMI reliability has consistently been reported in other contexts as well [20,30]. Finally, another source of internal structure validity is validating the item difficultydiscrimination. Item difficulty is the measure of the proportion of examinees who answered the item correctly and item discrimination is the measure of how well an item is able to distinguish between examinees who are knowledgeable and who are not [38]. One such strategy is to create a quality-controlled question bank to help assessment designers guarantee a degree of validity [39].
When high-stakes decisions are to be made using an instrument, there must be a high degree of reliability to make defensible decisions. In the case of selecting proficient candidates for a competitive health education program, admission committees must accurately assess all desired constructs reliably. Panel interviews have been shown to have ambiguous results, whereas the MMIs held more promise with stronger reliability [40,41]. Modifying the MMIs' format to include a larger number of assessors and scenarios could further improve the reliability.
Relationship to other variable validity refers to how well the MMI construct scenarios correlates with other variables. That is, convergent correlations are when a different type of assessment method is used to assess the same construct, and both results are similar. In contrast, divergent validity refers to another assessment method that evaluates a different construct unrelated to the first. In noncognitive measures, certain studies of MMIs have suggested a moderate correlation between scenarios testing the same construct [33,42]. Additionally, MMI scores have low correlations with scores on traditional measures of cognitive ability (i.e., grade point average) suggesting they measure different constructs [42].
Finally, the consequential area of validity refers to the impact on examinees from the assessment scores, decisions, and outcomes, and the impact of assessments on teaching and learning. The consequences affect many stakeholders including examinees, faculty, patients, and society. These consequences can be positive or negative, intended or unintended [29]. A test must reliably safeguard the public from individuals who will cause harm if they can proceed and practice. Likewise, the exam should not fail and withhold appropriate applicants who will eventually provide a much-needed service to the public as health professionals.
Consequences of the MMIs in reported literature have consistently shown that those accepted in medical school by the MMIs outperform applicants accepted by traditional admission processes in the educational context and in high-stakes licensure exams [20,40,41]. The commonly used, unstructured interview that the RRT admission committee uses offers little predictive validity of objectively assessing future performance of applicants [43,44] (Table 1).

FEASIBILITY
When attempts are made to implement a novel tool within practice or an existing curriculum, a significant discussion point is the tool's feasibility. Feasibility is often a point of concern for many stakeholders, predominantly those with administrative duties. Implementing the MMIs at face value seems more resource-intensive than the panel interview counterpart. Although there is some truth that the initial creation of stations, blueprinting institutional objectives, creating evaluation forms, and simulation-based scenarios take longer, each component can subsequently be modified and reused in the following years. Rosenfeld et al. [18] provided a synopsis of the cost-efficiency of MMIs relating to the administrative duties, miscellaneous expenses, interview training, assessors, staff, infrastructure requirements, and their experience creating the McMaster MMI process. Eva et al. [9] further elaborated on the McMaster process of interviewing 400 medical school applicants; excluding the expenses generated for the initial project framework and organization, the reported costs of the actual MMI operation on the interview day is about $35 per candidate. Hissbach et al. [45] discussed the slight modification of their existing MMI process that cut costs by approximately 50%. Similar results exist in other medical domains, Corelli et al. [46] noted a cost-effectiveness per candidate interviewed for pharmacy school of $75.30 with the MMI format compared with $136.34 with a standard interview [46].
Regarding logistics, Brownell et al. [47] reported many advantages of including the MMIs compared with panel interviews in medical school admissions. With the MMIs, interviews were completed in 2 days rather than the average 8 days. They required less personnel and it did not cost any more compared with previous years. Finlayson et al. [48] also reported on the success of implementing MMIs in a medical residency training process, claiming the process is cost-effective, required less time from volunteers, held good inter-rater reliability, and participants simply enjoyed the process more. While challenges and the absolute cost will vary according to the education institutions, frameworks exist to guide educators regarding the logistical requirements and anticipated budgets [18].

CONCLUSION
Noncognitive abilities are an essential aspect for RRTs and all other health care professionals. Rather than assuming candidates will eventually develop these skills, admission committees are well positioned to assess these noncognitive domains early on and select appropriate candidates who will exemplify these desired traits. Selection committees often rely on a panel interview to expose noncognitive abilities. Panel interviews are consistently reported as unreliable in detecting noncognitive abilities and predicting future performance. The MMIs are a commonly used alternative interview tool to better assess noncognitive domains of a candidate with strong validity and reliability.
Despite the strong validity and reliability, admission committees cannot simply duplicate an MMI from one school or health domain and expect it to work within a different context. The MMIs commonly reported validity and reliability is context specific and requires a "ground-up" approach when implementing. Educators interested in including MMIs within their admission criterions are expected to actively validate their (context-specific) MMI validity. Validity is a dynamic process that uses an exam tool to collect evidence to support or refute the assessor's hypothesis about an applicant. Using Messick's [28] framework to assess the MMIs, construct validity has repeatedly been favourable, and the MMI's feasibility of curricular integration has been reported to be relatively easy. For the MMIs to be justly included in assessing RRT applicants, the station scenarios and the constructs tested must be tailored and blueprinted against the RRT's educational institutions goals, objectives, and the desired noncognitive abilities of an entry-to-practice RRT.
Developing and implementing MMIs for respiratory therapy schools in Canada is worthy of consideration. To facilitate this one might establish a multicentered working group responsible for blueprinting MMI stations to RRT entry-to-practice constructs, developing training material for academic institutions and interviewers, facilitating pilot implementation at certain institutions, and of course evaluating the interview validity. We must be vigilant in continuously assessing and modifying the RRT-specific MMIs to maintain the validity in testing incoming applicants. Finally, the MMI must adhere to the institution's logistical allowance to be sustainable. The adoption of an RRT-specific MMI for assessing applicant's noncognitive abilities could be a unique way to avoid the pitfalls associated with traditional panel interviews and could lead to more reliable and professional students who, ultimately, will become strong professionals.

FUNDING
This study did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.