{'fraction_non_alphanumeric': 0.034790033717213294, 'fraction_numerical': 0.01690239523580155, 'mean_word_length': 5.511047754811119, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Performance Measurement in Mental Health and Addictions Systems: A Scoping Review

Objective: The purpose of this study was to evaluate how performance is defined, conceptualized, and measured in mental health and addiction service systems around the world. Method: We conducted a systematic scoping review of English-language scientific and gray literature published from 2005 to 2015. Eligible documents (n = 222) described performance measurement systems and outlined the theory or empirical evidence for indicators. We used a structured approach for data extraction and descriptive and thematic analysis, supplemented with stakeholder consultation. Results: We identified seven themes in the literature: similarity in performance domains across frameworks; the ability of frameworks to inform care quality at client, program/facility, and system levels; the predominance of indicators of process and outcome, over structure; the lack of evidence on the links between domains and/or indicators; common, but limited, evaluation of family/caregiver involvement; equity as a cross-cutting domain of performance; and limited attention to performance measurement in peer support services. Conclusions: The literature on performance measurement in mental health and addictions services is vast, and a wide variety of indicators is available to those designing a measurement system. Evaluations of commonly used performance indicators have yielded mixed evidence on their ability to discriminate high- and low-performing service providers, and their sensitivity to changes in policies and practices. As performance measurement efforts grow in scope and complexity, work will be needed to ensure that indicators are fair, appropriate, and suited to support quality improvement in services of different types. Objectif : Évaluer la façon dont le rendement est défini, conceptualisé et mesuré dans les services en santé mentale et en toxicomanie à travers le monde. Méthode : Nous avons mené un examen de la portée à l’aide d’une revue systématique des écrits scientifiques et de la littérature grise de langue anglaise publiés entre 2005 et 2015. Les documents admissibles (n = 222) décrivaient les systèmes de mesure du rendement et présentaient les théories ou les données empiriques concernant les indicateurs. Nous avons utilisé une approche structurée pour l’extraction des données, suivie d’une analyse descriptive et thématique en complément à une consultation des acteurs impliqués. Résultats : Nous avons identifié sept thèmes dans les écrits : similarité dans les domaines de rendement entre les systèmes; la capacité des systèmes à documenter la qualité des soins tant au niveau du client, du programme que de l’organisation; la prédominance des indicateurs de processus et de résultats sur ceux liés à la structure; le manque de données sur les liens entre les domaines et/ou les indicateurs; l’évaluation commune, mais limitée, de l’implication des familles et des proches-aidants; l’équité en tant que domaine transversal du rendement; et une attention limitée à la mesure du rendement dans les services de soutien par les pairs. Conclusion: L’ensemble des écrits portant sur la mesure du rendement des services en santé mentale et en toxicomanie est vaste et les concepteurs de systèmes de mesure disposent d’une grande variété d’indicateurs. Les évaluations des indicateurs de rendement couramment utilisés ont produit des données contradictoires sur leur capacité à distinguer un rendement faible d’un rendement élevé ainsi que leur sensibilité aux changements dans les politiques et les pratiques. Au fur et à mesure que les mesures du rendement prendront de l’ampleur et de la complexité, il sera nécessaire de s’assurer que les indicateurs soient justes, appropriés et adaptés pour soutenir l’amélioration de la qualité des différents types de services. Objetivo: Evaluar cómo se define, conceptualiza y mide el rendimiento en los sistemas de servicios de salud mental y adicción en todo el mundo. Método: Realizamos una revisión sistemática del alcance de la literatura científica y gris publicada en inglés entre 2005–2015. Documentos elegibles (n = 222) describen sistemas de medición de rendimiento y describen la teoría o evidencia empírica de indicadores. Utilizamos un enfoque estructurado para la extracción de datos y el análisis descriptivo y temático, complementado con la consulta a los interesados. Resultados: Se identificaron 7 temas en la literatura: similitud en los dominios de rendimiento a través de marcos; la capacidad de los marcos para informar a la calidad de atención al cliente, programa / instalación, y los niveles del sistema; el predominio de indicadores de proceso y resultado, sobre la estructura; la falta de evidencia sobre los enlaces entre dominios y / o indicadores; evaluación común, pero limitada, de la participación de la familia / cuidador; equidad como un dominio transversal de rendimiento; y una atención limitada a la medición del rendimiento de los servicios de apoyo entre iguales. Conclusiones: La literatura sobre la medición del rendimiento en servicios de salud mental y adicciones es enorme, y hay una amplia variedad de indicadores disponibles para quienes diseñan un sistema de medición. Las evaluaciones de los indicadores de desempeño comúnmente utilizados han arrojado evidencia mixta sobre su capacidad para discriminar a los proveedores de servicios de alto y bajo rendimiento, y su sensibilidad a los cambios en las pólizas y prácticas. A medida que los esfuerzos de medición del desempeño crezcan en alcance y complejidad, será necesario trabajar para garantizar que los indicadores sean justos, apropiados y adecuados para respaldar el mejoramiento de la calidad en servicios de diferentes tipos.

P ERFORMANCE MEASUREMENT in health care plays a crucial role in holding the system accountable to maintaining high quality of services for patients and consumers and in helping stakeholders make informed policy decisions (Smith et al., 2008). The primary objective of performance measurement is to monitor and evaluate how well specified measures for performance and quality are met. Population health, individual health outcomes, quality and appropriate-ness of care, responsiveness of the system, equity, and productivity are common areas of health system performance measurement.
In designing a performance measurement system, stakeholders will follow one of two broad approaches (Loeb, 2004). They may fi rst evaluate what data are available and develop indicators that can be measured using those data. Although this approach is cost-effective (as it uses preexisting data sources), the easiest, cheapest performance measures are often of little value for evaluating quality improvement initiatives and may not adequately represent strategic goals for the system. Alternatively, stakeholders may fi rst determine what should be measured and why, and then design indicators and seek out the required data. Given that health systems involve many and varied stakeholders with diverse needs, performance measurement efforts face a real challenge in serving all of these needs without requiring additional data collection and analysis (Smith et al., 2008).
Performance measurement in the healthcare sector has seen a dramatic increase in recent decades, stemming in part from increased costs and pressures to contain them, as well as patient demands for quality care (Smith, 2005;Smith et al., 2008). The kind of evidence that is generated from performance measurement is valuable for informing system evaluation and enhancement, needs-based planning, and resource allocation. Over and above clinical research and program evaluation that speaks to the effectiveness of individual services and therapeutic approaches, systems research is needed to provide evidence on how well (or not) services are coordinated and delivered so as to affect population mental health and substance-related harms (Babor & Poznyak, 2010;Babor et al., 2008). Despite a growing body of systems research related to substance use, addictions, and mental health services (exemplified in this issue), there is a lack of research that synthesizes evidence across mental health and addictions (MHA) performance measurement initiatives, including systematic evaluation of how high performance is defined, conceptualized, and measured. Shared understandings of system performance may help to guide performance measurement activities, as well as contribute to the development and evolution of service systems themselves.
We addressed this gap in the research literature by conducting a systematic scoping study of MHA performance measurement systems around the world. Our primary aim was to evaluate how performance is defined, conceptualized, and measured. Because the study was conducted to inform the development of an MHA performance measurement framework in the province of Ontario, Canada (estimated population 13.6 million), we also included evidence on frameworks generated for the broader health care system in Canada (in addition to those generated internationally for MHA service systems). This was done to facilitate alignment of the framework development effort with existing initiatives in Canada.

Search strategy
We used the approach described by Arksey and O'Malley (2005) and Levac et al. (2010) to explore how performance has been conceptualized and measured in existing MHA frameworks, including the content of domains and indicators BOX 1. Scoping review search strategy Databases/search engines: PubMed, MEDLINE, Cochrane, EMBASE, CINAHL, PsycINFO Terms (used in combination): Performance measurement, performance indicators, performance frameworks, quality indicators, quality improvement, system performance, scorecards, report cards; quality care, quality reports, healthcare quality, health system quality documents that outline frameworks for general health care and health system performance measurement in Canada (e.g., hospital care, primary care); literature that describes the prevalence, correlates, and outcomes of existing performance indicators Exclusion criteria: Commentaries, editorials; documents on performance measurement in areas other than health; documents that describe the purpose and goals of performance measurement without reference to performance domains or framework development; documents that describe statistical or methodological issues (e.g., risk adjustment, validity/reliability); documents that describe stakeholder views, experiences, ethical issues, or implementation of performance measurement (without describing the framework and its development); descriptions or evaluations of educational initiatives for providers or clinical guidelines; documents that outline performance measurement frameworks in Canada that are specific to non-MHA conditions (e.g., heart disease, cancer, diabetes); program evaluations, evaluations of specific services, types of interventions, or innovations in practice; evaluations of quality improvement initiatives that do not involve performance measurement; original research or reviews that report on treatment processes and outcomes generally, but not within the scope of a performance measurement strategy or framework FIGURE 1. Flow chart of document selection and the purported causal links between them. We included English-language scientifi c and gray literature published from 2005 to 2015. Documents were identified through keyword searches in online library databases, on websites of key government and academic organizations, and through requests to subject area academic experts (the search strategy, with inclusion and exclusion criteria, is detailed in Box 1). We also searched reference lists of eligible documents. Identified documents were screened for eligibility through title and abstract review. Where this was not sufficient to determine eligibility, the full document was reviewed. Eligible documents contained information on framework development, including methods, definitions, concepts, and theories. The primary focus of the study was to review the evidence on performance measurement in MHA service systems internationally. However, as noted above, to be responsive to the needs of local decision makers and health planners, we also searched for evidence on frameworks generated for the broader health care system in Canada. Therefore, documents produced internationally were included if they were focused on performance measurement in MHA service systems, whereas docu-ments produced in Canada were included if they addressed performance measurement in healthcare systems more broadly. There were no other geographical differences in the eligibility criteria used to select documents for review. Ineligible documents focused solely on implementation, ethical issues, stakeholder experiences, statistical issues in risk adjustment, or indicator psychometrics. In cases of report series, the most recent report was selected for inclusion. During the early stages of document screening and selection, 10% of documents were screened by two team members independently to check reliability. Discrepancies were settled by discussion.

Data extraction and analysis
We designed and used a standardized coding tool for data extraction. A random sample of 10% of documents was coded independently by two reviewers, and percentage agreement was calculated for fi ve selected fi elds. With one exception, agreement across reviewers was reasonably high (above 70%). For the fi eld pertaining to "the associations between domains, between indicators, and/or other corre-lates and outcomes," agreement was no better than chance (52%). Discrepancies occurred because of differences across reviewers in judgments as to whether the articles contained this kind of information. Articles were re-reviewed by the project lead (K.U.) for this fi eld.
We used a structured analytical approach to generate a descriptive summary of document types and characteristics, followed by a thematic analysis of data pertaining to conceptualization of performance. For existing frameworks, we tabulated performance domains and population and temporal dimension(s) at which indicators are measured. Population dimensions referred to the level of measurement, categorized as client, program/facility, and system; whereas temporal dimensions referred to structure (the resources needed for service delivery), process (what is done to and for people), and outcome (the results of care; Donabedian, 2005;Gaebel et al., 2015;Rush et al., 2009). For this part of the analysis, we used as a reference the framework for measuring general health system performance in Ontario (Health Quality Ontario, 2014b, 2015b), which is based on the U.S. Institute of Medicine (IOM) framework (Institute of Medicine, 2006). The content of frameworks developed in other jurisdictions was compared to this reference.
Theoretical and empirical fi ndings on the relationships between domains and/or indicators were synthesized separately, with additional themes allowed to emerge to yield a full picture of the conceptualization of MHA service performance. A single article or report could, and typically did, contribute evidence on more than one theme.

Stakeholder consultation
Preliminary findings were shared with stakeholders to obtain feedback and help shape key project messages. Three structured, interactive presentations were conducted in 2016 to a provincial advisory panel convened to guide system enhancement projects in Ontario, Canada (n~25); a provincial task group on MHA performance measurement (n~20); and a panel of people with lived experience that meets regularly to contribute to research at a large psychiatric teaching hospital (n~10). Individual consultations were held with quality improvement specialists working on system enhancement projects in the province (n = 2) and representatives of peer-led/peer-run support organizations (n = 2). The consultations with quality improvement specialists included questions designed to elicit feedback on the results of the thematic analysis (i.e., concerning the utility and feasibility of performance indicators on families and caregivers, equity, peer support, and structure/process/outcomes). The consultations with representatives from peer support organizations focused on discussing the theme of performance measurement in peer support settings. Feedback from stakeholders was synthesized with fi ndings from the literature review, under the relevant themes.

Descriptive summary of documents
The literature search identified 17,510 unduplicated articles and reports ( Figure 1). After excluding non-Englishlanguage articles, editorials, commentaries, and articles that did not meet the selection criteria, 225 documents remained for review. Of these, 3 theses that were completed at a small Canadian university could not be located. The fi nal review included 222 articles and reports (a full bibliography is available on request).
The review included studies conducted in 21 countries across 5 continents (North America, Europe, Australia, Asia, and Africa). The United States contributed the largest number of reports (39.8%), followed by Canada (31.7%), England (5.9%), the Netherlands (4.5%), and Australia (4.5%); the remainder were conducted in various other countries. Studies used a variety of approaches to generate evidence, including focus groups, interviews, expert panels, Delphi process, administrative data analysis, literature review, case studies, surveys, and concept mapping. The majority of reports concerned performance measurement in MHA service systems (80.6%), with the remainder pertaining to general health care services (19.4%). By design, the literature on general health care services was restricted to work conducted in Canada, whereas the literature on MHA services was international in scope. Of the included reports, a minority concerned services for children and youth (10.8%) or substance use and addiction services (18.0%). Most of the literature (69.4% of the 222 reports) concerned performance measurement within general or multiple health care settings; 13.1% hospital or inpatient services; 7.2% primary care; 6.3% outpatient services (exclusive of primary care); 1.4% services delivered in prisons; 1.4% pharmacy care; and 1.4% emergency or crisis care settings. Document types included original (peer-reviewed) research articles (66.7%), gray literature reports (21.2%), review articles (9.9%), and conference abstracts (2.3%).

Key themes in MHA performance measurement
Results are organized into seven key themes, including the fi nding of similarity in performance domains across frameworks; the ability of frameworks to inform care quality at client, program/facility, and system levels; the predominance of indicators of process and outcome over structure; the lack of evidence on the links between domains and/ or indicators; common, but limited, evaluation of family/ caregiver involvement; equity as a cross-cutting domain of performance; and limited attention to performance measurement in peer support services.
Similarity in performance domains across frameworks. There is a high degree of similarity across frameworks in the ways that MHA performance is conceptualized (Tables  1 and 2). In addition to the six IOM domains, frameworks The core services or clinical activities that fall within the scope of primary care; specifically, health promotion and primary prevention, secondary prevention, care of chronic conditions and care of acute conditions. b Comprehensiveness is defined as the "ability to ensure the tailoring of services to health care needs" (Hogg et al., 2008, p. 312), whereas the patient-provider relationship reflects interpersonal communication, respect and trust, whole-person care, cultural sensitivity, family-centered care, and advocacy. c Defined as "the physical, cultural, psychosocial and work/job design conditions that maximize health and well-being of health providers, quality of patient/client outcomes and organizational performance" ( Table 1 in Biron et al., 2012).   Defined as consideration for clients' values, preferences and expressed needs in the care they receive. c The WHO-AIMS/WHO-SAIMS are tools for assessing regional or national mental health care and substance use systems, respectively. Items are grouped into subscales assessing the jurisdiction's policy and legislative framework, population need for services, the types and characteristics of specialized and primary MHA services, human resources, public education and links with other sectors, and monitoring and research. Items were cross-referenced with the performance domains in the CQA. d Measured in terms of client engagement and perceptions of care and therefore aligned closely with acceptability or client centeredness. e Defined as assistance to communities in using evidence-based activities to foster healthy living, such as the use of prevention services; screening and assessment for alcohol, tobacco, and other drugs; housing; legal system involvement; and medication side effects.
commonly incorporated measures of appropriateness, integration or continuity, supporting resources (e.g., spending, workforce issues, competence, evaluation), and population health (e.g., prevalence, risk/protective factors, population characteristics). There is a fair amount of conceptual overlap between domains listed separately in the tables. For instance, appropriateness (whether services are relevant to people's needs and based on accepted standards/evidence) overlaps conceptually with effectiveness. In other frameworks, issues akin to appropriateness/effectiveness were captured through measures of core services specifi c to a given type of provider (e.g., primary care providers; Barnsley et al., 2005;Hogg et al., 2008;Puszka et al., 2015;Waraich et al., 2010), or to a given care setting (e.g., inpatient psychiatric liaison services; Solomons et al., 2011). These basic fi ndings with respect to performance domains were reflected in frameworks that emphasized strategic goals during development (e.g., Mental Health Commission of Canada, 2015; Ministry of Health and Long-Term Care 2008; Ve illard et al., 2010), as well as those that used techniques such as concept mapping or other group-based activities to identify the main components of performance (Graham et al., 2014;Holmes et al., 2014;McCusker et al., 2013;Nabitz et al., 2005aNabitz et al., , 2005bResnick & Griffiths, 2010;Roeg et al., 2005;Sayal et al., 2012;Va rgo et al., 2013). For example, in a study of performance definitions across stakeholder groups (clinicians, health system managers, patients, policy makers, and researchers), accessibility, continuity, patient centeredness, comprehensiveness, coordination, effectiveness, equity, and safety emerged as key components, named by three or more of the fi ve groups (UBC Centre for Health Services and Policy Research, 2015). Frameworks consisting of lists of indicators, without the organizing structure provided by domains, also did not contribute any new information over and above the more structured frameworks (e.g., American Society of Addiction Medicine, 2014; British Columbia Office of the Provincial Health Officer, 2015; Cole et al., 2012;Holmes et al., 2014;Laugharne & Shankar, 2009;Levitt et al., 2014;Nigam et al., 2008;Parameswaran et al., 2015;Samu et al., 2011).
In our own stakeholder consultation, discussions with people with lived experience of the system highlighted the importance of responsiveness (whether services are responsive to the needs, expectations, and preferences of service users and their families). Stakeholders also noted a preference for the term person-directed care, over client-or person-centered care.
Ability to inform care quality at client, program/facility, and system levels. With a couple of exceptions, all frameworks had the capacity to offer insights into care quality at the client, program or facility, and system levels (second to last column in Tables 1 and 2). Frameworks contained both indicators specific to a given level of measurement (e.g., provider density is inherently a system-level measure), and indicators that were defined at a lower level but could be reasonably rolled up for interpretation at a higher level (e.g., wait times are measured for individuals who use services, but also provide information at program and system levels).
Overall, few framework development initiatives addressed the issue of population dimensions explicitly (see Mental Health Transformation Workgroup, 2011; Turning Point, 2014b, for exceptions). During framework development in the state of Victoria, Australia, service providers identified client-level measures as the most relevant to quality improvement in their services (measures of wait times, follow-up activities, reductions in substance use, and client satisfaction), whereas accreditation (a program-level measure) was the indicator to which they most felt they could be held accountable (Turning Point, 2014a). Links between services (a system-level measure) were also seen as both useful and important for accountability.
Predominance of indicators of process and outcome, over structure. Much greater attention was paid in the literature to temporal dimensions, likely owing to the traction of Donabedian's conceptual framework for assessing quality in health care (Donabedian, 2005). A number of reports framed their evaluation of performance solely or primarily in terms of structures, processes, and outcomes (e.g., Cheng et al., 2010;Dausey et al., 2009;Grabowski et al., 2010;Kilbourne et al., 2010;Meehan et al., 2007;Roeg et al., 2005;Schaub et al., 2013). Most frameworks contained elements of all three dimensions (last column in Tables 1 and 2).
Structure tended to be the least frequently measured temporal dimension, typically incorporated as measures of sustainability or appropriate resources. In some cases, structure was limited to single measures of provider competence, or whether evaluations are conducted (Addington et al., 2005(Addington et al., , 2007(Addington et al., , 2012Chovil, 2010;Waraich et al., 2010). A number of reports cited a need for greater attention to exploring and developing structural indicators (Baars et al., 2010;Hogg et al., 2008), whereas others have highlighted specific structural features as key priorities for future indicator development work (Canadian Institutes for Health Information, 2015; Garnick et al., 2012). In a study focused on conceptualizing structure in addiction treatment settings, aspects identified by service providers included inter-organizational cooperation and coordination, fi nances and facility characteristics, professionalization, staff autonomy, and job requirements (Roeg et al., 2008).
Process measures predominate in performance measurement (Henderson et al., 2014;Lauriks et al., 2012;Patel et al., 2015). In addition to being relatively easy to capture with existing administrative data, there is a clear link between treatment process and provider accountability and costs (Asch et al., 2011;Carrick et al., 2013;McLellan et al., 2007;O'Brien et al., 2007). Relative to outcomes, processes are also more actionable and sensitive to differences in performance (Baars et al., 2010;Watkins et al., 2010).
An example of development work that has focused heavily on treatment processes is given by the Washington Circle, a U.S.-based multidisciplinary expert panel convened to develop performance measures for addiction treatment (Garnick et al., 2006(Garnick et al., , 2009(Garnick et al., , 2012. Outcome measures were also typically present in some form. The most common outcomes captured in existing frameworks related to service use (e.g., retention, length of stay, 30-day readmission), symptoms and functioning, and client or family perceptions of care. These are consistent with outcome domains identified as priorities by administrators and health planners (Herbeck et al., 2010). Less commonly measured outcome domains were quality of life, substance use, or social relationships (a full analysis of outcome domains tabulated across frameworks is available on request).
There was variability in the ways that outcomes are monitored. The importance of capturing multiple perspectives (e.g., client, family, provider) to get the full picture of service quality was noted (Gaebel et al., 2015;Institute of Medicine, 2006;Lauriks et al., 2012;National Mental Health Performance Subcommittee, 2013). On point, client perceptions of care, a common outcome measure, tend to vary more within than between providers (reviewed in Ruud, 2009). It is entirely legitimate for clients to experience care delivered by a single provider differently; however, this fi nding does raise questions as to whether client perceptions can be used to distinguish between high-and low-performing providers (this is true even when the clients' reports of their own experiences is valid). Several jurisdictions have opted for routine outcome monitoring using standardized tools, such as Health of the Nation Outcome Scales (HoNOS; Alberta Health Services, 2015; National Mental Health Performance Subcommittee, 2013; New Zealand Mental Health and Addictions KPI Programme, 2010). In addiction service settings, routine monitoring has traditionally been limited to urinalysis and measures of treatment attendance, although more recent efforts to monitor progress and outcomes over the course of treatment have emerged (Goodman et al., 2013;McLellan et al., 2007).
Lack of evidence on the links between domains and/or indicators. Conceptual linkages between domains are rarely specifi ed in performance frameworks (see Canadian Institutes for Health Information, 2015, for an exception). Relative to linkages between domains, the associations between indicators are more commonly addressed, although it is still the case that the causal relationships between indicators are typically either not specified or not validated through research (Lauriks et al., 2012). Demonstrating empirical links between indicators, particularly across temporal dimensions (i.e., structures, processes, and outcomes), gives evidence of indicator validity. There is evidence linking national, regional, and facility characteristics (e.g., income inequality, gross domestic product, material deprivation, provider density, facility size) with care processes (Costa et al., 2014;Olfson et al., 2010), but not necessarily outcomes (Desai et al., 2005;Hendryx, 2008). A substantially larger body of literature has examined the links between measures of processes and outcomes; for instance, showing better outcomes among clients who meet benchmarks for care continuity and the frequency of visits in the initial weeks of treatment (Desai et al., 2005;Garner et al., 2010;Garnick et al., 2007;Greenberg & Rosenheck, 2005;Harris et al., 2010;Rost et al., 2005). Although documented at the client level, many of these associations have not been replicated with indicators aggregated to the facility level (Desai et al., 2005;Harris et al., 2007Harris et al., , 2009, raising questions about the ability of these indicators to discriminate between high-and low-performing facilities. Common, but limited, evaluation of family/caregiver involvement. Family-centered care, family-provider relationship, and, to a lesser extent, provision of direct services to family are often incorporated as features of effectiveness and person-centeredness or responsiveness (e.g., Addington et al., 2012;Balfour et al., 2016;Chovil, 2010;Hogg et al., 2008;Jones, 2005;New Zealand Mental Health and Addictions KPI Programme, 2010;Va rgo et al., 2013;Waraich et al., 2010;Zima et al., 2005). Family involvement in care was particularly common in frameworks for youth (Chovil, 2010;Va rgo et al., 2013;Zima et al., 2005). Australia's National Mental Health Performance Framework offered the opportunity for potentially more comprehensive representation of family and caregivers by specifying both client and caregiver in their definitions of service effectiveness, appropriateness, responsiveness, and safety; however, indicators do not appear to have been developed in all of these domains (Brown & Pirkis, 2009;National Mental Health Performance Subcommittee, 2013).
Equity as a cross-cutting domain. Equity is typically conceptualized as cross-cutting other performance domains, allowing for exploration of how features of effectiveness, accessibility, safety, and so on are distributed across the population. Indicators are broken down by diagnosis and various sociodemographic characteristics (e.g., age, sex, gender identity, ethno-cultural background, immigrant status, socioeconomic status). Others have noted the importance of moving beyond patient characteristics to incorporating measures of program and system structures in the evaluation of equity (e.g., the numbers and characteristics of people who are excluded from services because of admission rules or capacity issues; the sociodemographic characteristics of staff members; staff training and support for culturally safe practice; the use of interpreters) (Nakaima et al., 2013;Wong et al., 2011Wong et al., , 2014. In Australia's system of MHA services, equity is used in framework development, with applicability across diverse populations being one of the criteria for indicator selection (National Mental Health Performance Subcommittee, 2013).
Limited attention to performance measures in peer support services. Ve ry limited attention is given to peer support services in the performance literature, despite their established presence in many jurisdictions. Most commonly, where peer support was reflected in existing frameworks, it was as a single indicator focused on availability, with no information captured on capacity, scope of activities, or effectiveness (Ganju et al., 2005;Mental Health Transformation Workgroup, 2011;Oregon Health Authority, 2014;Parameswaran et al., 2015). It was also unclear whether frameworks included peer support service within their scope (i.e., whether they were expected to report on and be held accountable to the same performance indicators as other services).
We identified one initiative to develop quality indicators specifically for parent-delivered support services in child mental health (Kutash et al., 2014;Olin et al., 2014aOlin et al., , 2014b. This initiative underscored some of the challenges inherent in applying traditional performance measures, with authors citing expert disagreement over whether the use of standardized assessment protocols was consistent with the principles of peer support. We heard similar feedback in our stakeholder consultation, with stakeholders from peer support organizations noting that the collection of unique identifiers for members is inconsistent with the values of peer supports and could be expected to interfere with recovery.
Despite the challenges, it was a repeated theme, in both the literature and our stakeholder consultation with peer support organizations, that some form of performance measurement is crucial to establishing the legitimacy and ensuring the viability of authentic peer support services going forward. The development of customized indicators that align with the values and objectives of peer support services was seen as both possible and preferable to exclusion from such initiatives.

Discussion
Our primary objective was to conduct a comprehensive and systematic conceptual exploration of performance in MHA service systems. Using a structured process, we identified a large number of documents from the academic and gray literature (222), which was supplemented with consultations with a variety of stakeholder groups. Our synthesis of fi ndings highlights a number of key considerations for jurisdictions engaged in designing and implementing performance measurement activities in the MHA service sector.
There are a fi nite number of ways to measure performance, and so there were, on the whole, few notable differences between frameworks in the content of performance domains and indicators. The majority of existing MHA performance measurement frameworks balance measures that can be used to express information about clients, programs/ facilities, and systems (i.e., they yield information about variability across episodes of care and service providers, as well as speaking to system and population health). From a public health perspective, it is of interest to understand the cumulative impact of a service system on population health (e.g., incidence of substance-related harms, substance use disorders, and suicide, crime rates, HIV infection; Babor et al., 2008). Overall, population health indicators featured in only a minority of frameworks, and there was little attention overall to the potential for performance measurement initiatives to further our understanding of the population health impacts of treatment and other supports. This is an area for future research, particularly with respect to substance use and addictions, as stakeholders develop strategies and policies to respond to escalating rates of substance-related harms in their jurisdictions (Jones et al., 2018;Wood, 2018).
Measures of process (what is done to and for people) and outcomes (the results of care) are relatively more common than are measures of structure (the resources that are needed for service delivery). There are a number of likely reasons for this. In addition to being difficult to define, a favorable structural measure indicates only the capacity for, rather than the occurrence of, high-quality services (Henderson et al., 2014)-it indicates only whether the conditions were in place for good quality care. That said, challenges related to chronic underfunding, including infrastructure shortcomings and difficulties recruiting and retaining staff (Eby et al., 2010;Gallon et al., 2003;McLellan et al., 2003;Ogborne & Graves, 2005) may be a rationale for measuring and monitoring structural inputs specifically in the MHA service sector. To the extent that they lend a perceived sense of fairness to performance measurement (for instance, by allowing service providers to highlight their challenges in meeting treatment demand and delivering high-quality services), they may be important tools for advocacy and equity in policy development. Furthermore, measuring outcomes in the absence of client and population characteristics runs the risk of rewarding good outcomes without controlling for illness severity or social determinants of health (Baars et al., 2010)-suggesting that the robust measurement of outcomes requires measures of structure. Despite recognized challenges, developing valid measures of structure was a stated priority for future work in many jurisdictions (Canadian Institutes for Health Information, 2015; Garnick et al., 2012).
It was noted that the causal relationships between indicators are typically either not specified or not validated through studies (Lauriks et al., 2012), but that the relationships between health system performance and actual health outcomes are varied and not always in the expected direction (Arah & Westert, 2005). This gap in evidence hinders our ability to use performance data to derive policy that will maximize population health (Etches et al., 2006). Evaluations of commonly used performance indicators (e.g., continuity of care, service intensity, client perceptions of care) have yielded mixed evidence on their ability to discriminate high-and low-performing service provid-ers, and their sensitivity to changes in policies and practices. Measures that vary more within than between providers, as has been found for client perceptions of care, may not be suited to performance measurement if the purpose is to distinguish between high-and low-performing providers within the system. However, such fi ndings also highlight the need to look at whether and how performance measures vary across client subgroups (e.g., whether women or people with a particular diagnosis tend to have a better or poorer experience with a given provider). In evaluations of the Washington Circle performance measures, the positive association between treatment engagement (based on service frequency in the initial weeks of treatment) and arrest was found to be weaker in Black and Latino clients (Acevedo et al., 2015), whereas its association with hospital readmission was apparent only among those who had a prior suicide attempt (Glass et al., 2010). Much more work of this nature is needed to clarify the interpretation and the use of fi ndings from performance measurement in service planning and policy development.
There is evidence of a potentially widening role for equity in performance measurement (e.g., Nakaima et al., 2013;Wong et al., 2011Wong et al., , 2014, with measures moving beyond patient sociodemographic characteristics to structural inputs. Although sex and gender fi gure prominently within existing frameworks, we identified only one framework that included parental status in the evaluation of equity (Turning Point, 2014b). Given the gendered nature of barriers to MHA treatment, particularly for women who are pregnant and parenting (Cormier et al., 2004;Greenfield et al., 2007), monitoring service access and effectiveness by parental status is a notable gap in existing frameworks.
Finally, there are legitimate challenges associated with implementing performance measurement in peer support services, combined with a general lack of examples in the literature on ways in which this has been done successfully in the past. As performance measurement in MHA continues to evolve in Canada and elsewhere, efforts should be made to partner with peer support organizations to ensure both that the services they offer are represented in these strategic initiatives and that the indicators on which they are evaluated are fair, appropriate, and able to support quality improvement.
The vast majority of evidence that we reviewed was produced in high-income countries, generated in relatively well resourced service systems. In this respect, our study was likely limited by the exclusion of non-English-language documents and biased toward representing MHA systems in We stern countries. We identified only one document from South East Asia (Singapore) and one from Africa (South Africa). No documents were identified from India, China, or South America (however, it should be noted that some of these countries were represented in the international studies that we reviewed). Systems research and performance measurement are less common in low-and middle-income countries (LMIC; Babor, 2017;Myers et al., 2012Myers et al., , 2015, where infrastructures for data and other supports may be lacking (Myers et al., 2014). Efforts to establish international quality indicators for substance use treatment that include LMIC (United Nations Office on Drugs and Crime/World Health Organization, 2016) may help to expand performance measurement initiatives globally.
Within this context, the present study provides a valuable empirical synthesis of how the concept of performance has been operationalized in MHA systems, albeit in predominantly high-income countries. The literature on this topic is vast. Systematic and structured syntheses such as this one are valuable tools for highlighting key considerations for researchers and planning bodies embarking on performance measurement activities in the MHA service sector. Offering a critical refl ection on the state of performance measurement and definitions of care quality, fi ndings may likewise serve to guide the development and evolution of MHA service systems, whether they are being newly established or expanded and enhanced to better meet population need.
As described, although precautions were taken to promote inter-coder reliability in study selection and coding, low reliability was obtained for one key fi eld related to the coding of information on associations between domains and/or indicators. We attempted to minimize the impact of this limitation through re-coding the fi eld.
Despite the limitations, we are confident that this study offers a reasonably robust picture of how performance has been conceptualized in MHA service systems globally. Future key considerations for performance measurement activities include the implementation of indicators to better capture the population health impact of MHA service systems and help drive public health responses to reducing substance-related harms, as well as the development and implementation of structural indicators to better capture system contexts and resource needs. Given the significant role played by peer support in MHA services, additional work is also needed to establish how its role can best be captured within performance measurement systems.
Finally, greater attention is needed overall to the causal linkages between indicators. This is particularly crucial to avoiding the creation of performance measurement systems that simply rely on available data, rather than capturing accepted and understood elements of care quality. Evaluations of commonly used performance indicators have yielded mixed evidence on their ability to discriminate high-and low-performing service providers, and their sensitivity to changes in policies and practices. As performance measurement efforts grow in scope and complexity, work will be needed to ensure that indicators are fair, appropriate, and suited to support quality improvement in services of different types.