{'fraction_non_alphanumeric': 0.02965912322756267, 'fraction_numerical': 0.009385284199406818, 'mean_word_length': 5.2266632937009865, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Engaging citizens in the co-production of a health system performance assessment framework: a case study in Ireland

Background The launch in 2017 of the Irish 10-year reform programme Sláintecare represents a key commitment in the future of the health system. An important component of the programme was the development of a health system performance assessment (HSPA) framework. In 2019, the Department of Health of Ireland (DoH) and Health Service Executive (HSE) commissioned the technical support of researchers to develop an outcome-oriented HSPA framework, which should reect the shared priorities of multiple stakeholders, including citizens. This study describes the method applied in the Irish context and reects on the added value of using a citizen panel in the co-production of an HSPA framework. Methods A panel of 15 citizens was convened, recruited by a third-party company using a sampling strategy to achieve a balanced mix representing the Irish society. Panelists received lay-language preparatory materials prior to the meeting. Panelists used a three-color scheme to signal the inclusion and importance of performance measures. An exit questionnaire was administered to understand how participants experienced being part of the panel. The citizen panel was the rst in a series of three panels towards the development of the HSPA framework, followed by panels including representatives of the DoH and HSE, and representatives from professional associations and special interest groups. Results The citizen panel generated 249 health performance measures ranging across 13 domains. Domains assessed as the most important included people-centeredness, coordination of care, and coverage. Prioritization of domains differed between panels. Citizen panelists shared a similar understanding of what a citizen panel involves and described their experience at the panel as enjoyable, interesting, and informative. Conclusions The engagement of citizens early on in the co-production process of the HSPA framework shaped the processes that followed, with the restating of priorities of the citizen panel informing decision-making throughout. Citizen engagement in HSPA development is essential for realizing value-based people-centered health systems and assuring an inclusive process that helps to generate trust and ownership of performance intelligence. Future research could expand on how citizen panels could be further engaged in co-creating mechanisms to assess, monitor, and report on the performance of health care systems.


Background
Health system performance measurement, and its use as performance intelligence, plays a central role in guiding the decisions of health system actors towards improved outcomes [1] and care experiences [2].
Publicly reporting performance measurement can also contribute to encouraging improvements [3], as well as system transparency and accountability [1,4]; the latter contributing to public trust and a sense of ownership of the health care system [5]. Of the varied approaches to measure performance [1], the use of Health System Performance Assessment (HSPA) is a well-established and widely-supported [6,7] means to evaluate the health system as a whole. HSPA is a country-speci c participatory process, relying on a limited number of indicators to link outcomes with system functions or strategies [8]. In the European context, the development of tools and methodologies to support performance measurement [9][10][11] and a range of country studies on the use of HSPA [12][13][14][15] have served to shift discussions from why to use an HSPA approach to focus on the what and how of the process.
Citizen engagement processes can take on varied forms: citizen panels, focus groups, citizen juries, consensus conferences, or citizen assemblies [16,17]. Importantly, these methods are distinct from more passive approaches focused on preference elicitation, such as polling and surveying. In contrast, citizen panels put emphasis on empowering citizens to actively deliberate in processes that precede policymaking [18]. This contribution is presumed to have important advantages to supporting ownership, legitimacy, and fairness in priority-setting processes, as well as building public trust and greater acceptance in decisions [5,19].
In Ireland, improving the governance, accountability, and performance of the health system was set out as a key priority in the 10-year reform programme Sláintecare 2019-2028 [20]. At the outset of the programme, key stakeholders recognized that a comprehensive performance measurement framework and management system was needed to foster accountability and capture achievements against the objectives of Sláintecare [21]. To this end, the Department of Health of Ireland (DoH) requested the support of the European Commission's Structural Reform Support Service and, in mid-2019, the project "Performance Accountability for the Irish Health System" was launched. A team of Health Care Performance Intelligence Professionals from Amsterdam University Medical Centers was selected to support this project. Early in the project, priority was given to selecting a set of domains and performance measures that re ect shared priorities of multiple stakeholders, including citizens.
While methods for engaging stakeholders are well-established in their own right [22,23], citizen participation applied to HSPA processes has yet to be embedded into development cycles [15,24]. The engagement of citizens in participatory policy processes such as the development of an HSPA framework links to the broader discussion on value-creating learning health systems [25]. Being considerate of the central role of citizens' care experiences and perceptions about the health care system can strengthen value creation and nurture public trust and a sense of ownership [5]; hence, contributing to the health care system becoming more people-centered and prioritizing care and outcomes that matter most to patients [26]. For example, the International Consortium for Health Outcomes Measurement has recognized the importance of participatory processes by engaging patients and health care professionals in developing patient-reported outcomes measures (e.g., [27]). Alike to these participatory processes, with this study we set out to apply methods for the engagement of both citizens and other stakeholders towards the development of Ireland's rst HSPA framework. This study aims to describe the method applied in the Irish context and to re ect on the added value of using a citizen panel in the co-production of an HSPA framework.

Design
This study focuses on the use of a citizen panel conducted in December 2019, acting as representation of the Irish public in the scope of developing an HSPA framework for Ireland. The citizen panel was the rst in a series of three panels towards the development of the HSPA framework, followed by panels including: 1) representatives of the DoH and Health Service Executive (HSE) -internal stakeholder panel; and 2) representatives from key professional associations and special interest groups such as the academia and patient organizations -external stakeholder panel (Fig. 1). Approximately, 30 people identi ed by the DoH and HSE participated in each of those panels. The citizen panel included 15 people of varying walks of life. Participants in the citizen panel were compensated for their travel and received a stipend for their time; they also signed an informed consent prior to engaging in the meeting and were aware that they could withdraw from participating in the session at any time.

Literature review
At the outset of the study, a targeted review of the scienti c and grey literature on HSPA was conducted to consolidate a list of potential themes (domains) for the framework being developed in Ireland. We took as a basis an international comparative study of HSPA domains applied across the World Health Organization European Region's 53 member states [12], as well as an internal review by the DoH. The list of domains (health and well-being, equity, accessibility, quality of care, social and nancial risk protection, coverage, safety, responsiveness, e ciency, effectiveness, people-centered, continuity of care, and coordination of care) and their de nitions were consolidated for use in the panels that followed.

Sample and recruitment
A target of 15 panelists were pursued based on prior experiences with the use of citizen panels in the health sphere [28]. Citizens were recruited by a Dublin-based company providing recruitment services for qualitative research. The de nition of a citizen eligible to participate in the panel was that of a lay person, resident of Ireland, who was not professionally involved with the health care system, nor a public o cial. The sampling strategy aimed to achieve diversity among participants, taking into consideration eight factors: representation of different counties; sex; age; highest level of education attained; nationality; ethnicity; health status; and religious beliefs. A recruitment scheme was developed based on the key characteristics of the population according to 2016 census data [29] (Additional le 1). The recruitment agency reached prospective panelists by phone through an established registry. A screening questionnaire was applied which informed citizens on the context of this work and assessed the diversity factors to ensure the targeted sample was achieved (Additional le 2).

Workshop
A one-day citizen panel workshop took place in December 2019. A lay language citizen brief and workshop programme -both revised by the DoH for clarity of language -were prepared and shared with panelists via email one week before the workshop (Additional le 3). Two moderators (EB and OBF) led the panel's work and one notetaker (DK) took detailed notes throughout the day. A letter of consent was completed at the outset of the workshop. The day's activities were organized in three blocks. First, the basis for convening the citizen panel, as detailed in the citizen brief, was restated to ensure this context was understood together with the aims of the panel. Second, the preferences of citizens on what information should be measured and reported on Ireland's health care system was collected through a prioritization exercise. Third, citizens were tasked with identifying priority approaches and channels whereby information on the health care system could be reported back to the public.
In the scope of the second block aforementioned, we proceeded as follows: rst, the panel moderators summarized the most commonly used domains in other countries to report on the performance of their health care systems. After the presentation of a domain, citizens were asked to write down on sticky notes what measures/topics were of concern to them. The color scheme sought to capture the degree of importance assigned to a measure/topic by a panel participant (red: extremely important; yellow: important; and green: somewhat important). When all domains were discussed, the workshop moderators grouped the sticky notes on a whiteboard, producing a heat map used to clarify priorities. At the closing of the workshop, participants were asked to complete an anonymous questionnaire that asked about: 1) the panelist's experience in taking part in the workshop; 2) how well prepared the panelist felt to take part in the panel; 3) the completion of a projective-inspired task [30] by completing the sentence "Citizen panels are…"; and 4) aspects related to the panelist's experience or comments regarding the organization of the event (Additional le 4).

Data analysis
The research team assigned the colored sticky notes produced during the workshop to the HSPA domains. Data from the prioritization exercise with the citizen panel were consolidated in a ranking chart. Thereafter, we compared the ranking of domains of the citizen panel relative to that consolidated during the internal and external stakeholder panels. These panels followed a similar structure to that of the citizen panel. The four-question exit questionnaire applied to the citizen panel participants was analyzed in full. To improve the validity of the results, we triangulated data from multiple sources [31], including notes from the panel sessions, feedback from the moderators, experience sharing and discussions among the research team, and questionnaire data. The latter was particularly valuable to synthesize the experience of citizens during the panel, using their own words.

Results
Characteristics of the participants A total of 15 participants (7 men and 8 women) participated in the citizen panel ( Table 1). The age group distribution was as follows: 18-24 (n = 2); 25-34 (n = 4); 35-44 (n = 1); 45-54 (n = 4); 55-64 (n = 1); and 65 and over years (n = 3). The highest educational level for one participant was Junior Certi cate; four participants attained a Leaving Certi cate, three a Certi cate/Diploma, six a Bachelor diploma, and one participant had a Graduate/Master's certi cation. Most participants had a paid job (n = 7), reported being in good health (n = 11) and lived in urban areas (n = 14). The nationality of most respondents was Irish (n = 13), including one of Asian descent, and two had nationalities of other European countries (Croatia and Italy) but resided in Ireland.  (Fig. 2). The domains with most measures proposed were related to peoplecenteredness (n = 47), coordination of care (n = 31), coverage (n = 29), health and well-being (n = 27), and equity (n = 24); within these, most of the measures were considered 'extremely important'.
We observed great variation in the priority assigned to each domain by the citizen panel compared to the prioritizations assigned by the internal and external stakeholder panels (Fig. 3). Speci cally, the topranking domains to the citizen panel (people-centeredness, coordination of care, and coverage) were less prioritized by the internal and external stakeholder panels; domains that were less prioritized by the citizen panel, such as accessibility, responsiveness, e ciency, and effectiveness, were of higher priority in the other panels. 'Health workforce' was suggested as a standalone domain by both the internal and external stakeholder panels, whereas in the citizen panel the discussion was brief and centered on the ine ciencies of the workforce working environment (e.g., paper workload).

Citizen panel participants' experiences
Participants described their experience during the citizen panel as enjoyable, interesting, and informative. The citizen panel was depicted as an opportunity to discuss and develop a broader understanding of the functioning of Ireland's health care system by the experiences and viewpoints of other participants, as highlighted by two of the panelists: "I am more aware now on how the health system works through others." (Panelist #15) "[Participating in the citizen panel was] Very interesting and informative. Did not realise how di cult it was for people going through the public system. Hardship on public system seems much bigger that shared across social media." (Panelist #3) All participants appeared well prepared for the panel work; this was highlighted by the participants themselves, who mentioned that the citizen brief was su cient for their preparation, as suggested by one participant: "Not an issue with pre-reading materials and guidance of facilitators." (Panelist #11) Most of the participants described that they relied on their experiences with the health care system to engage in the panel's discussions; hence, feeling con dent on their preparation for the panel work, as highlighted by a panelist: "[I felt] Very well prepared as the only thing requested was our own experience." (Panelist #9). Other participants prepared themselves differently atop of the citizen brief: "Done quite a lot preparation in the past week, including to ask others on their views of the health service and what they thought to be important to improve it." (Panelist #12) It was also noted that the use of lay language at all times during the panel session, including any written materials (e.g., citizen brief), was appreciated: "(…) [panel moderators] were very easy to follow and listen to" (Panelist #8), and thus, affected positively the participants' experiences.
In general, participants showed similar social representations of what a citizen panel entails, describing it as "[a method of] eliciting info from different ideas/experiences from a diverse group," (Panelist #11) and "(…) allowing debate to summarise and retain the opinions and ideas of the citizens into more actionable and basic concepts and thoughts," (Panelist #5), which could be "(…) very important for people to be able anonimously [sic] express their beliefs and talk about their experience in hope of a change for better." (Panelist #9)

Discussion
We described the methods of using a citizen panel in the co-production of an outcome-oriented HSPA framework for Ireland. The citizen panel prioritized more people-centeredness aspects of performance domains compared to the internal and external stakeholder panels. This prioritization by the citizen panel is aligned with an interpretation of the Irish Health Survey 2019 data, which suggested the need for reform of the health system to achieve greater value, performance, and person-centered quality of care [32]. To achieve such goals, it is key to include the citizens' views to steer the health system to better address the populations needs, expectations, and preferences.
Citizen panels are generally underutilized, despite its growing use in recent years in response to many societal challenges [17]. Factors that may be contributing to this underutilization are the limited empirical evidence concerning the effects of such panels, in addition to their time-consuming, labor-intensive, and costly nature, relative to other traditional approaches to elicit the priorities and preferences of citizens such as surveys. To our knowledge, the use of a citizen panel in the scope of the co-production of an HSPA framework is novel. The engagement of citizens early on in the co-production process shaped the processes that followed, with the restating of priorities of the citizen panel informing decision-making throughout. Following principles of inclusiveness in governance [6, 9,33] and in creating value in health care [26,34], by involving citizens we actively contributed to fostering the legitimacy, responsiveness, and accountability of the framework in safeguarding the interests of a broader range of stakeholders. This was discussed during project meetings with the International Advisory Board -a consultive group of HSPA experts from different European countries -which welcomed the inclusion of the citizen panel in the process of developing Ireland's HSPA framework. We think that including citizens in this process was possible in Ireland given the democratic culture of listening to the citizens' voice during policy-making cycles. In other countries where citizens experience less empowerment or engagement in policy-making processes, using a similar approach to develop an HSPA framework may prove (more) challenging.
In general, citizen panelists shared a similar understanding of what a citizen panel involves. This could signal an effect of the citizen brief used to communicate with participants prior to the panel, where we sought to address any information asymmetry by clarifying in lay language what a citizen panel was and presenting the key themes that served as a basis for discussions during the citizen panel workshop. We noted that most of the citizens came well prepared and were fully engaged in the discussions. This was contrary to our initial expectations, where we feared that health literacy levels in Ireland [35] and the participants' ability to think outside their personal experiences could hinder a full participation. Most panelists also considered their participation as part of their civil duty, particularly in terms of representing other fellow citizens. This was aligned with our views on the advantages of using the citizen panel in the co-production of the framework, where we considered citizen panel participants as proxies to other people alike. Hence, our focus was not on a panelist's personal experiences per se, rather about developing a collective perception of the experiences and preferences of the citizen panel as a whole.

Strengths and limitations
The key strengths of our study are the participatory and inclusive characteristics of the process of coproducing the rst HSPA framework for Ireland by including three stakeholder panels (including a citizen panel), with the advantage of comparing prioritizations among panels, and the pro le of the teams involved during the stages of developing the framework, bringing technical expertise but also having close collaboration with local experts and decision-makers. We recognize that using only one citizen panel limits any ambition to achieve representativeness of Ireland's population, and that different insights could have been generated, had we organized multiple citizen panels. However, organizing multiple panels would have not been possible due to time and cost constraints, notwithstanding, we strove to reach su cient diversity with the citizens recruited to participate in the panel, relative to the general population of Ireland. Another limitation relates to the di culty of controlling all aspects related to the process of recruiting panelists, either a database is already established from which a sampling strategy could be employed, or else commissioning the recruiting of panelists to a local agency is probably best.

Conclusions
In the context of Ireland's Sláintecare health reform, strengthening participatory citizenship and end-toend accountability on the performance of the health care system has become a topic of interest to a broader audience; hence, the intent of strengthening the voice of citizens as part of shared decisionmaking in the co-production of an HSPA framework. The engagement of citizens early on in the coproduction process of the HSPA framework permitted the restating of the panel's priorities in the processes that followed. Following principles of inclusiveness in creating value in health care, citizens' engagement in the development of an HSPA framework is a crucial contribution for realizing value-based people-centered health systems. It contributed to public trust and a sense of health system ownership of which the importance has only been enforced over the period of the COVID-19 pandemic. Although the HSPA is a country-owned process, it builds-off of international practices and experiences. Other countries that seek to strengthen value creation in their health systems via the engagement of citizens in HSPA development cycles, could draw from these experiences and learnings in Ireland. Future research could expand on how citizen panels could be further engaged in co-creating mechanisms such as HSPA to assess, monitor, and report on the performance of health care systems.

Availability of data and materials
Data sharing is not applicable to this article as no datasets were generated or analyzed during the current study.

Competing interests
The authors declare that they have no competing interests.

Funding
The participation of DI, EB, and OBF occurred within a Marie Skłodowska-Curie Innovative Training Network (HealthPros -Healthcare Performance Intelligence Professionals) that has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement Nr. 765141 (https://healthpros-h2020.eu). The participation of DK, NK, and TJ occurred within the European Commission Structural Reform Support Service project SRSS/C2019/046 -"Performance accountability for the Irish health system." The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Authors' contributions DK and NK led the study conceptualization and supervision. All authors were involved in data collection and analysis. All authors were involved in writing, revising and editing the manuscript, which nal version was read and approved by all authors.
33. WHO Regional O ce for Europe: Pathways to health system performance assessment: A manual to conducting health system performance assessment at national or sub-national level.   Citizen panel prioritization of HSPA domains with breakdown by degree of importance