{'fraction_non_alphanumeric': 0.01862055097526644, 'fraction_numerical': 0.0049064950733963405, 'mean_word_length': 5.142786561264822, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Patient engagement in healthcare: a preliminary set of measures to evaluate patient engagement in the European Reference Networks

Aim: The European Reference Networks (ERNs) provide clinicians and patients the opportunity to collaborate at EU level to improve diagnosis, care and treatment for people living with rare and complex conditions. However, building a partnership culture to systematically involve patients in ERN activities and decision-making structures is challenging, partly because the role of patient representatives and the value of this collaboration are not always understood. The objective of this project was to develop an evaluation framework to assess the impact of patient engagement in the ERNs and to provide evidence on the value of patient-clinician partnership. Methods: The evaluation was developed by EURORDIS and patient representatives involved in the ERNs (ePAG advocates) through a participatory and iterative process. The work was organised in three different phases: (1) clarify roles and identify common goals for ePAG advocates’ engagement in the ERNs; (2) define a set of measures; and (3) test the measures in three different ePAGs (European Patient Advocacy Groups). Results: The project allowed developing a common understanding among ePAG advocates of their role and goals in the ERNs and defining a patient-driven evaluation framework to assess their level of engagement in the ERNs’ activities and how effectively they were working to fulfil their role. Conclusion: Engaging with ERN clinicians to refine the framework would probably render it more relevant to the reality and priorities of the specific ERNs and more valuable as a tool to build a strong partnership culture. Such an evaluation framework could be integrated into the ERNs’ quality improvement system to ensure that the networks’ activities are driven by and remain responsive to patients’ needs.


INTRODUCTION
Patient-clinician partnerships to transform healthcare services in Europe are in their infancy, compared to other areas, such as medicine development, where there is a long tradition of patient involvement and where the role of patient representatives is recognised and well established [1] . Today, patient involvement in healthcare focuses primarily on their role as partners in their own care, with few countries systematically involving patients in decision-making structures for healthcare design and delivery [1] .
The European Reference Networks (ERNs) are virtual networks involving healthcare providers across Europe. They aim to facilitate discussion on complex or rare diseases and conditions that require highly specialised treatment and concentrated knowledge and resources [2] . The first 24 ERNs were launched in March 2017, involving more than 900 highly specialised healthcare units from over 300 hospitals in 26 EU countries. For the first time, ERNs gave clinicians and patient organisations the opportunity to co-design better healthcare services for people living with rare diseases (RDs) at an unprecedented scale. Clinicians and patient representatives from different countries involved in the networks are expected to collaborate in a wide range of activities that range from developing standards of care and quality of life measures to defining data access policies and conducting clinical research. While there are some good examples of good collaboration between ERN clinical leads and ePAG advocates (see Table 1 for a definition) [4][5][6] , there is work ahead to build a similar patient-clinician partnership culture across the 24 ERNs.
The lack of formal recognition in the EU legislation of patient representatives as ERN members left a vacuum regarding the mechanisms for patient engagement and the role of patient organisations and its representatives in these networks and the mechanisms to facilitate this collaboration. To address the first shortcoming and structure patient involvement in the ERNs, EURORDIS [7] and the rare disease patient community created in 2017 24 European Patient Advocacy Groups (ePAGs). Today, there are more than 300 ePAG advocates involved on a voluntary basis in these groups and all ERNs have at least one ePAG advocate in their board.
To address the uncertainty around the role of patient organisations and their representatives in the ERNs, EURORDIS and the ePAG Steering Committee, currently composed of 41 ePAG advocates, started to discuss in 2019 potential measures to evaluate the impact of patient engagement in the networks as a way to understand what worked, what was the value of patient-clinician partnerships and what were the areas for improvement. The loose delineation of patient representatives' role in the ERNs, overall and in each of the specific collaborative activities, emerged as a challenge to measure the impact of the collaboration between patient representatives and clinicians. The lack of clear defined goals for patient engagement it hard to evaluate the patient representatives efforts in a meaningful way [8] . This is why the first step of the process The active, meaningful and collaborative interaction between patients and clinical leads in the ERNs' activities and decisions, where the networks' activities and decision making is guided by patients' contributions as partners, recognising their specific experiences, values and expertise (definition adapted from [3] )

Measures
A set of metrics identified to assess the level and maturity of patient engagement in the ERNs in the context of an evaluation framework that interrelates each metric with a goal, intermediate outcomes and activities initiated in 2019 was to clarify the role and goals of patient representatives in the ERNs, as a basis to develop a set of measures to evaluate this impact in terms of outcomes and facilitate continuous improvement in this area.
As such, the process had an important positive externality as it served to clarify and build a common understanding of the ePAG advocates' goals (why they were engaging in the ERNs) and what were the key activities and processes that allowed them to fulfil their role (how they were engaging in the ERNs). Once this was established, EURORDIS supported the ePAG Steering Committee members to define a set of measures to assess progress towards the three main goals that they had identified as being the central reasons for engaging in the ERNs: (1) improve quality of care, diagnosis and treatment for rare diseases; (2) ensure patient-centric ERNs; and (3) ensure a good level of awareness among the patient community about ERNs and their work.
This paper presents the conclusions of this work: a patient-driven evaluation framework that can be used to assess the level of engagement of ePAG advocates in the ERN activities and how effectively they are working to fulfil their role. The framework tested in three different ePAGs to test the data collection tools and the relevance of the measures. This phase has not concluded, and therefore the results of this pilot phase are beyond the scope of this article.
As the commitment and resources of the ERNs to support meaningful patient engagement evolve, we hope to engage with the ERN clinical leads to co-design an evaluation framework to assess the quality and impact of patient engagement in the ERNs. The work described in this paper is an important first step towards cobuilding a more robust evaluation framework for patient involvement in the ERNs that could eventually be integrated in the ERNs quality improvement system.

Terminology
Throughout the article, the authors use different terms that are relevant to the European Reference Networks and the involvement of patient representatives in these Networks. Definitions for these terms are provided in Table 1.

METHODS
The ePAG Impact Assessment Framework was developed by ePAG advocates and EURORDIS through a participatory and iterative process. EURORDIS provided support with the methodological approach throughout the different phases of the process and facilitated the discussions. The process included three different phases (see Figure 1): (1) identify ePAG advocates' common goals, intermediate outcomes and activities; (2) identify measures; and (3) test the measures.

Phase 1: Identify ePAG advocates' common goals, intermediate outcomes and activities
The aim of this phase was to build a common understanding across ePAG advocates involved in different ERNs on a common set of goals and link them back to intermediate outcomes and to the activities that these patient representatives should perform to achieve the goals identified. In a face-to-face workshop with members of the ePAG Steering Committee, we articulated why they had decided to engage as patient representatives in the ERNs, what were short-term direct results that would show progress towards those goals and what activities they needed to focus on to achieve them (20 ePAG Steering Committee members involved in 14 of the 24 ERNs accepted the invitation to attend this workshop). To facilitate the discussion, participants used the logic framework developed by the UK National Health Service in England to measure the impact of health programmes [9] . The discussions focused on articulating what will change, how and why ( Step 1 of the methodology in Figure 1). Some preliminary measures were also identified (Step 2 of the methodology in Figure 1).

Phase 2: Identify measures
A smaller Task and Finish working group composed, composed by five ePAG advocates from four ERNs, was created to review the results of the workshop and identify metrics for the different activities, outcomes and goals. Taking as a basis the results of the face-to-face workshop, four calls were held by the working group members and three with the ePAG Steering Committee to reach consensus on the final set of measures (see Table 2) between October 2019 and March 2020. EURORDIS facilitated these discussions and supported the ePAG advocates on the methodological aspects.
First, the members of the working group discussed a preliminary set of 14 measures, identified by EURORDIS based on the outcomes of the workshop, prioritised and selected nine measures based on three criteria: (1) relevance of the measure in the short-medium term in the context of the work of the ERNs; (2) adequacy of the measure to measure the contribution of the ePAG; and (3) clarity of the measure. In a second step, the members of the working group discussed and rated together the selected measures in terms of their validity, reliability, feasibility and relevance. All measures were rated offline individually on a scale of 1-5, and the group later agreed on a final rating. Finally, each member ranked them according to their own opinion on the importance of each measure. Measurement methods were also discussed within the working group. The working group shared at three different points in time the results of their work with the ePAG Steering Committee to validate their ratings and collective ranking and refine the set of measures with the feedback received. Once the ePAG Steering Committee had reached a consensus on the set of nine measures, all individual ePAG groups were invited to rank the measures and discuss them with their coordinators and ERN project management teams. The feedback collected was discussed and analysed in a call with the ePAG Steering Committee to agree on a final list of eight measures, two of which were broken down into two sub-measures.

Phase 3: Test the measures
The objective of this phase was to apply the framework in several ePAGs to test its relevance and identify areas for improvement. Three groups were selected to test the framework: ERN RITA ePAG (RIPAG), ERN ITHACA ePAG and ERN ReCONNET ePAG. Each ePAG group designated two ePAG advocates to coordinate and lead the application of the framework in their respective ePAG. EURORDIS set up a working group to bring all of them together and provide support throughout where needed. The activities in this phase were organised around four tasks: (1) development of data collection tools; (2) data collection; (3)  (4) recommendations. The results of the pilot phase will be presented in another article.

RESULTS
The framework (see Table 2) includes goals, activities and intermediate outcomes, metrics and measurement methods to assess the impact of patient engagement in the ERNs: (1) Improve quality of care, diagnosis and treatment of rare diseases.
(2) Ensure patient-centric ERNs that respond to the rare disease patient community needs. Patient needs steer ERN activities and their structure.
(3) Ensure a good level of awareness among the patient community about ERNs and their work (so that all can benefit).
Initially, the ePAG advocates identified a fourth goal for their involvement in the ERNs as patient representatives: "Equity of access to quality care across Europe for rare disease patients". However, it was difficult to establish a direct link between the activities developed by the patient advocates in the ERNs and changes in equity of access to healthcare for rare disease patients. Due to this attribution challenge, the consensus was to remove this goal from the framework.

Goal 1: Improve quality of care, diagnosis and treatment of rare diseases
This goal includes metrics to assess the level of involvement of ePAG advocates in the development of ERN guidance on care, diagnosis and treatment. The main tools used by the ERNs to offer this type of guidance are clinical decision support tools (Clinical decision support tools include clinical consensus statements, care pathways, evidence-based protocols, "do's and don'ts" factsheets and quality measures) and clinical practice guidelines.
The metric "listed as co-authors of Clinical Practice Guidelines and Clinical Decision Support Tools" is complemented with additional information gathered to determine the type of involvement and input provided by ePAG advocates throughout the different stages of the process. For the ePAG advocates who led the application of the framework in their respective groups, this measure was perceived as the most relevant one due to the importance that they place on Goal 1 as well as on having their contributions formally recognised in peer-review publications. Following their suggestion, the initial definition of the measure, which read "listed as co-authors of Consensus statements and Clinical Practice Guidelines", was modified to include any type of clinician decision support tool, and not just consensus statements. Going forward, after discussing the results of all three ERNs, they suggested further exploring the possibility to formulate the measure in even broader terms; to include any type of ERN publication addressing improvements in quality of care, diagnosis and treatment in general; and to include publications where ePAG advocates' names appear in the acknowledgement section.

Goal 2: Ensure patient-centric ERNs that meet our community needs
The metrics included under this goal assess the mechanisms in place to facilitate the effective involvement of ePAG advocates in the work of the ERNs, such as registries, as well as their work to capture the patient community care needs and sharing and discussing them with clinicians. Most importantly, this set of metrics includes a subjective measure to assess the perception of patient representatives about their collaboration with clinicians in the respective ERNs.

Goal 3: Ensure a good level of awareness among the patient community about the ERNs and their work (so that all can benefit)
The measures clustered under this goal assess how the ePAG is performing its "bridge" function connecting the patient community to the ERNs and vice versa. Specifically, the measure included under this goal is broken down into two different sub-measures to assess the perception of the ePAG group around two different aspects: (1) how they engage with the patient community to help them understand how to access the ERNs' information and advice; and (2) how they engage with the patient community to collect their feedback on topics that are under discussion in the ERN.

Value derived from developing the framework
While learning and accountability were identified as the main purposes for the framework, the work done to design the framework was served a third purpose, namely to further clarify the objectives of the ePAG advocates in the ERNs and identify the core activities they needed to focus on to fulfil their role as patient representatives. The workshop held in Phase 1 helped to crystalise a common understanding across the ePAG leads from 14 different ERNs of the role and goals of patient representatives in the ERNs. It is worth noting that some participants had difficulties understanding how the outcome of this workshop was later used to identify measures and following the logic of how these linked back to the activities, intermediate outcomes and goals.
Throughout Phase 2, when the measures were being developed, some ePAG Steering Committee members also expressed reservations to use the framework for different reasons. They either did not have the time to apply it or thought it was too early to use it given the limited level of engagement of ePAG advocates in their ERN.
Another important aspect discussed by the ePAG Steering Committee in the early stages was with whom to share the results. The group agreed that, in this first stage, each ePAG would share the results only with their own ERN clinical leads and ERN coordinating teams to engage with them and discuss any shortcomings and improvements on the ERN patient engagement strategy and related activities. In addition, the ERNs have different priorities reflected in the focus of their annual plans, therefore it was not recommended to use the results to compare ERNs' performance on patient engagement, but rather to use them to share best practices where relevant.

Methodological strengths and limitations
The main strength of this framework is that patient representatives involved in different ERNs developed a common vision on the change that they want to effect as patient representatives in the networks, developed the measures and defined the measurement tools through a participatory process. Overall, this approach ensures that the aspects assessed in this framework are of relevance for the ePAG advocates directly involved in the work of the ERNs.
The main methodological weakness is the lack of meaningful involvement of ERN clinical leads in designing and implementation of the framework. Resources to facilitate their participation in this effort were limited. We invited all ePAG Steering Committee members to share and explain the framework to their ERN coordinators and ERN project managers. However, not all were able to engage with them and only two ERN coordinators provided feedback (the EpiCare and BOND ERN coordinators reviewed and ranked the measures together with their ePAG advocates). As a result, the selected measures are prone to bias, as they only capture aspects relevant to patient representatives.
Using the logic model depicted in Figure 2 helped ePAG advocates understand in a simple way what they were hoping to achieve and what they needed to do to get there. We also used a participatory process to populate the model. However, this measurement and evaluation approach also revealed to be limited in the sense that some of the activities identified could be contributing to different goals. For example, ePAG advocates' involvement in the development of outcome measures is linked to Goal 2 (patient-centric ERNs), but this activity will certainly contribute to achieve Goal 1 (improve quality of care, diagnosis and treatment). The ePAG advocates piloting this framework have been encouraged to share the results and organise feedback sessions with ERN clinicians, but the model falls short of showing the bigger picture as it cannot capture the value of collective and shared contributions to achieving a certain goal. Going forward, using an alternative evaluation methodology based on a reflexive approach to refine the framework would possibly enhance its value as a tool for shared learning and reflection and be better adapted to capture the complexity of the environment within which the ERNs operate [10,11] .
The theoretical framework behind the tool is also weak, since we did not perform a literature review to inform the development of the tool and relied on our experience as ePAG advocates and the work done by EURORDIS to assess the impact of patient engagement in medicines development [11] .
We built on the extensive knowledge and experience of EURORDIS and the ePAG Steering Committee members around the ERNs and the specific context of each ERN, but "assumptions" and "contextual factors" were not discussed in depth by the group, therefore the framework lacks specific measures rigorously evaluate the contextual factors and assumptions.
All the measures that were selected focus on the implementation and procedural aspects of patient engagement in the ERNs (e.g., whether ePAG advocates are participating in ERN working groups; whether they felt that they are collaborating effectively with the clinicians; or whether they have put in place outreach mechanisms to connect with their patient community). The literature has identified the lack of outcome measures as a limitation to present concrete evidence on the added value of patient engagement [6] .
In the current stage of development of the ERNs, patient engagement is still in its infancy, and many processes and structures to support patient engagement are lacking. This partially explains why the framework does not include any outcome measures but rather focuses on tools and processes to enable patient engagement. As such, the application of the current measures does not allow evaluating how patient engagement contributes to achieving the intermediate outcomes and goals identified in the framework. In the longer term, we expect the framework to evolve to a more outcome-oriented approach, that will eventually allow to assess how the collaboration between patients and clinicians has specifically contributed to achieve the ERN goals.
Finally, not all 24 ePAG groups were equally involved in the process, and not all ePAG Steering Committee members had the same level of understanding of what was being done and why. For example, when we asked all of them to rate the selected measures with their fellow ERN ePAG advocates in terms of their validity, reliability, relevance and feasibility, most of them were unable to do it without additional support. Therefore, only the ePAG advocates of the Task and Finish group were able to engage in this exercise.
Setting up a small working group allowed us to progress at a good pace and ensure a good understanding of the concepts and the methods, but conveying the results of the work to the larger ePAG Steering Committee group and engaging with them required more time than expected.
Evaluation tools and frameworks are increasingly being developed to assess the impact of patient engagement in research and health system decision making. A systematic review conducted in 2018 concluded that most of them featured a limited participation of patients in their design and reporting and were not scientifically robust [12] . Overall, our framework was designed with a high level of involvement of patient representatives who are active in the ERNs and have a very good understanding of the context, but further work is needed to refine it and to improve its scientific rigor.
In conclusion, conducting high-quality patient engagement in the ERNs and evaluating its impact requires a major commitment from the ERNs. Additional and sustained organisational capacities are needed to plan, conduct and evaluate patient engagement in the ERNs. To enable a fruitful patient-clinician partnership, patient organisations, ePAG advocates and clinical leads must co-build the patient partnership strategy and develop together the tools and processes to support this engagement. They must also agree on their approach to evaluating the value of this collaboration. In doing so, they should establish partnerships to collaborate with social researchers, universities and other organisations that could enhance and complement the ERNs evaluation capacities and expertise. The framework and measures described in this paper could be used as a basis to co-design an evaluation framework for patient engagement in the ERNs informed by multiple perspectives. Ideally, such a refined framework could then be integrated into the future ERN quality improvement system as a way to support a meaningful patient-clinician partnership and ensure that the networks' activities are driven by and remain responsive to patients' needs.