{'fraction_non_alphanumeric': 0.024511726409914503, 'fraction_numerical': 0.011530316103223782, 'mean_word_length': 5.413229376257545, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Q-Method Evaluation of a European Health Data Analytic End User Framework

MIDAS (Meaningful Integration of Data Analytics and Services) project is developing a big data platform to facilitate the utilisation of a wide range of health and social care data to support better policy making. Our aim is to explore the use of Qmethodology as part of the evaluation of the implementation of the MIDAS project. Q-methodology is used to identify perspectives and viewpoints on a particular topic. In our case, we defined a concourse of statements relevant to project implementation and goals, by working from a logic model previously developed for the evaluation, and structured interviews with project participants. A 36-item concourse was delivered to participants, using the HTMLQ system. Analysis was done in the qmethod package. Participants had a range of professional backgrounds, and a range of roles in the project, including developers, end-users, policy staff, and health professionals. The qsort is carried out at 14 months into the project, a few months before the intended first release of the software being developed. Sixteen people took part, 6 developers, 5 managers, 2 health professionals and 3 others. Three factors (distinct perspectives) were identified in the data. These were tentatively labelled ‘Technical optimism’, ‘Enduser focus’ and ‘End-user optimism’. These loaded well onto individuals, and there were few consensus statements. Analysis of these factors loaded well onto individuals with a significant number of consensus statements identified.


Introduction
Health care like many modern activities generates large amounts of data, a proportion of which is stored, in some accessible form as usable information, but rather less of which used to guide practice, planning or policy (Murdoch & Detsky, 2013). Information systems are a key tool to support this, and assist with effective decisionmaking. The need for effective use of data is particularly critical in public health organizations, where it is required to support areas such as epidemiologic surveillance, health outcome assessment, program evaluation and performance measurement, public health planning, and policy analysis (Studnicki et al., 2008). To take appropriate actions, health policymakers require many different kinds of information. The knowledge translation literature contains many studies on information synthesis methods for producing best available evidence. However, less attention is paid to methods of disseminating epidemiological information to policymakers (Zakkar & Sedig, 2017). To satisfy this need, more flexible health data representation, analysis, querying and visualization methods (analytic software tools) are desirable (Tilahun et al., 2014).
The literature on information systems development recommends that end users should be involved in the process of IS development (ISD) (Engler, 1996). In practice, user involvement may be limited or completely absent (King, 1995). Developers are therefore forced to "design in the dark." Further, software engineering development models do not take into consideration all the dimensions of software development, in particular the organizational, economic, and human dimensions (Toffolon, 2000;Ilavarasan et al., 2003). End users differ greatly in experience, and professional background, yet visualization tools and other software platforms are designed for a single idealised end user (Ziemkiewicz et al., 2012). The effectiveness of knowledge integration in a software system determines the quality of the overall system. This knowledge gap is the commonest reason for the rejection of a software system by the intended users (Dakhli & Chouikha, 2009).
It is therefore critically important to ensure that a thorough evaluation is conducted throughout the development process to minimise the potential for software rejection. A rigorous evaluation of information systems is of great importance for policy makers and end users of the technology (Kaplan et al, 2002).
Health informatics evaluations provide an objective measurement of processes and outcomes against expectations, with the intention of identifying strengths and successes, whilst finding means of addressing and improving weaknesses or even system failures (Rigby, 2006).

Context of the Study
MIDAS project is developing a big data platform to facilitate the utilization of a wide range of health and social care data by policy makers. The platform will enable the integration of heterogeneous data sources, provide privacy-preserving analytics, forecasting tools and bespoke visualizations of actionable epidemiological data.

Study Design
Longitudinal semi structured interviews are performed at critical time points throughout the duration of the project. This involves stakeholders (lead technical developers, platform end users and policy makers) utilizing a novel parallel case study design. The data collection process was developed based on a logic model, semi structured interviews and Q sort to evaluate health analytic software acceptance of the MIDAS platform and identification of system requirement gaps at each iteration of the tools development.

Participants
Stakeholders, lead technical developers and the platform tools end users with a background in epidemiology and heath policy development (n =19) were recruited through the MIDAS project policy board.

Q Methodology
The Q method has been described as the scientific study of subjectivity (Webler et al., 2009;Watts et al., 2012). Concourse theory proposes that people form their belief and value systems within a universe of ideas, feelings, thoughts, and related referential material (Brown, 1980;Stephenson, 1986a;Stephenson, 1986b;Wingreen et al., 2009). The concourse is the universe of ideas or statements on any given topic, and a person's belief or value system with respect to the concourse manifested by how that person prioritizes the ideas and thoughts within the "universe" of the concourse. Qmethodology is the proposed means of operationalizing and analyzing a concourse, and the person's unique system of beliefs and values with reference to the concourse (Martin et al., 2015).
The primary benefit of using q-methodology is that it provides a rich and interpretive understanding of the phenomenon of interest, and minimal demands on the sample size (Brown, 1980). The application of q-methodology commences with the development of q-statements which represented the concourse, in this case technical development teams' expectations of the requirements of end users of the MIDAS platform and essential factors for successful delivery of the MIDAS project. These dimensions representing the concourse were sourced from the project delivery protocol's, a logic model developed in conjunction with the MIDAS consortium, and one to one interview with end users of the MIDAS platform, lead technical developers and policy makers.

Concourse Development
The task of concourse construction is to identify components for relevant subjects at relevant moments in relevant contexts (Kampen et al., 2014). The concourse is the population from which a representative sample of statements is to be drawn. The concourse, according to Farrimond et al. (2010), "can never be fully known but the sample of items (usually written statements) should give a workable estimate of it." The basis for development of the concourse were semi-structured research questions based on stakeholders' expectations of the project and platform tools development. The primary objective was to identify expectations of the platform tools and its utility for the purpose of assisting effective public health decision making and policy formation.
Prior to undertaking the interviews both stakeholder groups were provided with the general themes of the interview questions to assist them consider their answers in advance of the interviews process. Interview questions related to: the big data collection process, barriers to adoption of the project, and discussion of early outputs, outcomes and impacts of the MIDAS project. Stakeholders (end users, policy makers, and lead developers) were interviewed in person via conference software. Questions in subsequent interviews with stakeholders will focus on the projects overall progress, platform tool implementation, technology adoption issues and resolutions and the level of collaboration between end users, and technical development teams at each iteration of the platform tools development.
Each phase of interviews will inform the next round as a means of identifying gaps between end users' expectations of the platform, achievement of the logic model outcomes, impacts. The duration of the semi structured interviews ranged from 30-40 minutes, they were recorded with the consent of stakeholders and transcribed verbatim. On completion of each round of interviews, the stakeholders provided with a copy of their transcript for review. Development of the concourse for this study was based on interview themes, technology acceptance literature, the project deliverables for the MIDAS project as a means of guiding the development of a pool of statements (n = 97).

Concourse Refinement
The process of refining the concourse statements involved face and content validation (Valenta et al., 1997). The face validation process involved refining statements for clarity, readability and repetition. Content validity was performed by the research team and collaborators to check items of ambiguity, applicability and completeness within the context of the study. On completion of this validation process the number was reduced to (N = 36) statements.

Q Sort Ranking
The objective was to evaluate how stakeholders view the MIDAS project progress at the current iteration of the platform tools development, achieved through ranking and rating statements. Prior to undertaking the online Q sort stakeholders were provided a copy of the concourse items and instructions in advance. Appointments were scheduled separately with each participant to conduct the Q sort with a member of the study team to provide assistance to clarify statements and the ranking process procedure if required. Under the instruction of the researcher, participants were requested to reading through each statement and rank them into three columns "agree "neutral", and "disagree". On completion of ranking each set, participants were instructed to further rank their statements into the ±3, ±2, ±1 column until all of the statements were populated on the grid (refer to Figure 1). The final stage of the q sort required stakeholders to provide a brief explanation for their assignments of the "+3" and "-3" "agree" "disagree" statements and answer questions relating to their professional involvement in the MIDAS project.

Q-sort analysis
Factor analysis extractions were obtained through principal components analysis. The factor structure was simplified using varimax rotation. Composite factor scores for each statement in the Q set were determined from the defining Q sorts for each factor. Prior to factor interpretation ( Table 1) and normalized and weighted average statement scores (z score) or factor scores were calculated. Statements with a significant factor score (p < 0.05) were considered and assessment of the preliminary factor loadings, primary factors were extracted with eigenvalues (> 1.00) (Kelly et al., 2016).

Results
Each factor represents a viewpoint, held by one of more of the 16 participants, and expressed in their views towards the 36 statements [S01] to [S36] in the concourse. Three separate factors were identified, described as "Technical Optimism" "End User Focus" and "End User Optimism" explaining 47% of the total variance. The factor score arrays from participants contributed to the process used to develop an understanding of each viewpoint. Common agreement statements endorsed positively included participant agreement that effective direction is essential for successful implementation of the MIDAS platform tools (S03) and for the to generate awareness of the benefits of big data (S28) in the context of public health and for these tools to assist public health professionals (S30). Statements what were not endorsed by participants were a matter of timing, as the platform was at the early stage of development when these questions were posed. These negatively endorsed statements related to achievement of multisite collaboration (S33), strategies in place to integrate data sources for each demonstration project or the utility of the platform tools to enhance public health decision making (S20) within the six months (S14). However, at this time the majority of participants were of the opinion that there were no strategies in place to integrate gaps between data sources (S15) is concerning.

Factor 1 -Technical Optimism
Factor 1 explained 17% of the total variance for 5 out of 16 participants significantly loaded on this factor. There was strong agreement across participants that the MIDAS platform will enable end users combine datasets to develop expert knowledge systems and data models (S22). It is essential for the platform to generate awareness of the benefits of big data (S28) and technical meetings were beneficial for the early identification and provision providing solutions to issues encountered during the early stages of the platform tools development (S04).
A set of perspectives that were viewed positively but not necessarily shared with others, related to the benefit of training workshops for end users (post implementation (S06) and confident the platform tools will be sufficiently flexible to allow non-MIDAS researchers develop their own data mapping and forecasting models (S25). Since the Rovinj, Croatia commencement of the project, there is a clearer view of the scope of technical issues between technical developers and end user groups (S12).
Participants strongly disagreed with statements relating to the development of indicators (S19) and enhance public health decision making (S19) within the next six months. Participants disagreed that open source cloud tools were an essential component of the platform if (time, manpower) resources need to be reallocated (S18) or that the MIDAS project use the EU Data Portal to standardise meta-data collation techniques (S34). The following statements were viewed negatively by participants relating to the quality of technical documents (S13) gaps between data sources to achieve the required outcomes and impacts (S15) data integration and data sharing achieved this year (S16) or that the tools will be sufficiently developed the end users can provide technical development teams (S17).

Factor 2 -End User Focus
Factor 2 explained 16% of the total variance 4 out of 16 participants significantly loaded on this factor. This perspective strongly endorses the process of completing legal agreements between stakeholder groups negative impact on the pilot demonstrations progress (S02).
Participants strongly endorsed that training workshops should be underway (S06) and the process of resolving governance and consent issues (S01). Participants strongly disagreed that there are strategies in place to integrate gaps between data sources (S15) as identified in Factor 1 the platform will develop indicators (S19) red flags (S21) to identify at risk population groups, provide information (S32) and sufficiently flexible (S14) for policy makers and enhance public health decision making within six months (S20).
Interestingly participants disagreed with the utility of the platform tools utility to develop expert knowledge systems and data models (S22), work packages are on target within the agreed deliverables schedule (S08). They agree that there is a need for greater understanding between developers and end users of the scope of technical problems encountered so far (S18) the consortium should encourage newcomers to use the platform (S29) or that a secure cross EU data source integration framework with open APIs is an essential component to allow newcomers to use the tools (S26). Participants agreed less strongly there is a discrepancy between developers and end users' expectations of the final platform tools (S11).

Factor 3 -End User Optimism
Factor 3 explained 14% of the total variance 3 out of 16 participants significantly loaded on this factor. Participants endorsed effective direction from the policy board as essential to the successful implementation of the platform [S03] more strongly than the other statements.
Participants endorsed moderately strongly use the EU Data Portal to harvests the metadata of public sector information [S34] and system dynamics simulations to facilitate improved decision making [S35]. They weakly endorsed development of indicators to support effective public health and health policy decision within six months [S19] and the utility of the platform tools to provide 'red flags' identifying 'at risk' population groups to support decision simulations [S21].
They strongly reject the process of completing legal agreements slowed progress developing demonstration test platforms [S02] more strongly than factor 1, adhering to data governance, data standards, GDPR [S01]. This suggests participants are satisfied with the regulatory environment in which they work. They disagree more

ENTRENOVA 12-14, September 2019
Rovinj, Croatia strongly than others the platform tools are sufficiently flexible so that both senior policy makers and data analysts can use it effectively [S14]. At this point, in the MIDAS platform's development participants are stronger than perspective 2 in their rejection of the quality of technical documents [S13] and that the expectations of each work package are clearly defined and feasible within the agreed deliverables schedule [S08]. They also rejected statements relating to expectations of each work package clearly defined and feasible within the agreed deliverables schedule and [S17]. The demonstration tools for each work package will be sufficiently developed to allow end users to provide work package 6 with timely feedback.
Participants rejected feasibility of the MIDAS platform to generate social media campaigns to get feedback from the public relating to public health policy [S27]. They agree with factor 2 that there is a need for a greater understanding between developers and potential end users of the scope of technical problems encountered so far [S12] and that the platform should be sufficiently flexible to allow non-MIDAS researchers develop their own data modelling, forecasting and mapping algorithms and that [S25]. The MIDAS tools will support policy makers and not just analysts in adopting data driven problemsolving mind set. Adhering to data governance, data standards, GDPR, and concerns relating to consent, will slow progress implementing the MIDAS platform.

s21
The MIDAS platform will provide 'red flags' identifying 'at risk' population groups to support decision simulations.

Discussion
Lead technical developers and end users of a data analytic framework from a range of professional backgrounds participated in two rounds of semi-structured interviews.
The objective was to explore project progress and the utility of the MIDAS platform tools to meet end user (epidemiologists, policy makers) requirements from the system. The q sort was performed with these stakeholders a few months before the intended first release of the software being developed. The concourse of 36 statements was constructed based on themes identified through coding the semi structured interview transcripts working from a logic model to identify outcomes, and impacts to achieve successful completion of the project. Three factors were identified, labelled as 'Technical optimism', 'End-user focus' and 'End-user optimism'. Common agreement statements endorsed positively by participants related to effective project management generating awareness of the benefits of big data analytics were essential to ensure buy in from end users identified to work with the platform tools.
The principal findings from the first factor (perspective) "Technical optimism" indicated overall participants acknowledge the project is moving in the right direction facilitated through technical meetings and acknowledge the platform tools were in the early stage of development. At the time when the Q sort was conducted significant resources were focused on cleaning and structuring datasets for each of the four pilots. As a result, progress developing technical indicators, data integration across the project which is an ongoing process and the projects impact at that point to enhance public health development were not endorsed by participants.
The primary findings for Factor 2 -End-user focus (perspective), related to GPDR governance and consent issues negative impact on progress rolling out of the demonstration for each pilot and platform training with end users. As expected, and highlighted in Factor 1 participants did not expect the platform tools to develop indicators and enhance public health decision making within six months of administrating the Q sort. Some concerns were evident from participant's pessimistic opinion of the utility of the platform tools to develop expert knowledge systems and data models, or that the project is on track within the deliverables schedule.
The final factor 'end-user optimism' (perspective) as participants expressed positive expectations that the expected data analytic modelling and forecasting utilities of the platform could be used to generate red flags to identify at risk population cohorts from the pilot datasets. Those loading on this factor also strongly rejected the statement that GPDR compliance and ensuring the pilot datasets meet, other data governance criteria impacted on progress to complete the demonstration projects. However, technical issues were highlighted with the need for end users and technical development teams to discuss and resolve these issues. Over the past few months the number of technical meetings with lead developers and end users' groups has become more frequent in order to undertake training and user experience testing.
The strengths in this study are the use of Q methodology, in conjunction with semi structured interviews as a means of studying individual perspectives a systematic and rigorous manner, enabling statements to be quantified statistically using validated research techniques (Kelly et al., 2016). Potential limitations include the fact the MIDAS platform tools were in the early stages of development at the time the interviews were conducted. Administration time of the Q sort varied as English was not the first language of some participants, and participant fatigue cannot be ruled out. However, all participants had a good working knowledge of English.

Conclusion
Q methodology was utilised to identify perspectives of lead technical developers and end users during the development of a data analytic framework through semistructured interviews. Prioritised requirements of the system were clustered into three factors, which were namely "technical optimism" (factor 1). Indicated participants acknowledge the project is moving in the right direction in terms of meetings end users' requirements. However, the second factor "end user focus" indicated that in the early stages of the project, GPDR governance and consent issues had a negative impact on progress rolling out the demonstration for each pilot and platform training with end users. The final factor "end-user optimism" participants expressed positive expectations that the expected data analytic modelling and forecasting utilities of the platform could be used to generate red flags to identify at risk population cohorts from the pilot datasets.
Some of the previous studies utilising the Q method related to health systems software platforms focused on e health (Banna et al., 2010), health professional adaption and use of technologies in clinical practice (Ladan et al., 2018) and the definition and utility of clinical health research (Kim & Bates, 2011). The present study contributes to the available literature through the evaluation of stakeholders (technical development teams, end users) perspectives at critical time points during the MIDAS project. As the MIDAS platform tools become more advanced further insights will be captured from stakeholders using longitudinal interviews and logic models expected outputs, outcomes and impacts to create additional concourse statements.
As part of this realist evaluation framework of the platform tools development Qmethodology facilitates on understanding the viewpoints of stakeholders focusing on end users' subjective standpoints on issues affecting them. The primary outcome is to bridge the gap between end users' expectations and technical development teams acknowledging these requirements at each iteration of the platform tools development.
We are confident that the stakeholder interviews on which the concourse statements are based are valid. Even though the MIDAS platform tools were in the early stages of development when the interviews the conducted to construct the concourse statements, given that, stakeholders were requested to verify interview consent and elaborate on their interview transcripts (if required). A final Q sort will be performed with stakeholders closer to the end of the project.