{'fraction_non_alphanumeric': 0.03505306744086431, 'fraction_numerical': 0.009114273475798391, 'mean_word_length': 5.317348165965123, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Choosing interventions for hip or knee osteoarthritis - What matters to stakeholders? A mixed-methods study

Summary Objective To identify what factors drive choices among interventions for hip or knee osteoarthritis (OA), and to rank the characteristics of interventions. Methods In this phased, mixed-methods design involving cross-sectoral and multi-disciplinary stakeholders – healthcare consumers, providers, policy-makers, Māori health advocates and OA experts – we used the Nominal Group Technique in focus groups to generate data. We conducted thematic analysis of the focus group data to inform a framework of categories and sub-categories describing factors and characteristics influencing the choice of OA interventions. We then used a dual-panel, two-round e− Delphi survey to verify the framework and rank the characteristics of interventions. Results From six focus groups (n = 38 participants), 364 factors were identified and clustered into 56 themes (mean 9 themes per focus group; range 5–15). Thematic analysis revealed a framework of 3 core categories: characteristics of interventions (10 sub-categories), characteristics of consumers (10 sub-categories) and characteristics of the health system (7 sub-categories). In Delphi round 1, the framework was verified by each of two panels (n = 65, ≥80% acceptability). In round 2, two characteristics of interventions were combined, resulting in 9 characteristics (in decreasing order of importance): effectiveness, appropriateness, quality of the evidence, accessibility, harm, cost, duration, passivity, and immediacy of intervention effect. Conclusion Stakeholders make choices among interventions for hip or knee OA within a framework of characteristics of interventions, of consumers, and of the health system. We identified and ranked 9 key characteristics of interventions that stakeholders consider when choosing or recommending interventions for hip or knee OA.


Introduction
Clinical practice guidelines (CPGs) for the management of hip and knee osteoarthritis (OA) consistently recommend core management strategies of exercise, weight loss, education and self-management [1,2]. Yet the care received by people with OA is often at odds with these core strategies [3,4]. Successful implementation of interventions and health system strategies for OA are influenced by numerous factors, often involving multiple actors [5]. For example external context (e.g. health policy and infrastructure); organizational culture (e.g. culture and leadership); professional attributes (e.g. beliefs and attitudes to change) [5]; and consumers' and providers' attitudes about OA and OA interventions [6,7].
Fundamental to an intervention's successful implementation is its 'fit' within a local health system (context), which depends on the compatibility of the characteristics of interventions with stakeholders' preferences for those characteristics [5]. The preferences and expectations [8] of health care consumers and providers about characteristics such as treatment effectiveness, cost and accessibility may impact the intervention's overall suitability within a particular context. For these reasons, characteristics of interventions must be considered when developing or implementing local health system strategies, such as Models of Care (MoC) [9]. Preferences for these characteristics are often studied at one level of the health system and are less focussed. For example 'healthcare decision-making' among clinical decision-makers and policy-makers [10], or separately among patients [11]. An important knowledge gap remains concerning the characteristics multi-level stakeholders consider, and their preferences for these characteristics.
The aims of this study were to: 1) identify the factors considered by relevant stakeholders, across a range of health care settings, when choosing or recommending OA interventions and express these in a conceptual framework; and 2) define and rank the main characteristics of interventions according to the relative importance placed on them by stakeholders.

Design
In this two-phase, mixed-methods study (Fig. 1), phase 1 consisted of focus group discussions conducted using the Nominal Group Technique (NGT) [12]. The NGT data were pooled and thematically analysed to identify priority themes [12,13]. Phase 2 was a dual-panel, two-round, eDelphi study. Round 1 aimed to verify the framework and the focus of round 2 was to rank the importance of the characteristics of interventions. Our design included these validation processes: (i) independent parallel coding of NGT data with reduction into a composite framework to minimise researcher bias (ii) a dual-panel eDelphi which included an external and independent panel to verify the content validity of the Framework.
The study was conducted in New Zealand (NZ) between November 2016 and August 2017. The methods and results for phase 1 are presented according to COREQ-32 [14] criteria for reporting qualitative studies (Supplement 1). Study phase 2 follows guidance on reporting and conducting Delphi studies (CREDES) [15] and the recommendations of Diamond and colleagues (2014) [16]. Ethical approval was obtained from the University of Otago Human Ethics Committee (D16-329).

Focus groups
We sampled 5 stakeholder groups: health care consumers, providers, policy-makers, M aori health advocates and OA experts. Convenience and snowball sampling generated lists of stakeholders, who were screened against inclusion criteria adapted from a previous qualitative study of cross-sectoral experts in musculoskeletal MoC [17] (Table 1). Purposive sampling of eligible stakeholders was used to maximise the diversity of participant perspectives, to reduce the risk of limited perspectives being represented within stakeholder groups. The M aori health advocates group were included as M aori experience overall disproportionately worse health outcomes relative to the non-M aori population in Aotearoa/NZ [18]. For each of the above stakeholder groups, focus groups with 4-8 participants were formed [19].
Stakeholders from professional, government and non-government organisations and health care provider agencies were invited to participate by telephone and email. Consumers were targeted using local newspaper advertisements and recruitment flyers at local community centres. Participants from across the public and private health sectors were sampled from a NZ business directory and an online health-service database (https://healthpages.co.nz/). OA experts were initially identified using an online database (http://expertscape.com/) and reviewed by two authors (JC and JHA) with regards to their eligibility. Because of the small population of OA experts in NZ, Australian OA experts were also included on the basis that beliefs about the management of OA should not differ between the two countries given similarities of the health systems in these neighbouring countries and consistency of international clinical practice guidelines [1,2].

eDelphi survey
The dual-panel eDelphi survey was conducted using Qualtrics (Provo, USA). Panel A included phase 1 participants. Panel B, convened to concurrently assess the content validity of the Framework, comprised participants satisfying the same inclusion criteria but who were unable to take part in the focus groups.

Focus group protocol (phase 1)
In phase 1, focus groups were facilitated using the mixed-method NGT. The NGT structures group discussion in response to a question that can then be prioritised through group consensus in real-time. Focus groups generated data comprising: factors (representing the factors affecting stakeholders' choice of OA interventions), themes (factors grouped into themes, by participants), top 7 themes ranked by importance (by the participants), audio transcripts and field notes (hereinafter Fig. 1. Summary of the study design.  [36]. "NGT data"; see Fig. 1). From these data, thematic analysis resulted in a framework (hereinafter "the Framework") of categories and subcategories (term descriptors are seen in Fig. 2). Face-to-face and virtual (digital) focus groups were conducted for each stakeholder group separately using the NGT [12]. Each focus group was audio-recorded and transcribed verbatim for analysis. The question posed to participants to discuss was deliberately broad to capture all possible characteristics of interventions: "What factors do you consider when recommending or choosing one treatment option over another, for hip or knee OA?" Consumers were posed the question in the context of their own experience: "What factors do you use to choose one treatment option over another for your OA?" The first author (JC) was trained in the NGT by a researcher (CC), experienced in the method; CC facilitated the first focus group, assisted by JC, who facilitated all subsequent sessions. Two hours were allocated for each group, structured using five standard NGT steps [12]: (1)  In anticipation of the focus groups generating potentially many themes, participants were asked to identify and rank their top 7 themeson the basis that this number of characteristics is commonly used in health valuation studies employing multi-criteria decision analysis (MCDA) [20]. In the case of OA experts, participating in a virtual environment using a modified NGT, steps 1 and 2 were performed online using Qualtrics. Responses were collated and clustered offline by JC. For step 4, the preliminary clustered ideas were then presented to the OA experts, using Zoom videoconferencing (San Jose, USA) and Realtime-Board (www.realtimeboard.com).

Qualitative data analysis
For the thematic analyses we used the abductive approach [21] which builds on the general inductive approach described by Thomas [22]. From this perspective we accepted a priori that the research team's primary frame of reference (evidence-based practice and evidence-based policy-making) [23,24] would influence the thematic analyses.
First, the NGT data generated from the focus groups were organised using the software package Atlas.ti (Ver.7, Berlin, 2015).
Second, JC performed a thematic analysis of the data following the steps described by Thomas (2006) [22], and guided by the advice of McMillan and colleagues (2014) to aggregate diverse and multiple-group NGT data [13]. The iterative process of open-coding raw data began with reading all results and transcripts from the consumer and health care provider focus groups, verbatim. Preliminary codes emerging from the text resulted in the creation of categories and sub-categories (hereinafter, "themes" and "factors" refer to patterns identified in the raw data; "categories" and "sub-categories" refer to the analysts' constructions of the data). The remaining focus group transcripts were coded using this abductively-derived coding framework, until a broad thematic framework of participants' considerations emerged. Next, the analysis focussed on identifying specific NGT data that were related to the overarching aim of the study. These data were open coded by JC until a primary thematic framework of stakeholders' considerations emerged.
Thirdly, a thematic framework representing participants' choicemaking with respect to OA interventions was developed. Using the same analytical approach undertaken by JC, two researchers experienced in qualitative analysis (AMB, JHA) independently coded and analysed the transcribed NGT results and illustrative quotes to develop two independent frameworks. JC then mapped these two frameworks onto the primary framework to identify areas of concordance and discordance. In two meetings (arbitrated by PH) these 3 analysts reviewed the three frameworks and the results of the mapping exercise to agree on a consensus framework (hereinafter referred to as the Framework) consisting of categories and sub-categories.
Consistent with the a priori research question, i.e. to focus on the characteristics of interventions, the sub-categories describing the characteristics of interventions were further refined by consensus (JC, JHA, AMB) into discrete sub-categories that were generalisable, complete and non-redundantdesirable characteristics of criteria in MCDA [25,26].

eDelphi protocol (phase 2)
Phase 2 was a two-round, dual-panel online Delphi (eDelphi) survey to verify the Framework and to rank the sub-categories describing characteristics of interventions (Fig. 1). The protocol was based on a previously published process [9].
A two-round eDelphi was planned a priori, regardless of the overall level of agreement with the Framework after round 1. Two panels were planned for verification the Framework and validation of the results. Participants anonymously completed the survey and were eligible for a NZ$100 gift voucher upon completion of both rounds. Email reminders were sent after each round to increase the response rate.
The aims of round 1 were to: i) verify that no characteristics of interventions important to stakeholders were missing from the analysis (Panel A) and; ii) establish the content validity of the Framework (Panel B). Both panels provided ratings of agreement for the overall Framework (assessed using a 5-point Likert scale: strongly agree, agree, neutral, disagree, strongly disagree) and acceptability of the derived subcategories (assessed using a 3-point Likert scale: completely agree, partially agree, disagree). Free text fields allowed participants to comment on any component of the Framework, and participants were prompted to explain their answer on any disagreed (disagree/strongly disagree) component. Participants were also asked to add any essential characteristics of interventions missing from the Framework. Demographic data were captured in round 1. Round 2 focussed on ranking the importance of the characteristics of interventions. Panels A and B were pooled for this quantitative analysis.

eDelphi survey data analysis
An a priori consensus for round 1 of the eDelphi was defined as !80% complete or partial agreement for the sub-categories (3-point Likert scale: completely agree, partially agree, disagree), and !80% agree or strongly agree for the overall framework (5-point Likert scale: strongly agree, agree, neither agree or disagree, disagree, strongly disagree). The qualitative aspect of round 1 comprised content-analysis of participants' free-text comments to identify new categories or sub-categories emerging from either panel. We performed further analysis of free-text comments across all the sub-categories rated as 'disagree' from >20% of respondents (per group) to clarify the descriptions of the characteristics of interventions carried into round 2.
The aim of round 2 was to rank the characteristics of interventions by importance. Consistent with the a priori research question, from this point on the eDelphi survey was limited to the characteristics of interventions; other categories and sub-categories of the Framework were not evaluated further. We calculated the average rank of the characteristics of interventions, from most-to least-important, within stakeholder groups and for the overall sample. Level of agreement was assessed using Kendall's W within and across the groups, where 0.1, 0.3, 0.5, 0.7 are very weak, weak, moderate and strong levels of agreement respectively [27]. To ensure the final set of characteristics was representative of all groups (which differed in size), the top two ranked characteristics for each group were included in the final aggregated list of characteristics, regardless of their overall ranking, providing that adequate within-group agreement was demonstrated (Kendall's W > 0.1, p < 0.05). Data were analysed using IBM SPSS Statistics (ver.24, Armonk, NY).

Focus group discussions
Six focus group meetings were held (N ¼ 38, 61% female; n ¼ 3-11 participants per group, median 6 [IQR 3.3]), each lasting approximately 90 min (range 60-120 min). Roles, experience and number participating in each stakeholder group are summarised in Table 2. Steps 2 and 3 of the NGT produced 364 factors. After the removal of duplicates, 258 unique factors emerged across focus group discussions (mean 43 per group; median 39; range 29-62).
Step 5 produced themes ranked by importance which broadly overlapped across groups concerning consumer-related factors (e.g. clinical status and consumer preferences), themes related to efficacy and safety of intervention and health system factors (e.g. intervention accessibility and culturally appropriate practices). The OA expert group, conducted by the modified NGT that did not include step 5, produced themes echoed in the other groups. The results of the NGT exercise (the NGT data) are reported in Supplement 2, Table S1.
Initial independent thematic analysis by the researchers (JC, AMB, JHA) produced three frameworks, each with 3-10 categories, comprising 5-17 sub-categories. Two rounds of discussion leading to consensus resulted in a thematic framework of three over-arching categories: 1) characteristics of consumers (10 sub-categories), 2) characteristics of interventions (10 sub-categories), and 3) characteristics of health systems (7 sub-categories) (Fig. 3). A summary description of the Framework is presented in Table 3(sub-category descriptions and supporting illustrative quotes are in Supplement 2, Tables S2-S3).

eDelphi survey
Delphi round 1 included 65 participants (4 phase 1 participants lost to follow-up in Panel A, N ¼ 34; Panel B, N ¼ 31). In round 2, 3 participants were lost to follow-up due to personal circumstances (95% retention rate). The panels' demographic characteristics are summarised in Table 2.

eDelphi round 1 outcomes
The overall acceptability of all the categories of the Framework was confirmed by panels A (91% strongly agree or agree) and B (94% strongly agree or agree) and by the stakeholder groups (Supplement 3, Fig. 3. Thematic representation of the three over-arching categories and sub-categories, derived from the 56 themes generated by the focus discussion, that stakeholders consider in their choice of hip or knee osteoarthritis intervention.   Characteristics of the intervention Stakeholders gathered information about interventions to assess for suitability against a patient's circumstances or feasibility within a health system. The characteristics or attributes of interventions, applicable to any number of interventions, were used to compare and contrast between interventions. Key characteristics of interventions included: evidence for effectiveness and safety, consideration of the benefit and tradeoff, cost-effectiveness (to the consumer), immediacy of treatment and access to treatment. Information about the intervention options were also gathered from health professionals, peer/family experience and advertising sources (e.g. newspaper and online advertising) in the decision to select a treatment option for consumers.
Access to treatment for the consumer. The suitability of access considered in terms of an intervention's distance to nearest provider and wait time. Active versus passive intervention options. The extent to which an intervention allows a patient to self-manage their condition over the long term, versus passive care.
Cost of the intervention. The financial costs of intervention relevant to the use or provision of health care and society. Duration of the intervention effect. The ability to delay or avoid more invasive interventions later through improvements in quality of life (e.g. physical function). Evidence about the intervention appropriateness. Providing the right treatment or services for OA at the right time (e.g. surgical intervention for early-or advancedstage OA). Evidence about the intervention effect. Considered on the basis of evidence outside a clinical trial, including short-term and long-term evidence of effectiveness and change in health system and societal costs. Evidence about the intervention harms. The risk of short-and long-term side effects of intervention. Immediacy of the intervention effect. The time delay between starting and experiencing the benefits of intervention. Quality of the evidence (þviews and opinions). The extent to which one could be confident that the effects of the treatment or service described were real. Views and opinions concern the value which different people assign to different sources of evidence. For example, peer or family advice and/or personal accounts were viewed as more trustworthy for some participants than health professionals' or advertisement (e.g. newspaper, Facebook) advice, claims or recommendations about the benefits or harms of intervention. "Availability; for example, I explored physiotherapy and exercise options but I live an hour or so out of town … to access services whether it be the public system, whether it be close to; in my case travel's a factor." Consumer #6 Capacity and confidence and motivation might be different; I might put that as a different 'cause somebody might have the capacity and confidence, but they're not motivated to change … if you look at the goal-setting care plans, the way they're done, those sorts of things [factors related to self-management] would definitely be taken into consideration." Osteoarthritis expert #1 "A whole lot around cost; what the particular cost is. The difference to the lifetime cost system and also the patient's lifetime costs. The value for money. (Health Policy #5) "I made a note about this just while conversation was going on earlier about surgery and last resorts because from my point of view I would much prefer surgery to a drug medication that goes on and on and on." Consumer #7 "I guess in a sense we might say that all the evidence or whatever we might come up with on paper says that this is the right treatment, but actually for certain groups of people in society it might be the wrong treatment because it might be unfeasible because they're rural or can't afford to travel. So therefore it's the wrong treatment for that person or even that group at that time." Health Policy Maker #2 "Well, efficacy, simply does the drug work; or does the treatment work? And with more complex interventions, say we've got multidisciplinary clinics, or whatever, then you'd need effectiveness which is showing that it works within a context." Osteoarthritis expert #2 "It doesn't matter whether it's a drug that's actually prescribed by the doctor or this Arthrem or any of those other things; you are putting something into your body and things like that. Is it going to be worth it? Is it not? You're weighing this up all the time." Consumer #4 "The problem is though you see on the advertisements and things like that and these people stand up and say, 'This is the best thing since sliced bread,' and you think 'Right, okay, I'm going to give this a go.' How much are they being paid to say that and in actual fact have they had that working for them; because you've got no way of checking. All it is, is you think "Well if it's worked for that person do I spend the money and give that a go," but you have no knowledge of it." Consumer #9 Characteristics of the consumer When choosing or recommending an intervention for OA, stakeholders considered it important to match an intervention to the biopsychosocial profile of the patient. Key factors included an assessment of a patient's severity of OA, clinical comorbidity profile, preferences (inclusive of attitudes, beliefs, ability to self-manage, motivation and treatment goals) and access to interventions. Social considerations included support to/from the family and/or community responsibilities.  "I would say family support to and of as well. Like for some M aori it depends on their beliefs and their upbringing but some people would choose a spokesperson; and so everything is dealt with that spokesperson that the patient has nominated as the spokesperson. So you wouldn't necessarilyand this is really weird as a health professional that you talk to someone other than the patient, 'cause it's really the patient that you're dealing with and their experience. But to some M aori they do choose a spokesperson and so everything gets [9.10]. So let's say it's an older person; they might choose their youngest son, or they might choose the youngest daughter and that's the person that you deal with, with all cares concerning or all treatments concerning the patient. So you actually don't really deal with patient; which is different." M aori advocate #1 Characteristics of the health system Participants considered whether there was the need for the society to invest in health care for OA and, whether the interventions considered were feasible to adopt or implement from the perspective of the health system and the consumer. Factors considered by stakeholders included the burden of disease profile for OA, socio-political interest, health system capacity, access and equity, culturally appropriate services and, the cost-effectiveness of intervention to society.
Political landscapeis the disease a compelling problem? Before recommending interventions for OA, policy-makers considered if there was community and political interest in addressing the burden of OA.
Costs and benefits to society. Before recommending treatments for OA, policymakers considered if there was community and political interest in addressing the burden of OA. National health system capacity. Policy-makers also considered the capacity of a health system to support OA health care including policy direction, funding availability, evidence in support of funding or implementing a new service or intervention and workforce capacity and capability Local health system capacity. Health policy makers considered the feasibility for a local health system to provide care services for OA in terms of the costs, communication capacity among care providers, the current availability of services (public and private), access to services and scope for integration with other existing services Access and equity. Consideration of the burden imposed on the consumer to reach an intervention or service for OA was reasonable (e.g. geographical remoteness, fit with lifestyle) and, whether or not it was affordable to those who need it (fairness/equity of access) Culturally appropriate services. Whether or not a health care service for OA was culturally appropriate for diverse communities. Characteristics of the health service provider. The beliefs, experiences and preferences of the health service provider.
"We've talked about the burden of disease but actually there's a far more less palatable political reality, which is about is there a burning platform? Is there some advocacy for this? Is this both accepted and publicly accepted as being needed? And it might be quite small. So I wonder if burden of disease is only one part of is there a burning platform or public support for this … Sometimes the burning platform is because of the cost to the system. Sometimes it's because of the consumer voice, from some effective advocates, and that's all captured in the how is this framed in the public's mind or little P politically. Because there's a political element to it which might just be local community; I don't mean national politics." Health Policy #2 "A whole lot around cost; what the particular cost is. The difference to the lifetime cost system and also the patient's lifetime costs. The value for money. Health Policy #5 "… I think the question is what resources are available? What funding is available? Because you're likely to cut your service, cut your coat according to your cloth. You're likely to come up with a service that is feasible within the funding available and other resources available." Health Policy #5 " … it's actually I guess in a way saying more about the role of the District Health Board. [A DHB] plan for commissioned services in its own district, right? Once it might be a national policy or capability that we're going to provide these services in the country and then the DHBs end up providing those services in their own district according to what's available in their district to do that." Health Policy #2 (continued on next page) Tables S1-S2). Analysis of pooled panel and stakeholder-group agreement scores for each sub-category revealed consensus across all subcategories, except for the "desperation" sub-category (77% partial or complete agreement), confirming the content of 26 of the 27 subcategories (Supplement 3, Table S3). Analysis of agreement within stakeholder groups identified lack of agreement for the sub-categories (Supplement 3, Table S4): [n (%) stakeholder group] 5 (45%) OA experts for "desperation", 3 (30%) policy-makers for "views relating to the characteristics of the intervention" and 5 (22%) providers for "local health system capacity". Analysis of stakeholder's free-text comments to the overall framework did not reveal any new categories or sub-categories. The three subcategories with >20% disagreement are discussed here. Content analysis of free text comments from 4 of the 5 OA experts (1 did not leave a comment) who disagreed with the "desperation" sub-category posited that consumers may be desperate to try new treatments but did not consider it good practice; i.e. they agreed with the meaning of the subcategory per se. Likewise, the comments from three provider's who disagreed with the "local health system capacity" sub-category (2 did not leave comments) concerned local health system barriers, rather than disagreement with the sub-category. One policy-maker commented about their disagreement for the "views relating to the characteristics of the intervention" sub-category, however it was unrelated.
In summary, the eDelphi panellists agreement with the overall Framework and it's sub-categories (excepting the "desperation" subcategory) indicate strong alignment between the stakeholder-generated synthesis and the qualitative analysis, with validation provided by an independent stakeholder panel. Within the "characteristics of interventions" category, consensus was not reached by the stakeholder groups about the "views relating to the characteristics of the interventions" sub-category. Two participants' comments guided the analysis team to conclude that these views related to the evidence concerning interventionsfrom a range of sourcestherefore these two subcategories could be merged within the "quality of the evidence" subcategory for round 2. Thus, apart from this one change, no other modifications were made to the Framework, confirming its completeness and content validity, and leaving nine discrete characteristics of interventions (sub-categories) to be ranked by participants.

eDelphi round 2 outcomes
There was significant but weak agreement in the rankings across the groups (N ¼ 62; between-group W ¼ 0.333; p¼<0.00). A weak-tomoderate level of agreement in rankings was observed within stakeholder groups except for the M aori health advocacy group (N ¼ 5), which did not demonstrate adequate within-group agreement (Kendall's W ¼ 0.089, 'very weak'; p ¼ 0.893). The top two ranked characteristics of interventions across the resulting stakeholder groups were: evidence about the intervention effectiveness, evidence about the intervention appropriateness, quality of the evidence, and immediacy of treatment effect. The M aori advocacy group was the only group to rank immediacy of treatment effect among their top two characteristics; however we were unable to achieve significant agreement within this group. Our failure to observe reliable estimates in this stakeholder group was likely due to the low sample size, despite our extensive efforts, so data from the M aori health advocate group were pooled with those of the healthcare consumer group. The aggregate ranking of the characteristics of interventions, across all groups, was (in decreasing order of importance): effectiveness, appropriateness, quality of the evidence, accessibility, harm, cost, duration, passivity and immediacy of effect (Table 4).

Discussion
To the authors' knowledge, this is the first study to investigate, using a cross-sectoral and multi-disciplinary sample, what stakeholders consider when choosing or recommending interventions for hip or knee "… educating the family is important also because that patient potentially is living with them, and so they will be offering the cares in some way, shape or form; so having the family, the patient and the whole team, the MDT, all on the same page. Like M aori don't tend to -this could be generalising but in my experience M aori don't put their loved ones into a rest home; so they're at home living with the younger daughter or the niece or somebody. So having those people involved in all aspects is really helpful 'cause that will help the patient overall." M aori Advocate #2 OA. A Framework of three over-arching categories of factors considered by stakeholders were identified: characteristics of interventions, characteristics of consumers, and characteristics of health systems. We identified, and ranked by importance to stakeholders, nine characteristics of interventions.
The categories and sub-categories of the Framework developed here overlap with other conceptual frameworks [5,28], underlining the Framework's construct validity. A review [5] of the factors influencing implementation success in primary care identified themes, many of which meshed with our Framework, including: "external context", "organisation", "professional" and "characteristics of interventions". Constructs of the Consolidated Framework for Implementation Research [28] also overlapped with our Framework: "intervention characteristics", "outer setting", "inner setting" and "characteristics of individuals". Further, in a study of arthritis MoCs [29] the following concepts were also considered important: "patient self-management", "provider skills and expertise", "health care delivery", "health system", "community and public awareness" and "primary prevention". While supporting existing (generic) frameworks [5,28,29], the present Framework provides important context-relevant information about factors relevant to the New Zealand health system for selecting and delivering OA care, which have not been established previously. Establishing context-relevant factors are important in informing implementation initiatives [9].
The nine characteristics of interventions of our Framework span clinical (appropriateness, effectiveness, harms and the quality of the evidence) and health system-related characteristics (accessibility and cost of the intervention) that align with the principles of contemporary MoCs for OA; i.e. "the right care, delivered at the right time, by the right team, in the right place and with the right resources" [9,17]. Alignment of these characteristics with stakeholder preferences is important to optimise the quality of OA care delivered within a health system. The description of the characteristics of interventions as discrete sub-categories that were generalisable, complete and non-redundant makes them particularly useful for informing MCDA methods, which go beyond rank-ordering (such as in this study) to more sophisticated approaches that explicitly evaluate the characteristics quantitatively [30], which may help decision-makers identify which interventions offer the greatest value to the health system and society or, in the case of consumers, the individual. This research highlights a discordance between what influences stakeholders' choice of intervention relative to the evidence typically provided to guide decision-making (e.g. CPGs). Of the nine characteristics of interventions, 'effectiveness' and 'quality of the evidence' are routinely evaluated in the traditional evidence-based approach, and 'appropriateness' and 'harms' are often considered. The identification of these characteristics was, therefore, unsurprising. However, five other characteristics also arose, including: accessibility, cost and passivity of the intervention, immediacy and duration of the intervention effect. These additional characteristics are highly relevant to health consumers and providers, but typically lie outside the evidence-based paradigm and in healthcare provider and policy-makers' considerations in practice [10]. The GRADE approach to clinical guideline formalisation does encourage guideline committees to also consider these contextual factors when making recommendations, however their consideration remains subjective [31]. These results may therefore carry implications for guideline developers including greater transparency in decision-making. Further, a principle aim of CPGs is to provide recommendations [1,2], which are an indicator of 'appropriateness' of the intervention for a given application. However these recommendations are generally formed on the basis of the characteristics of 'effectiveness', 'quality of the evidence', and 'harms', meaning they are non-independent of the characteristic 'appropriateness'.
Matching stakeholders' preferences for characteristics of interventions with the interventions provided and delivered in a health system may be fundamental to maximising uptake, in a context where resources are limited but many choices exist. In this study for example, immediacy of the intervention effect was considered more important by the consumer and M aori health advocacy groups, whereas the other (delivery sector) groups ranked it consistently last. These findings reinforce the importance of considering context-specific preferences in efforts to translate evidence into practice and policy [5,32] that may be particularly important in the current policy landscape with calls for a national OA Model of Care and programmatic funding for OA care from the Ministry of Health [33,34].
The Framework's categories reflect the themes generated by the focus groups. Many of the themes overlapped between groups and were not mutually exclusive, highlighting the complex nature of intervention choice. The M aori and health care provider groups, in particular, Table 4 Characteristics of interventions after eDelphi round 2, in decreasing order of importance (top to bottom), for each stakeholder group. Data presented as the mean characteristic and rank by stakeholder group. generated themes specifically about culturally appropriate practices and health services which incorporated "wh anau" (family) and community needs (e.g. education about OA) that were not specifically raised by the policy-maker group. Briggs and colleagues [17] note that inclusion of family support factors into MoCs are poorly integrated, perhaps due to inadequate consumer participation in co-design. Due to low sample size, our results with respect to the M aori health advocate group's preferences for characteristics of interventions should be interpreted with caution, however these preliminary results suggest M aori have culturally moderated healthcare considerations which must be considered by healthcare providers in NZ to achieve successful implementation and reduce health disparities, and that providers recognise its importance. The NZ Ministry of Health's M aori Health Strategy [35] exemplifies active M aori participation in planning, development and delivery of appropriate health services for M aori people, reinforcing the benefits of inclusive policy-making.
Our study is not without limitations. Recruitment relied on convenience and snowball methods, which are prone to selection bias, but we filtered participants through purposive sampling which, while intended to enhance diversity, may have inadvertently biased our results towards the views and opinions held by the authors. Moreover, the sample size for each stakeholder group was modest, which may have constrained the breadth of considerations reported in this study when choosing or recommending OA interventions, and the ranked importance of the characteristics of interventions. The necessity to use a modified NGT process for the OA Expert group, conducted remotely due to their geographic dispersion, limited comparison of the NGT results with the other groups. Analysis of NGT data from multiple groups that vary in sample size and produce diverse themes is complex [13], however we included several verification and validation processes in our design to ensure validity, robustness, and generalisability of these results for the NZ population.
Intervention choice is complex and influenced by the characteristics of interventions, consumers and the health system. This finding has implications for developers of multi-level care strategies, such as MoCs. Currently NZ has no such strategy for managing OA, making the reporting of these context specific characteristics timely [33]. Future research could incorporate the nine discrete characteristics of interventions identified in the current study into MCDA, to support complex decision-making, such as those faced by guideline developers.

Author contributions
JC, AMB, PH and JHA contributed to the conception, design and primary analysis of data in the study. JC and CC contributed to preparation and data collection. JC had full access to the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. All the authors contributed to revisions of the manuscript, the interpretation of the data and critically reviewed the manuscript for publication. JHA conceived of the programme of enquiry, and is the corresponding author for this manuscript. All authors read and approved the final manuscript.

Role of the funding source
The funding source had no role in the design or conduct of the study; collection, analysis, or interpretation of the data; or writing of the report.

Declaration of Competing Interest
AMB declares travel support funding from the University of Otago <$10,000. AMB is supported by a fellowship awarded by the Australian National Health and Medical Research Council (113548).