{'fraction_non_alphanumeric': 0.03564734849876888, 'fraction_numerical': 0.02013891440961376, 'mean_word_length': 5.29615918556224, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Measuring fidelity to behavioural support delivery for smoking cessation and its association with outcomes

Abstract Background and aims Behavioural support increases smoking cessation in clinical settings, but effect sizes differ among providers, due possibly to variations in delivery. This study evaluates a measure (‘fidelity index’) intended to capture fidelity to delivery of content‐ and interaction‐based items of a behavioural support (BS) for smoking cessation and the association of fidelity with quit rates. Methods A fidelity index for scoring the adherence and quality domains of a specific BS intervention, ‘5As for quit’, was developed by classifying the intervention components using the taxonomy of behaviour change techniques. The index was applied to code 154 BS sessions audiotaped among 18 chest clinics in Pakistan to assess their fidelity and explore reliability of coding. The association between intervention fidelity and successful quit achieved by the same providers in a previous study was explored using regression analysis. Results The index represented two domains: adherence to delivery of content‐based activities of 5As (37 items) and quality of interaction‐based activities (eight items). The intercoder reliability was good for content‐based (average Krippendorff's α = 0.80) and moderate for interaction‐based (average Krippendorff's α = 0.66) items. Approximately 70% (intraclass correlation coefficient: adherence scores = 0.72, quality scores = 0.71) of variation in BS delivery was contributed by providers, which increased to 97% (g‐coefficient: adherence scores = 0.973, quality scores = 0.974) after accounting for other sources of variation. Higher quit rates were positively associated with average quality scores [risk ratio = 2.15; 95% confidence interval (CI) = 1.43–3.24], but negatively associated with average adherence scores (risk ratio = 0.55; 95% CI = 0.40–0.77) within services. Conclusions The fidelity index is a reliable measure for quantifying intervention fidelity of delivering smoking cessation behavioural support. Recommended revisions of the fidelity index include incorporation of additional interaction‐based items, such as the relational techniques used in motivational interviewing.


INTRODUCTION
Behaviour change interventions are complex, consisting of multiple, often interacting features [1]. Behavioural support, a promising intervention for smoking cessation in low resource settings, consists of advice, discussion, encouragement and activities designed to change smoking behaviours [2,3]. Behavioural support, if delivered as planned, on average produces expected cessation outcomes [4,5]. For inferences to be drawn from effectiveness studies, participant characteristics, intervention delivery, competence of the provider and the interaction between provider and patient need to be measured and their influence estimated [6].
In practice, behavioural interventions are often delivered inconsistently [7,8], resulting in variations in outcomes [9][10][11]. For example, a behavioural support (BS) intervention was offered to smokers attending chest clinics in Pakistan: Action to Stop Smoking In Suspected Tuberculosis (ASSIST), and was found effective [relative risk 8.5, 95% confidence interval (CI) = 3.7-19.6)] [12]. However, the wide variation in quit rates among chest clinics (7-70%) and a high intraclass correlation coefficient (ICC = 0.28), indicated the strong influence of clinics in determining the success of the intervention. Context, factors extrinsic to the intervention that tend to hinder or bolster its effect [13] including training and technical support to the providers, is likely to influence outcomes [14]. The competences of the providers, their motivation and intention to provide support, the efficiency of the system and organizational structure stretched practice time for cessation provision and the providers' own smoking status; all supplemented the complexity of the BS delivered in ASSIST [15].
Intervention fidelity helps to attribute effects more accurately by explaining variation and increases confidence in interpretation of outcomes [7]. For example, where there are no fidelity assessments, effects could be wrongly attributed to the intervention itself rather than a difference in provider competence [6]. Equally, the potential emergence of an intervention effect may be masked by variations in the extent and quality of its delivery [6]. Conceptual models of fidelity differentiate between fidelity of design, training, delivery, receipt and enactment [13,16]. Fidelity of delivery is the degree to which an intervention is actually implemented as intended to the patient by the provider [17] and encompasses two main concepts: what is delivered and how it is delivered [18]. Fidelity to delivery of BS for smoking cessation was the mainstay of this study, and thus the term 'fidelity' in this paper refers broadly to the element of delivery. Measuring fidelity could document differences in delivery practices [19] and compare the implemented intervention components to the theory base by linking with outcomes [20,21].
It is feasible to reliably assess the fidelity of behaviour change interventions in clinical practice [22]. However, there is limited evidence regarding methods for quantifying fidelity within complex behaviour change interventions [23]. A method to measure fidelity to BS delivery requires capturing what is delivered: both its content-and interaction-based activities [13,16] and quantifying how these are delivered [24,25]. These can be conceptualized as compositional/structural features forming items of the index and functional/process features forming the anchors to quantify each item. While studies have previously developed methods to characterize the content of BS using the behaviour change techniques (BCTs) taxonomies [26][27][28][29] and BCT associations with outcomes [30], these limit ratings to the delivery process and their predictive value for outcomes. The extent to which fidelity of delivery of a BS intervention influences smoking cessation outcomes remains unclear.
The purpose of this study was to develop a BCT-linked, reliable method of rating intervention delivery ('fidelity index') and determine fidelity's association with smoking cessation outcomes. We report how we developed this index using BCT taxonomy, evaluated its psychometric properties and explored associations with quit rates.

METHODS
The study was conducted in routine tuberculosis (TB) services in Pakistan, involving the same sites as the ASSIST trial [12] and the health workers trained to deliver BS for smoking cessation. Ethics approval was obtained from the Health Sciences Research Governance Committee (HSRGC) at the University of York and the National Bioethics Committee at Pakistan Medical Research Council (PMRC).

STUDY 1: DEVELOPMENT OF THE FIDELITY INDEX
Quantifying fidelity was conceptualized using measurement methods in the psychotherapeutic [31], health and education literature [21,32,33]. First, items were generated, then response scale options for each item were constructed, and finally experts carried out the content validation of the fidelity index.
The logic model for measuring fidelity and the conceptualization of its compositional and functional features are shown in the Supporting information, Figs S1 and S2. BS in ASSIST study was structured using the 5As (Ask, Advise, Assess, Assist, Arrange) to quit model [34] (Supporting information, Table S1). A confirmatory approach [31] to critical components technique was used to generate index items (Supporting information, Table S2) by identifying intervention activities that were measurable [32]. Activities were mapped using the smoking cessation BCTs [35], which provided the coding framework for the intended BS delivery. BCTs with a focus on specific behaviour (address motivation, maximize self-regulation) were mapped to the content-based items, which were expected to constitute a score representing the adherence to the BS (A score). Those with a focus on general aspects of interaction (delivery of intervention and general communication) specified the interaction-based items, expected to represent the quality of interaction (Q score) (Supporting information, Table S3).
To detect meaningful variation in delivery rather than the mere presence or absence of delivered items [36], a three-point ordinal response scale-fully, partially or not implemented-was chosen per item, such that the sum across item scores would provide a fidelity score reflecting the degree of implementation [37,38]. The response scale of each item used behavioural anchors that were operationally defined using the interaction style specified in the BS training manual (Supporting information, Table S4).
The items of the fidelity index were evaluated in a small Delphi study, which did not lead to any changes to the index (details in Supporting information, Appendix S1).

Design and settings
The index was validated using an observational study in chest clinics in the Jhang and Sargodha districts of Pakistan in 2014, the sites of the ASSIST trial.

Participants
Providers were health workers responsible for dispensing medicines and recording patient data at the chest clinics. All those who delivered smoking cessation in the ASSIST trial were eligible to participate. Those who consented were offered a refresher training on BS for smoking cessation; each chest clinic had one provider.
Adult TB patients attending the chest clinics and smoking on a daily basis, except those who required hospitalization, were eligible to participate upon consent.
The number of providers (n = 18) was set by the number of participating chest clinics from ASSIST and was considered sufficiently diverse for the main analysis focusing on inter-coder reliability of the individual items. The sample size for individual items was evaluated via Krippendorff 's α [39]. As the distribution of this statistic is not known [39], determining a minimal sample size for a specific level of accuracy is not directly possible. We evaluated the impact of different sample sizes (n = 50-250) on the breadth of its bootstrapped confidence interval for scenarios with α ≈ 0.70 and low as well as average fidelity ratings across three coders. The results indicated that the breadth of the index's confidence interval shrank substantially from n = 50 (~0.26) to n = 100 (~0. 15), but that the gains were less substantial beyond that (e.g. n = 200, interval breadth~0.13). We therefore aimed for at least n = 100 observations.

Measures
Three bilingual coders with a graduate degree in social or health sciences, trained on reliably specifying BS [40], coded sessions independently by listening to audiotapes. Disagreements on item ratings were resolved by discussion and a fourth (consensus) index filled for each item on agreed scores.

Analyses
The scores were summarized both for each item using mean, median and standard deviations and for each domain of the index, i.e. adherence (Ascore) and quality (Qscore), using the mean and 95% CIs supplied by the provider.
Psychometric properties of the fidelity index were assessed using three approaches: inter-coder reliability, principal components analysis (PCA) and via a generalizability study (G-study) [41]. Inter-coder reliability and PCA assessed the item scores, while the G-study assessed the domain scores. All analyses were conducted in SAS version 9.4 (Cary, NC, USA; syntax in Supporting information, Table S5).
Inter-coder reliability of the item scores was computed using Krippendorff's alpha [42], and interpreted as poor for values of α < 0.67, moderate between 0.67 and 0.80 and good for α > 0.80 [43]. To identify items that reliably loaded across all coders, PCA with promax rotation was employed [44]. Adherence was further analysed to discriminate dimensions of the content-based items. The G-study assessed the reliability for measuring provider differences in intervention fidelity under generalizability theory. It involved three sequential steps: identification of important sources of variation, variance partitioning and computation of coefficient of generalizability (g), detailed in the Supporting information, Appendix S2.
Sensitivity analyses included: (i) estimation of the crude ICC and (ii) modelling district as a fixed effect to test for systematic differences in fidelity (Supporting information, Appendix S2).

Design
The fidelity scores (study conducted in 2014) for both adherence and quality were linked with quit rates achieved by the same providers in the ASSIST trial, conducted in 2010 [12]. Patient data were aggregated at service-level for both studies.

Measures
Outcome involved self-reported 6-month continuous abstinence, verified by carbon monoxide measurement of 9 parts per million (p.p.m.) or less [45] for 1299 patients from the ASSIST trial [12]. The proportion of patients who quit by providers (n = 22) were used in analysis.
The predictor variable involved standardized adherence and quality scores aggregated by provider.
Contextual variables (from both studies) used provider practice scores based on the self-record checklist of selected intervention components. The purpose was to establish consistency in providers' practice despite the time interval between ASSIST trial and this study (detailed in Supporting information, Appendix S3).

Analysis
Intervention fidelity was described by the percentage of sessions that fully, partially or did not implement BS activities. Patient characteristics for both studies are provided in Supporting information, Table S6. Kendall's W statistic measured the pairwise concordance in providers' practice over time.
Binomial regression was used to estimate the association between average fidelity scores and quit rates, within a service (SAS PROC GENMOD). Risk ratios (RR) and 95% CIs were reported. Both fidelity scores were centred for use in the interaction term.

RESULTS
The fidelity index (Supporting information, Table S7) consisted of 45 items overall, 37 for measuring adherence: content-based items and eight for measuring quality: interaction-based items.

Study 1. Psychometric validation of the fidelity index
Between February and May 2014, 180 patients were enrolled into the study and received BS for smoking cessation at 19 chest clinics (Fig. 1). Three coders scored 154 audiotaped BS sessions for fidelity providing 462 datapoints.

Description of item and domain scores and inter-coder reliability
Items 10, 12, 39, 40 and 44 showed very little variance, exhibiting floor and ceiling effects (Supporting information, Table S8). For items 10 and 12 the average score approached the maximum possible value of 2 (ceiling effect) as the majority of the providers fully implemented them. For items 39, 40 and 44 a floor effect was observed, as the majority of the providers did not implement them.  (Table 1). The overall mean Krippendorff 's α was 0.80 for content-based items and 0.66 for interaction-based items.

PCA
Fifty-two per cent of the variance for the full index was explained by a single component. Items 10, 12, 15, 16, 37, 39 and 40 did not load well (threshold < 0.4) across the coders. When analysing the items from the two fidelity domains separately, a single component explained 60 and 91% of the variance, respectively. Content-based items (10, 12, 15, 16 and 37) showed loadings below the threshold of 0.4; the interaction-based item (39) also did not load well (Supporting information, Tables S9, S10 and S11).
For adherence domain (Table 3), four components showed eigenvalues >1 (Supporting information, Table   S12) in addition to the first component that explained 60% of variance. These components roughly map to the five aspects of the BS intervention: assessments of dependency, providing information about the consequences of smoking and cessation, preparing for the quit attempt, managing triggers to smoke and managing nicotine withdrawals. Items 1 and 37 did not load well. Overall, three items (10, 12 and 15) did not load well in spite of being delivered.

G-study
The g-coefficients were 0.973 for adherence and 0.974 for quality, showing that the providers, after accounting for other sources of variation (Supporting information, Appendix S2), contributed approximately 97% of the variation in BS delivery. The crude ICC for provider differences in adherence scores was 0.72 and in quality scores was 0.71, indicating that providers contributed approximately 70% of the variation in intervention delivery when other sources were not adjusted. The inclusion of district in the model as a fixed-effect neither correlated significantly with the adherence scores (β = À7.21, F (df = 1, 272) = 0.94, P = 0.33) nor the quality scores (β = À0.29, F (d.f. = 1, 272 = 0.04, P = 0.84).
The findings of the psychometric analysis do not suggest removing items from the fidelity index, as the core problem was largely lack of variance in the sample. Overall, fidelity and item scores showed variability across providers, indicating the applicability of the coding system (i.e. few ceiling or floor effects; services not bunched together clumping in one limited score range). Kripendorff 's alpha and gcoefficients estimates confirmed that the coding structure of the index items reliably captured variation in providers intervention delivery.

Study 2. Association of fidelity with outcomes
Relative to content-based items, interaction-based items of the BS intervention were poorly implemented (fully implemented in < 10% and partially in 10-20% of the sessions) (Fig. 2).
The degree of similarity in the providers' practice between the current study and ASSIST was found to be moderate for management of patient triggers (Kendall's W: 0.69, P = 0.06) and high for withdrawal symptoms (Kendall's W: 0.75, P = 0.03).
The binomial regression (model 1, Table 4) indicates that an increase in the quality scores of the provider led to a 50% increase in the quit rates (P < 0.05). An increase in the adherence scores of the provider led to a 24% decrease in the quit rates (P = 0.055). When including both variables in the same model (model 2, Table 4), the association of adherence scores with quit rates became statistically significant, although remaining negative, and the association of quality scores with quit rates became stronger (model 2, Table 4). A negative interaction was found between adherence and quality scores (P < 0.0001) (model 3, Table 4).

DISCUSSION
The BCT taxonomy for smoking cessation was used to code the activities of the ASSIST 5As intervention into contentand interaction-based items. The derived index was applied to BS sessions recorded in chest clinics in Pakistan to measure fidelity and explore reliability. Comprising 45 items, 37 on adherence (content-based) and eight on quality (interaction-based), the fidelity index was found reliable for coding BS despite varying coder skills. Items on setting a quit date, offering an information leaflet, explaining treatment expectations and providing reassurance seemed valid, but were not fully assessed due to low variability across the sample. Linking average fidelity scores for providers with successful quit provided useful insights into the delivery of BS for smoking cessation. Quantifying BCTs for the quality domain that required tailoring to patient needs, using an anchored scale, might be an over-simplification of a somewhat complex process. Most frequently delivered BCTs in the study included information-gathering and addressing motivation (e.g. providing information about the harms of smoking and assessments of nicotine dependence). Least implemented were maximizing self-regulatory capacity, communication and delivery of intervention (e.g. management of triggers and withdrawals, building rapport, providing reassurance), as reported by the fidelity study in England's Stop Smoking Services [7]. Some of the least implemented BCTs in our study, such as emphasizing choice, confirming client decisions, advising on changing routine and facilitating relapse prevention, might have been difficult to capture, being more subjective than the contentbased activities, or it might reflect provider competence, motivation or patient preferences. These phenomena need to be explored in future research.
Given the interpersonal nature of counselling, it is unsurprising that BS effectiveness was strongly influenced by interaction with providers [24]. A recent meta-analysis of intervention fidelity for motivational interviewing [46] found that interventions with higher implementation of interpersonal BCTs produced larger effects on behaviour change. The association of successful quit with better quality scores could also reflect a non-specific effect due to higher patient satisfaction. The finding that adherence was negatively associated with successful quit, even after adjusting for the quality, appears counter-intuitive [5]. However, complete adherence to intervention ingredients as a desirable goal is contested [7,16]. High and moderate quit rates showed similar levels of adherence across a sample of English Stop Smoking Services [7]. Furthermore, the Item 1 À0.02 À0.11 À0.02 À0.09 0.05 À0.03 À0.04 À0.15 À0.09 À0.14 À0.06 À0.04 À0.04 À0.02 (Continues) service with lower adherence (56%) was potentially more effective in achieving higher patient quit compared to services with higher adherence (69%). Delivering complex interventions possibly requires providers to manage too many dimensions at the same time, making the task longer and tedious, and so less effective if fully implemented [47,48]. More simultaneously delivered behaviour change elements could also potentially make it complex for the patients to understand and engage in. We should consider simplifying the content of smoking cessation interventions and shift focus to the interpersonal aspects of the patientprovider interaction as 'more is sometimes less' in health behaviour change [49].
A fidelity index offers numerous applications for smoking cessation practice. Compared to practice manuals, it can make the content of time-limited interventions more focused if used as a self-recording checklist by the providers [7]. It could help to standardize approaches to reduce wide variations in implementation of active ingredients and reduce failure to achieve the original effectiveness of the interventions [50]. In addition to the content-based items, using the quality domain on interaction-based items, the index can facilitate identification of specific skills and competence, provider training needs. Local practices could link implemented ingredients with quit outcomes and recommend adaptations to the services for their respective context by studying trends over time. The development of this fidelity index focused on provider practice and attitude in delivering BS, relying heavily on BCT taxonomy. This may have missed the wider psychological processes of behaviour change not covered by BCT taxonomies. Incorporation of additional interaction-based items in the index, such as the relational techniques used in motivational interviewing [24], might improve the operational value of the fidelity index. There is potential for developing shorter versions of the fidelity index and for optimized intervention delivery if the active ingredients can independently predict quit outcomes. Studies using multi-phase optimization strategy and sequential multiple-assignment randomized trials could move the field forward in providing such mechanistic evidence [51][52][53].
The study had several limitations. The use of audiotaping meant that important aspects of non-verbal interaction will have been missed. Capturing both verbal and non-verbal interaction using conversation analysis [54] of videotaped BS sessions should be considered. Nevertheless, in this study, direct observation of patient-provider sessions via audio recordings was considered less intrusive than video for capturing the key elements of the intervention delivery process [7,40].
The retrospective linking of provider fidelity scores with patient quit rates from the past raises questions about causality. The practice scores, used to test the assumption that providers' BS delivery practice was similar in the gap between the two studies, were self-recorded and might be subject to reporting bias. Such retrospective linkage can introduce bias in the study estimates, especially when there is state dependence in respondent's choices [55]. True state dependence, where behaviour relevant to future choices is altered because of experiencing an event, is unlikely in the case of providers completing the self-record checklist in this study, as they did not receive any behaviour change training between the two studies.
The study had to consider data at the provider-rather than patient-level, thus reducing the statistical power of Figure 2 Smoking cessation behavioural support intervention fidelity the analysis. Although the implementation pattern of BCTs indicated that the delivery of smoking cessation BS in this study was similar to that of stop-smoking advisers in the United Kingdom, generalizability of the findings from this study is still limited. The fidelity index represented a specific smoking cessation intervention delivered in chest clinics of Pakistan. Replication of the methods used in this study and application of different versions of the fidelity index in diverse settings is required.

CONCLUSION
The fidelity index is a reliable measure of the delivery of behaviour change features of a smoking cessation intervention in clinical practice. It provides a method to score intervention fidelity for individual ingredients, enabling its use as an intermediate variable to explain the intervention-outcome relationship. Recommended revisions of the fidelity index include incorporation of additional interaction-based items, such as the relational techniques used in motivational interviewing.

Author contributions
O.D. conceptualized the research idea, conducted the study, performed statistical analyses and drafted the manuscript [56]. K.S. and T.A.S. contributed to conceptualizing and refining the research idea, interpretation of statistical analysis and discussion of findings. J.R.B. contributed to research design of the psychometric elements of the study, selection of statistical tests, interpretation and discussion of findings. F.L. provided advice for fidelity coding and support for training of the coders. All authors provided critical revisions and approved the final manuscript.

Declaration of interests
K.S. received a research grant from Pfizer (2015-17) to study the effect of varenicline on waterpipe smoking cessation.

Supporting Information
Additional supporting information may be found online in the Supporting Information section at the end of the article.

Figure S1
Logic model for measuring intervention fidelity. Figure S2 Concept of compositional features versus functional features to form items and response options of the fidelity index. Table S1 Behavioural Support intervention provided in AS-SIST study.