{'fraction_non_alphanumeric': 0.026343643419969424, 'fraction_numerical': 0.011701752322709633, 'mean_word_length': 5.211468224981738, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
The Doctors’ Effect on Patients’ Physical Health Outcomes Beyond the Intervention: A Methodological Review

Background Previous research suggests that when a treatment is delivered, patients’ outcomes may vary systematically by medical practitioner. Objective To conduct a methodological review of studies reporting on the effect of doctors on patients’ physical health outcomes and to provide recommendations on how this effect could be measured and reported in a consistent and appropriate way. Methods The data source was 79 included studies and randomized controlled trials from a systematic review of doctors’ effects on patients’ physical health. We qualitatively assessed the studies and summarized how the doctors’ effect was measured and reported. Results The doctors’ effects on patients’ physical health outcomes were reported as fixed effects, identifying high and low outliers, or random effects, which estimate the variation in patient health outcomes due to the doctor after accounting for all available variables via the intra-class correlation coefficient. Multivariable multilevel regression is commonly used to adjust for patient risk, doctor experience and other demographics, and also to account for the clustering effect of hospitals in estimating both fixed and random effects. Conclusion This methodological review identified inconsistencies in how the doctor’s effect on patients’ physical health outcomes is measured and reported. For grading doctors from worst to best performances and estimating random effects, specific recommendations are given along with the specific data points to report.


Introduction
A fundamental question in medical research is whether medical practitioners have an effect on patients' health beyond the intervention, patient risk, and hospital variables. Previous research has revealed that when a treatment is delivered by a doctor (ie surgeon or medical physician), patient outcomes may vary systematically by medical practitioner. 1,2 It is well known that hospitals can have an influence on patients' health outcomes, with wide variation between hospitals. [3][4][5][6][7] Such outcomes include adverse events, 4 prescribing errors, 4 hospital readmission, 5,6 and mortality. [7][8][9] Comparing hospitals requires a sound methodology and reliable estimates that take into account the multiple variables involved. 8,10 In contrast to the substantial research on hospital effects, there is minimal research on the effect of doctors.
The influence of doctor-patient communication has been investigated as a "doctor effect" on patients' health outcomes, 1,11,12 including symptoms, 13,14 readmission rates in the emergency department, 13,15 health-related quality of life, 16 and improved diabetes control. 17 Research on the therapist effect in psychotherapy has shown significant effects of therapists on patient outcomes beyond the therapy technique or modality applied. 18,19 This wide variation among practitioners has been acknowledged and incorporated into the training material for psychotherapists. 20,21 In surgery, outcomes associated with procedure volume, seniority, level of experience, or doctor specialty, include mortality rate, 22 length of hospital stay, 23,24 postoperative complications, 25 and readmission. 26,27 While research on the doctors' effect in non-surgical specialties is limited, there is evidence from studies in primary care, 1,28 intensive care, 29 acute care, 30 and obstetrics, 31 where medical practitioners had an effect on patients' health outcomes.
Given the significant therapist effect in psychotherapy, and the known wide variation in patient outcomes across hospitals, but unclear effect of individual doctors on patient outcomes, we conducted a systematic review of the effect of doctors on patients' physical outcomes. We aimed to assess whether doctor effects vary with specificity, outcome and intervention. However, in conducting the review, we found substantial variation in the way a doctor effect is measured and reported, therefore making data synthesis challenging and meta-analysis impossible. This has led to the present study where we have conducted a methodological review of studies that measure and report on doctors' effect on physical patient outcomes. The focus of the methodological review is on the method of measurement of the doctors' effect as well as how it is reported. The data source for the review is the included studies from our systematic review. 32

Objective
To conduct a methodological review of studies reporting on the effect of doctors on patients' physical health outcomes and to provide recommendations on how this effect could be measured and reported in a consistent and appropriate way.

Materials and Methods Design
The present study is a methodological review where the focus is on statistical analysis and reporting. 33 The search strategy, data collection, and extraction are explained in detail in a previous report of a systematic review of the surgeons' effect on patients' physical health outcomes. 32

Search Strategy
Three databases were searched initially: PubMed, Embase, and PsycINFO; and over 10,000 publications were screened. For each of the studies identified that met the inclusion criteria, a citation analysis on Scopus was conducted to identify further eligible studies. The full search strategy and keywords can be found in the Supplementary Material.

Study Selection and Eligibility Criteria
The studies selected in the initial electronic search and the studies added through the citation analysis were independently reviewed by two researchers with a third reviewer acting as an arbitrator if required. This process resulted in 79 included studies, all of which are included in the present study. Any physical patient health-related outcome was eligible for inclusion. Studies that fulfilled any of the following criteria were excluded: (1) studies that only described a doctors' effect on particular doctor-related variables (such as specialty of doctor), (2) studies with fewer than 15 doctors, (3) cross-sectional studies, and (4) studies that mention fixed or random effects but did not list them either graphically or in numerical form.

Methodological Review
We planned to describe the methods used to estimate and report the doctors' effect on patients' physical outcomes including the statistical model used, types of confounding variables adjusted for (patient variables, hospital/institution variables, doctor variables), and the method of reporting the doctor effect.

Results
Of the 79 included studies, 62 used a multivariable multilevel regression model to estimate the doctors' effect, 72 studies included patient variables in their model, 41 studies included hospital or institution variables in the model, 60 studies included doctors' volume, and 24 studies included other doctor variables. There were two different ways that the doctors' effect was reported: fixed effects and random effects, 37,38 with 54 studies reporting fixed effects and 34 studies reporting random effects. Table 2 provides details for each included study, presenting in part the wide variety of statistical methods used.

Fixed Effects -Grading Doctors by Their Effect
Fixed effects are represented by the range of patient outcomes that doctors are responsible for after all available confounding variables have been accounted for. They are shown visually using a caterpillar plot, which ranks doctors by outcomes from lowest to highest, or a funnel plot, which shows each doctor as a dot and indicates whether doctors are outside a 95% or 99% confidence interval. For example, Papachristofi et al 39 showed caterpillar graphs with an ICC of 4.0% (surgeons) and an ICC of 0.25% (anesthetists) (Figure 1), while Kunadian et al 40 showed a funnel plot with an ICC of 6.5% (Figure 2), redone at a higher resolution by the authors (Figure 3) and the same data as a caterpillar plot ( Figure 4). Measuring fixed effects allows       Figure 2 using the same data. It is a funnel plot with each cardiologist represented by a dot with 95% and 99% confidence intervals. Cardiologists whose mortality confidence interval is above the 95% line are marked in red, those below marked in green.  identification of high and low outliers and how heterogeneously doctors perform. They also show whether variation in performance is consistent with chance or whether the variation is more significant than that. Fixed effects are calculated through "modelling fixed provider effects". 41

Random Effects -Estimating the Variation Due to the Doctor
Random effects represent a percentage of the total variation in outcomes between patients that the doctors are responsible for. They are estimated via the intra-class correlation coefficient (ICC), which is the proportion of the total variation in the patient outcome attributed to doctors. There are many different ways to describe this effect. 37 The ICCs measured and reported in the studies ranged from 0% to 47% with a median of 3%.

Discussion
This methodological review of studies that report a doctors' effect on a patient's physical outcomes has identified wide variations in how researchers measure and report a doctors' effect. However, there were 2 broad methods identified: fixed effects that allow doctors to be ranked; and random effects where the proportion of variance attributed to unexplained differences between doctors is estimated. The most common statistical model used in the analyses was a multivariable multilevel regression where the types of confounding variables adjusted for included those assessing patient risk, known doctor attributes, and, to a lesser degree, differences between hospitals or institutions. Glance et al 38 discuss in some detail three approaches of provider profiling for binary outcomes, namely conventional logistic regression, hierarchical logistic regression, and fixed-effects logistic regression. They conclude that hierarchical logistic regression is generally preferred, except in the case where providers have low case volume, where hierarchical logistic regression understates the provider effect. We agree that hierarchical logistic regression is an acceptable method for provider profiling as it allows measurement of the strength of the providers' effect on physical patient health.
This review identified substantial heterogeneity in how the percentage of the variation due to the doctors is reported. For example, Goodwin et al 42 reported the percentage of the variation for the null model as the "ICC" and the variation calculated after taking all available information into account as "partitioned variance". It is helpful to calculate the variation of the null model as, if there is negligible or no variation, there is no need to include additional levels in the analysis. In both cases, the null and adjusted models, the ICC was calculated. In contrast, Gutacker et al 43 referred to the random effect measure as the "variance partition coefficient".
A crucial element of reporting fixed effects is the calculation of the confidence intervals of each doctors' performance. Glance et al 38,44,45 provide a detailed technical discussion of the respective advantages of using fixed (grading doctors from worst to best) and random effects (calculating the percentage of outcome variation due to the doctor). One pertinent issue discussed is that the smaller the cluster is, ie the fewer patients the doctor has, the greater the shrinkage towards the mean, 46,47 reducing the calculated ICC, and leading to an underestimate of the difference in performance between doctors.

Interpreting the Doctor's Effect
The clinical importance of the findings from the studies assessed in this methodological review depends on how common the outcome assessed is and how varied the doctors' effect is among practitioners. The more common and the more varied, the more the finding is clinically important. The choice between a doctor with an above or below average effect will have implications for the patients' health outcomes at different levels of how common the outcome is and how strong the doctors' effect is. The stronger the doctors' effect and the more important the outcome, the more the choice of doctor matters for the individual. The more common the outcome is, the more the choice of doctor matters for population health. Table 3 by Baldwin et al, 21 originally from Wampold et al, 48 and augmented by Kraemer et al, 49 shows effect sizes for different ICCs. The intra-class correlation coefficient (ICC) can measure the percentage of the variation in patients' physical health outcomes due to each component of a medical interaction, 21 which is typically the patient, the doctor, the hospital, and the intervention. Table 3 shows a scenario where 50% of the patients recover from an intervention when there is no doctors' effect, ie for an ICC of zero. However, an ICC of 5.9% is reported to produce a medium-sized effect

How to Report a Doctors' Effect
If researchers wish to report a doctors' effect that has been estimated, we recommend the following: • Including "doctors" effect' or "physicians" effect' in the keywords and optionally in the title or abstract • Using multivariable multilevel regression for the analysis with adjustments for patient risk, doctor experience, hospital effects, and any other potential confounding variable • For describing fixed effects -grading doctors from worst to best, showing individual results for each doctor in a Table or a Figure  • For describing random effects, calculation of the intra-class correlation coefficient (ICC), describing the variation with 95% confidence interval and whether the outcome is a binary or continuous variable • Making the dataset used for the analysis available for other researchers to conduct their own analyses.

What to Report Observational Studies
We recommend reporting doctor effects after all available confounding variables have been taken into account, either by (a) the percentage of variation in the patient outcomes which is attributed to the doctor but unexplained by known attributes such as their experience, or (b) the ordering of doctors by their patients' physical health outcomes, or (c) ideally both.
Reporting this data offers the potential to identify both low and high outliers among doctors, as well as how much of an unexplained doctors' effect there is on patient outcomes. Table 4 lists the data points that are recommended to report. Table 5 shows a specific example of those reported data points employing the dataset used in Kunadian et al. 40   This does not concern patient-reported data recorded by the doctor, like levels of pain or mobility.

Count of doctors
Count of doctors overall. For randomized controlled trials, the count of doctors for each arm.

Count of patients or procedures
If available, both patients and procedures.

Randomized controlled trials:
In addition to the above: number of arms to the study.
If the trial is not too large, a matrix showing how many patients of each arm were served by each doctor.

Strengths and Limitations
This is the first methodological review on the reporting of doctors' effect on patient outcomes. The clarity and simplicity of how doctors' and surgeons' effects are described here and the suggested standardization of such reporting should allow meta-analysis to be conducted, allow robust identification of outliers, and make the re-analysis of much existing data feasible. However, a limitation is that, as all of the included studies were conducted in North America or Europe, it is unclear whether the findings can be generalized to other regions, particularly in developing nations.

Conclusion
A doctors' effect on patients' physical health can be measured and reported in two ways: Firstly, by calculating the percentage of variation in patients' physical health outcomes due to the doctor in the form of the intra-class correlation coefficient (ICC). Secondly, by grading doctors from best to worst patients' physical health outcomes, assigning a confidence interval to those outcomes, and reporting how many doctors' confidence intervals fall wholly above or below the overall average. Ideally, both should be reported. Were funnel plot(s) provided? Yes, provided in Kunadian as Figure 2.
Were caterpillar plots provided? Not by Kunadian et al 40 . See Figure 4 as provided by authors.
Were confidence intervals calculated? Neither the downloadable document nor Kunadian state how the confidence interval was calculated.
13. Pre-shrinkage ICC calculated through simulation Using simulated data with the same number of doctors, cases per doctor, and deaths per doctor, resulted in an average ICC of 6.48%, 95% CI (4.47%, 9.32%) after 550 simulations. Therefore, there is no substantial shrinkage at work, which is not unexpected as the mean number of cases per doctor is high at 558.

Ethical Approval
As this is a methodological review, no ethical approval was required.