{'fraction_non_alphanumeric': 0.027938950554994953, 'fraction_numerical': 0.013559535822401615, 'mean_word_length': 4.810553316233053, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Inter-unit reliability for quality measure testing

Facility-specific quality measures are commonly used to monitor dialysis facilities. To successfully develop, test and validate quality measures, a subset of facilities are often recruited for preliminary evaluations. To ensure that the facility-specific measures will achieve a desirable precision, it is often of interest to determine a minimum number of facilities that should be recruited. To achieve this, we propose a method based on the inter-unit reliability (IUR), which is commonly used to assess quality measures. Accurate estimates of the IUR are important to ensure that the quality measure will achieve a desirable precision. We first review existing methods of estimating the IUR for quality measures that are based on sample averages. We then generalize the IUR estimations to more complicated standardized measures. In particular, the confidence intervals of the IUR are calculated, with the width of this confidence interval measuring the precision of the estimate of the IUR. To assess the performance of the estimated IUR with various numbers of facilities, a simulation study is conducted. The IURs are then computed to develop and implement a quality measure that is used to guard against high ultrafiltration rates for adult dialysis patient with End-Stage Renal Disease. The estimated values are helpful to determine a minimum number of facilities that should be recruited in the measure testing process.


INTRODUCTION
Multi-provider (e.g. hospital, transplant center or dialysis facility) data arise very often in biomedical studies. In order to identify extreme (excellent or poor) performance, providerspecific clinical outcomes are routinely monitored. [1][2][3] The motivating example is to develop and implement quality measures that will be used to guard against high ultrafiltration (i.e., rapid fluid removal) rates for adult dialysis patient with End-Stage Renal Disease (ESRD). ESRD is one of the most deadly and costly diseases in the United States. As of December 31, 2015, there were 703,243 prevalent cases of ESRD in the United States, which represents an increase of 80% since 2000. [4] Hemodialysis (HD) is the most commonly used treatment option for patients with ESRD. However, despite the majority of dialysis patients achieving urea removal, the mortality rate among HD patients is substantially higher than the general population. [4] Existing literature suggests that higher ultrafiltration rate (UFR) is an important predictor of mortality. [5][6][7] The UFR measures the rapidity with which fluid is removed at dialysis per unit (kg) body weight in unit (hour) time. The UFR is under control of the dialysis facilities. Rapid UFR can lead to higher frequency of intradialytic hypotension, which occurs at high frequency and has been associated with higher mortality. Phenomena, such as myocardial stunning, [8,9] could result if large volumes of fluid are removed rapidly during each dialysis session.
This report is intended to develop a safety measure to guard against high UFR. To successfully develop such quality measures for dialysis facilities, measure testing is typically conducted. [2,3] However, due to access feasibility and funding limitations in the measure testing process, it is not always possible to enroll all (more than 5,000) dialysis facilities across the United States. As a result, a subset of facilities are often selected for preliminary evaluations. To determine a minimum number of facilities to ensure a desirable precision, accurate estimates of the inter-unit reliability (IUR) are important.
Statistically, the IUR of a facility-specific measure is defined as the ratio of the between-facility variance to the total variation, [10,11] where the total variation equals the sum of the between-facility variance and the within-facility variance. IUR values range between 0 and 1. A small IUR indicates that most of the variation in the facility-specific quality measure can be regarded as random noise. A large IUR indicates that most of the variation is due to real differences between facilities. Accurate estimates of the IUR are important to ensure that the quality measure will achieve a desirable precision.
The report continues as follows: Section 2 introduces the motivating quality measure and describes the methods used to estimate the IUR. Finite-sample properties are examined in Section 3 through simulation studies. Section 4 summarizes the analysis results on the motivating example data. We conclude with a discussion in Section 5.

Motivating example
Our motivating example for quality measures is to evaluate percentage of adult kidney hemodialysis patients with a monthly ultrafiltration rate (UFR) greater than 13. Higher dialytic ultrafiltration rates have been shown to be associated with higher all-cause and cardiovascular mortality. [7] The UFR is calculated based on the weight gain after the dialysis session and the delivered session time. It is well known that hemodialysis patients have high rates of morbidity and mortality that may be related to rapid ultrafiltration. Thus, a quality measure based on the ultrafiltration rate could encourage dialysis facilities to consider prolonging treatment time and ensure stable dialysis sessions.

IUR for UFR without risk adjustment
Methods for estimating the IUR vary by whether adjustment for risk factors are needed. We first provide a brief review of IUR for UFR without adjustment for risk factors. In next subsection, we then generalize the IUR for more complicated risk-adjusted quality measures.
We first consider a logistic regression model without adjustment of risk factor. Let Y ij denote the observed outcome (e.g. indicator for whether UFR greater 13) for patient j in facility i, where j = 1, . . . , n i and i = 1, . . . , F , with n i being the number of patients in facility i and F being the number of facilities. Here Y ij = 1 if the UFR rate is greater 13, and Y ij = 0 otherwise. The corresponding logistic regression is of the form (see Equation 1): where µ is the intercept for the population norm, is the facility effect. In this case, the facility-specific measure is defined as the sample average of observed outcomes across facility, e.g., To compute the IUR for the facility-specific measure, a key ingredient is to decompose the variance of the measure into the between and within-facility variations. To achieve this, we adopt the following variance decomposition (see Equation  2): Applying a first-order Taylor expansion, the following approximations can be computed (see Equation 3): where p is the sample average of Y ij in the overall study sample. The corresponding IUR is given by (see Equation  4): We note that a similar deviation was considered by Yelland et al. [12] to estimate the intra-class correlation coefficients and calculate appropriate sample size for clustered clinical trials. There the number of clusters was fixed and the main goal of the sample size calculation was to determine an appropriate number of patients to be enrolled. In contrast, in our motivating setting, the main goal is to determine an appropriate number of facilities.

IUR for UFR with risk adjustment
In practice, patient characteristics may vary across facilities and a large number of high-risk patients can make a facility's outcomes appear substandard. To account for the covari-ance imbalance, a risk-adjusted logistic regression can be implemented (see Equation 5): where µ is an intercept for the population average, X ij is a vector of covariates, β is a vector of regression parameters, Under this logistic model, the standardized ratio (SR) is defined as (see Equation 6): where the numerator ni j=1 Y ij is the observed number of events in facility i, and the denominator ni j=1p ij is the corresponding expected number, with (see Equation 7): Thus, the denominator is the sum of the estimated probabilities within each facility under a population norm for the facility effect, which is estimated fromμ.
This standardized ratio reflects the extent to which the facility under evaluation has a higher or lower rate of events (e.g. UFR greater than 13) than the overall population average. A standardized ratio lower than 1 indicates that the facility's observed number of events is less than expected based on the national norm. A ratio greater than 1 indicates that the facility has a number of observed events higher than expected.
To compute the IUR of the standardized ratio, we exploit the large-scale structure of the data, which allowsμ andβ to be estimated precisely when the sample size is large. Thus, we adopt similar techniques as in He et al. [2] and treatμ andβ as given without losing precision. Using the variance decomposition (see Equation 8): and applying a first-order Taylor expansion, the approximate marginal variance of SR i is given by (see Equation 9): The facility-specific IUR can be calculated as (see Equation  10): where n * i = ni j=1p ij (1 −p ij ) is the effective sample size. The overall IUR across facilities can be computed as (see Equation 11): 3. SIMULATION

IUR for UFR without risk adjustment
We first consider the setting without risk-adjustment. We simulate the outcomes from the logistic model defined as in Equation 1 with µ = 0.5 and α i ∼ N (0, 0.3 2 ). We vary the numbers of facilities from 20 to 600. The numbers of patients within each facility are generated such that one third of facilities has 25, 50 and 100 patients, respectively. Figure  1 shows the average estimated IUR based on 100 simulation iterations and the confidence intervals. Clearly, the results indicate that small numbers of facilities are not sufficient for reliable estimation of IUR.

IUR for UFR with risk adjustment
To mimic scenarios with risk adjustment, we simulate a setting with the outcome Y ij generated from a multivariate logistic model with probability (see Equation 12): (12) where α i ∼ N (0, 0.1 2 ), and X ij1 and X ij2 are generated Published by Sciedu Press from independent standard normal distributions. We let the number of facilities vary from 20 to 600. The number of observations in each facility ranges from 120, 140, . . . , 300.
We repeat the simulation 100 times. The average IUR and its empirical confidence interval are shown in Figure 2. Similar to the results in Section 3.1, a relatively large number of facilities is needed for reliable estimation of the IUR.

DATA APPLICATION
The goal of this section is to determine the minimum number of recruited facilities to ensure precise estimation of the overall IUR of a tested measure. We exemplify our procedure by using a dialysis adequacy measure, which is the percentage of adult hemodialysis patients (older than 18) in a facility with a monthly UFR greater than 13 ml/kg/hr. Patients with time since imitation of dialysis of less than 90 days, time within a facility of less than 90 days, missing weight information before or after dialysis session or missing delivered time of dialysis sessions are excluded from further analysis. Based on the 2012 CROWNWeb data, the average UFR is 9.2 with a median of 8.7 ml/kg/hr. The corresponding average percentage of UFR greater than 13 at the facility level is 19.3%. The total number of facilities is 5,317, with the 10th, 33th percentile, median, 67th, and 90th percentile for number of patients within each facility being 15, 35, 50, 68, and 108, respectively. The IUR based on the whole study population is 0.738. Figure 3 shows a histogram of the facility-level average UFR across a total of 5,317 facilities nationwide.
We randomly draw a certain number of facilities from the study population and calculate the IUR based on the method described in Section 2.2. This way, the variations of IURs are accounted for across different subsamples. To further improve the representativeness of the selected subsample, we consider facilities with various sizes (small, moderate or large).  We then randomly draw facilities such that one third of the selected facilities are from the smallest tertile, one third are from the middle tertile, and one third are from the largest tertile. The confidence intervals of the IUR are calculated, using a logit transformation with the standard error of the logit IUR obtained from the Delta method. Here the logit transformation is applied to improve normality assumptions and assure that the confidence interval is within the range of 0 and 1. We repeat this procedure 1,000 times and obtain the average confidence interval. The width of this average confidence interval measures the precision of the estimate of the IUR, and half the width estimates the corresponding average margin of error. Table 1 shows the calculated average IURs and their confidence intervals against the number of facilities. It appears that wider confidence intervals are associated with smaller numbers of facilities, indicating unreliable estimation of IURs. Moreover, IURs tend to be under-estimated given a small number of facilities. To ensure the length of the confidence intervals is less than 0.2, at least 75 facilities are needed.

DISCUSSION
Facility-specific quality measures are commonly used to monitor dialysis facilities. To successfully develop, test and validate quality measures, a subset of facilities are often recruited for preliminary evaluations. To ensure that the facility-specific measures will achieve a desirable precision, the objective of this report is to determine a minimum number of facilities that should be recruited. To achieve this, we implement a method based on the IUR, which is commonly used to assess the reliability of quality measures for evaluating facilities. In general terms, the reliability of a measure reflects to what extent the measure reflects the actual differences between facilities as opposed to the random variation of patient outcomes within the facility. We calculate IURs and their confidence intervals for various numbers of facilities to determine a minimum number of facilities that should be recruited to ensure that the overall reliability of the facility specific measure "percent of patients with UFR > 13" will achieve a desirable width of confidence interval or margin of error.
The wide average confidence interval indicates that a small number of facilities do not provide enough information to precisely estimate the reliability. Moreover, a small number of facilities leads to biased estimation of the IUR (e.g. the estimated average IUR is smaller than the population based value 0.738). The number of facilities required depends to some degree on the true reliability and the nature of the measure being considered, but this conclusion that a small number of facilities (e.g. less than 30) is insufficient would be quite robust. Of course other considerations besides reliability will also need to enter into measure testing, depending on the context.
In this report, we utilize the accurate estimation of the IUR to determine a minimum number of facilities to ensure a desirable precision. However, it is worth noting that the size of the IUR may not indicate the usefulness of the measure for profiling extreme facilities. [13][14][15] A measure may have a very low IUR, but could still be useful to identify extreme outcomes. Conversely, a high IUR does not necessarily indicate a wide disparity in the quality of care provided by facilities. In fact, Kalbfleisch et al. [15] show that a large IUR can be a signal of the need for further risk adjustment to account for differences between patients across facilities. Nevertheless, the findings presented in this report have implications in practice. First, the reported results will assist in the planning of future measure testing. Second, we provide methods for computing the IUR for complicated quality measures.