{'fraction_non_alphanumeric': 0.029193236997601357, 'fraction_numerical': 0.011993213596208974, 'mean_word_length': 5.078947368421052, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
The use and quality of reporting of propensity score methods in multiple sclerosis literature: A review

Background: Propensity score (PS) analyses are increasingly used in multiple sclerosis (MS) research, largely owing to the greater availability of large observational cohorts and registry databases. Objective: To evaluate the use and quality of reporting of PS methods in the recent MS literature. Methods: We searched the PubMed database for articles published between January 2013 and July 2019. We restricted the search to comparative effectiveness studies of two disease-modifying therapies. Results: Thirty-nine studies were included in the review, with most studies (62%) published within the past 3 years. All studies reported the list of covariates used for the PS model, but only 21% of studies mentioned how those covariates were selected. Most studies used PS matching (72%), followed by PS adjustment (18%), weighting (15%), and stratification (3%), with some overlap. Most studies using matching or weighting reported checking post-PS covariate imbalance (91%), although about 45% of these studies relied on p values from various statistical tests. Only 25% of studies using matching reported calculating robust standard errors for the PS analyses. Conclusions: The quality of reporting of PS methods in the MS literature is sub-optimal in general, and in some cases, inappropriate methods are used.


Background on Propensity Scores Propensity Scores
In a study, each subject has observed pre-treatment covariates (L), receives a treatment at baseline, and has an outcome recorded at the end of the follow-up. For convenience, we will refer to A=1 for treated, and A=0 for control; 'control' refers to a subject who is untreated or is receiving a placebo or standard therapy. In an observational setting, (a subset of) the covariates under consideration may be confounders; that is, they are informative of both the probability of being treated and of the outcome. As a result, differences in the outcome between the two treatment groups may be attributable to differences in the distribution of these covariates between the two groups rather than to the treatment itself. Imbalance in covariate distributions across treatment arms means that the relationship between treatment assignment and outcome may be confounded, preventing any causal conclusions about the effect of the treatment on the outcome.
The PS is defined as P(A=1|L), the probability of being treated given a subject's baseline covariates. The PS is a balancing score, meaning that conditioning on the PS balances the distribution of measured covariates between the treatment groups, thus removing the dependency between treatment assignment and outcome 1 . Three assumptions are necessary for this property to hold: (i) a subject's outcome is not influenced by other subjects' treatment assignments, (ii) no unmeasured confounders, meaning that all covariates that predict both treatment assignment and outcome are recorded in L, and (iii) positivity 2 , meaning that each subject has a positive probability of being treated or not at every combination of the confounders.

Estimation of Propensity Scores
In practice, the PS is unknown and must be estimated. Usually, the probability of being treated is modeled using logistic regression with the observed covariates as predictors. An analysis based on the estimated PS assumes that the PS model is correctly specified; that is, sufficient covariates to control confounding are included in the model and in their correct functional forms. This assumption may be relaxed in some cases by the use of tree-based machine learning methods or other flexible methods 3 . Interaction and high order (polynomial) terms are usually added to the PS model if researchers suspect that a non-linear or non-additive relationship should be accounted in the PS model, which is often evident from plotting relevant bivariate relationship. For a covariate that is not balanced after fitting a simple main effect PS model 4,5 (e.g., when the respective SMD is still high after PS matching or weighting), changing the specification of the PS model (e.g, adding non-additive or non-linear terms of that covariate) is often helpful to achieve better balance.

Selection of Covariates
Ideally, the selection of the covariates to be included in the PS model should be based on expert opinion and evidence from the literature. The general recommendation is to include covariates that are causally related to the outcome, irrespective of whether they are causally related to the exposure or not, as this increases the likelihood of controlling for confounding while not increasing variance 6 . In the absence of knowledge about causal relationships, several confounder selection criteria are proposed 7 . Recently, automatized variable selection procedures are gaining popularity, in particular, in the context of high-dimensional settings [8][9][10] .

Types of Propensity Score Analyses
Four analyses using the PS are commonly used in MS literature: matching, weighting, stratification, and regression adjustment.

Matching
Matching creates pairs (or sets) of treated and control patients who have "similar" values of the PS and compares the outcome within pairs to study the effect of the treatment. Pair matching or 1:1 matching matches one control patient to one treated patient while many-to-one matching, or 1:M matching, finds M control matches for each treated patient. The matched sample refers to the subset of patients successfully matched. Its size may differ from the original sample size, depending on the algorithm used to create matches. The choice of outcome analysis after matching is then driven by the type of outcome (binary, continuous, time-to-event). It is recommended that inferences for the treatment effect should account for the dependencies within matched sets created via matching by conducting matched pairs analysis (i.e., by calculating robust standard errors, accounting for the clustering in matched pairs) 11 , although such an analysis may lead to loss of power 12 .
Among many more options, two matching algorithms are popular in clinical applications: greedy and optimal matching 13 . In greedy matching, each treated patient is matched to the control patient who has the nearest estimated PS value. A caliper width may be used to define the maximum allowable distance between the estimated PSs of the matched treated and control patients, e.g., 0.05, 0.1, or a fraction (usually 0.2) of the standard deviation (SD) of the PS on the logit scale. A treated patient may remain unmatched and therefore be deleted from the matched sample if no control patient with a similar estimated PS (e.g., within a prespecified caliper width) can be found. Unlike greedy matching, optimal matching finds the set of matched pairs that minimizes the global distance in estimated PS across all matched pairs. Greedy and optimal matching can be implemented with or without replacement. Matching with replacement implies that a control patient may be matched to more than one treated patient while matching without replacement uses each control patient as a match at most once. Full matching can be achieved with either algorithm. In full matching, all patients in the original sample are used by matching each control patient to one or more treated patients in addition to matching each treated patient to one or more control patients. The size of the matched sets varies depending on the number of suitable matches available for each patient, and constraints can be imposed to bound the ratio of treated and control subjects in each matched set 14 .

Weighting
The estimated PS can be used to derive weights; weighting the sample creates a pseudopopulation in which the distribution of the covariates matches some target population. Different choices of weights yield different target populations 15 . The most common weights are the inverse probability of treatment weights in which each patient is weighted by the inverse of the probability of being assigned to their observed treatment, i.e., the PS for treated patients and one minus the PS for control patients. In the weighted sample, the distribution of the confounders is independent of treatment assignment and matches the distribution of the overall population.
The existence of substantially large weights (e.g., compared to sample size) often leads to increased variability of the estimated effect. Stabilized weights may be computed to reduce the variance of the estimated effect 16,17 . Alternatively, ad hoc adjustment methods such as weight trimming or truncation are popularly used to reduce the impact of the extreme weights, i.e., weights larger than a fixed threshold are truncated to that threshold 17,18 .

Stratification
Stratification uses the estimated PS to create strata; for example, one may classify subjects into five strata based on quintiles of the estimated PS distribution 19 . Stratum-specific estimated treatment effects are subsequently combined into an overall estimated treatment effect.

Regression or Covariate Adjustment
Regression adjustment includes the estimated PS as a covariate in a regression of the outcome on the treatment group indicator. Compared to standard regression that adjusts for each confounder, regression adjustment reduces the dimension of the confounders into a single variable (the estimated PS) at the cost of losing the interpretability of the associated regression coefficient. Unlike the three other approaches, regression adjustment generally assumes that there exists a linear relationship between the outcome and the estimated PSs, and does not directly aid in achieving balance of the covariate distributions (e.g., no clear separation between the design and analysis stages) 20 . Therefore, the ability of regression adjustment to remove systematic differences between treatment groups is limited compared to other approaches 21 , although some extensions can improve performance 22,23 .

Standard Errors
Calculation of the standard error of the treatment effect estimator should account for (1) the uncertainty of PS estimation (because the true PS is unknown and is estimated with the data), (2) the within-set correlation that may have been induced by matching (for PS matching), and (3) the within-subject correlation in weighted observations (for PS weighting). Ignoring these factors may lead to biased estimation of the standard error, which in turn leads to invalid confidence intervals and inferences. Methods to account for these factors include calculating robust standard errors, which incorporate the uncertainty of the PS estimation. Alternatively, one could use computationally intensive bootstrap-based methods 17 .

Diagnostics
The balance of the confounders within matched sets, in the weighted sample or across strata, can be assessed with a diagnostic tool popularly known as standardized mean differences (SMDs) for covariates. A successful balance is then inferred if SMDs are below a chosen cutpoint, usually 0.1 or 0.2 25 . Graphical display of SMDs (e.g., side-by-side boxplots or empirical distribution functions, or love plot 21,27 ) are often used to assess the balance. Formal statistical evaluations such as C-statistics, goodness-of-fit test, Kolmogorov-Smirnov test, or t-test for continuous covariates are sometimes used 28 but many researchers criticize the significance of test-based approaches due to their reliance on sample size 24,29 . Other than checking balance, researchers often report various sensitivity analyses, for example, to estimate the amount of hidden bias due to unmeasured confounding, i.e., the likelihood that an unmeasured confounder would change the conclusion of the analysis 30,31 .

Additional Methods
The review was conducted through December 2018 and updated in July 2019 with the most recent articles. The following two terms were used for the database search: "propensity score" and "multiple sclerosis." A single reviewer was responsible for the eligibility screening and retrieval of data to a pilot-tested form. The form was pilot-tested, reviewed, and refined after

Studies Included in the Review
Thirty-nine of 65 articles were retained for data extraction

General Data Analysis Approaches
Supplementary Table 5 presents the general methodologies adopted in the studies. Only 8% of studies used single or multiple data imputation to address missing data. In 13% of studies, treatment discontinuation and drop-outs were not mentioned. In the remaining 34 studies, treatment persistence was an eligibility criterion (10 studies), was measured or tracked as an outcome of interest (9 studies), was accounted for by censoring the patients at the time of discontinuation or drop-out (22 studies), or was handled via pairwise censoring 69 in studies that used matching (7 of 28 studies). One study mentioned the possibility of treatment discontinuation but conducted an intention-to-treat analysis. About 64% of the studies reported at least one sensitivity analysis. The influence of unmeasured confounding was tested with sensitivity analyses in 34% of studies, with 21% of those studies using Rosenbaum bounds 70 . About 11% of studies performed on-treatment sensitivity analyses to assess assumptions made about treatment discontinuation. On the extreme end, the presence of the 'zero-cell' issues (i.e., no subjects in at least one cell of the confounder categories vs. the treatment group tabulation), a potential (practical) violation of the positivity assumption, was apparent in 13% of studies, whereas the existence of such issue was difficult to determine in 15% of studies. In 54% of studies, the PS adjusted analysis was compared to a basic unadjusted outcome regression.
Most studies (85%) explicitly reported the statistical software used for the analysis. R (R Core Team: Vienna, Austria) was the most commonly used software (51%), followed by SPSS (IBM

Comparison with Other Reviews
Supplementary Table 6 summarizes the findings from previous reviews on the use and reporting quality of PS in the following disease areas: cardiovascular 11,71 , infective endocarditis 72 , intensive care and anesthesiology 73 , psychological and educational research 74 , sepsis 75 , critical care 76 , and cancer 77 . Three reviews 11,71,72 only focused on studies that used PS matching while two other studies 75,76 only reviewed studies that could be matched to an RCT.
Matching was the most commonly used method across studies reviewed in all disease areas.
Our review found the highest proportion of studies that used weighting as the PS method (15%) compared to the other reviews, which were more likely to report adjustment or stratification as PS method. Our comparison with other reviews also highlighted good practices and opportunities for improvement. For studies using matching, between 19% and 35% of the studies in previous reviews did not report the matching algorithm that was used. This proportion was only 7% in our MS review. Our review reported the smallest median number of variables used in the PS model despite reporting relatively large sample sizes and numbers of treated patients. Further, there was a relatively high sample size reduction due to matching in our review (average: 46%) and previous reviews (32-56%). Encouragingly, a high proportion of PS matching studies checked the post-PS balance in our review (96%) as well as previous reviews (50% to 88%). Future reviews of PS methods should extract key elements of the overall analysis approach to allow better comparisons between reviews. These reviews should include sample size, balance diagnostics, sample size reduction, treated-to-variable ratio (i.e., number of treated subjects divided by the number of covariates included in the model), and matching algorithm.

Areas of Potential Knowledge Gaps
From the PS literature, it is known that even slight PS model misspecification may result in a biased treatment effect estimate 78 . In real-world applications, analysts are generally trying to guess the correct specification of the PS model, while trying to obtain covariate balance. While estimating the PS, the prediction can be improved by including a non-linear function of the confounders or with interaction terms in the logistic regression. Robustness to PS model misspecification can also be improved by using machine learning and ensemble methods, in terms of PS prediction, the bias in the effect estimate, and covariate balance 79 . These alternative methods for fitting PS models are not commonly practiced within the MS research community.
Our review also highlighted that a large proportion of studies used p-values to check for post-PS imbalances (43% of studies that used matching and 35% of studies that used weighting), which is generally discouraged.

17.
Austin PC, Stuart EA. Moving towards best practice when using inverse probability of treatment weighting (IPTW) using the propensity score to estimate causal treatment effects in observational studies.