{'fraction_non_alphanumeric': 0.024335918165319253, 'fraction_numerical': 0.006434581752186108, 'mean_word_length': 5.30421216848674, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Improving clinical trial interpretation with Acceptability Curve Estimation (ACE)

Effective decision making from randomised controlled clinical trials relies on robust interpretation of the numerical results. However, the language we use to describe clinical trials can cause confusion in both trial design and in comparing results across trials. Acceptability curve estimation (ACE) aids comparison between trials (even where of different designs) by harmonising reporting of results, acknowledging different interpretations of the results may be valid in different situations, and moving the focus from comparison to a pre-specified value to interpretation of the trial data. ACE can be applied to historical trials or incorporated into statistical analysis plans for future analyses. An online tool enables ACE on up to three trials simultaneously.

The specification of trial type (e.g. as superiority or non-inferiority) is important to enable assessment of whether a trial has met its aims. However, the different terms are in themselves confusing. Additionally, seemingly paradoxical situations can arise when comparing across trials, such as trial type differing depending on which treatment is assigned as "control", and trials with similar numerical results reaching different conclusions.
All clinical trial types can be linked by the pre-specified 'unacceptable value' that the 95% confidence interval limits of the estimate of the difference between treatments are compared to. The unacceptable value is zero (or one for a relative effect measure) in superiority trials, the noninferiority margin (less than zero) in non-inferiority trials and greater than zero in super-superiority trials.
Comparison to the pre-specified unacceptable value is an appropriate part of trial interpretation but leads to a binary conclusion of 'trial aim met' or 'trial aim not met'. Outside drug regulation, binary conclusions are widely viewed as problematic 1 , as evidence should not be reduced to a single threshold but should be considered in context with other factors such as the point estimate and confidence interval 2 . Interpretation of non-inferiority trial results also suffers from added complexity around whether the pre-set non-inferiority margin was justified or is relevant for settings outside the trial. Importantly, stakeholders such as clinicians, patients or policy makers may have differing but equally valid unacceptable difference values depending on the relative importance placed on secondary factors such as cost or toxicity.
We advocate the wider use of acceptability curve estimation (ACE) as secondary analyses in clinical trials. We illustrate this using two HIV trials, EARNEST 3 and SECOND-LINE 4 , which had similar quantitative results but drew different conclusions 5 . We demonstrate how alternative presentation of results could aid better comparison and integration of their findings. We present the trials together, but imagine ACE being presented in each trial results paper separately as secondary analyses. For ease throughout we measure differences as treatment minus control for a favourable outcome, so that positive values indicate higher efficacy in the tested treatment.
EARNEST and SECOND-LINE investigated raltegravir as a second-line therapy for HIV in comparison to standard therapy of nucleoside reverse transcriptase inhibitors (NRTI). EARNEST, was carried out in low and middle-income countries. It was pre-specified as a superiority trial because raltegravir was more expensive than NRTI: so it was thought that clear benefit would have to be shown for implementation. The pre-specified unacceptable value was consequently zero. SECOND-LINE, was carried out predominantly in high-income countries. It was pre-specified as a non-inferiority trial because raltegravir was considered to have a better toxicity profile than NRTI: implementation was therefore considered to be worthwhile with similar efficacy. The pre-specified unacceptable value was the non-inferiority margin of minus 12% on the risk difference scale.
The original analysis of EARNEST compared the lower limit of the 95% CI of the difference between treatments (-2.4%) to the unacceptable value of 0, drawing the conclusion of 'superiority not shown' and implementation was not recommended. Analysis of SECOND-LINE compared the lower limit of the 95% CI (-4.7%) to the unacceptable value of -12%, drawing the conclusion of 'non-inferiority' and implementation was recommended. The question then arises of how two trials with numerically similar results can reach opposing conclusions regarding implementation.
Differing, valid, opinions on the unacceptable differences values (0% in EARNEST and -12% in SECOND-LINE), driven in part by different emphasis on secondary benefits, led to the selection of different trial types and the resulting seemingly opposing recommendations. Interpretation through ACE would have clarified this paradox and enabled more nuanced interpretation of the results. treatments being above an 'acceptability threshold' for a range of possible threshold values ( Figure   1). ACE can be presented for all trial types and outcomes. ACE has only been used sporadically in clinical trials [6][7][8][9][10][11][12] with no common naming. ACE is similar to cost-effectiveness acceptability curves widely used in health economics, where weight of evidence, rather than binary conclusions, is a more widely accepted paradigm.
ACE output is best presented in a graph with a continuous range of acceptability thresholds where greater uncertainty around point estimates with larger associated confidence intervals is reflected in a shallower slope. An additional table can present selected acceptability thresholds or the probability that the true value is between selected thresholds. The choice of acceptability thresholds should include the unacceptable difference and lead to a wide range of probability that the treatment difference is above the threshold. is the probability that the true treatment difference is at least the acceptability threshold. Positive values indicate raltegravir is better than NRTI. For example, the probability that the true difference between treatments was at least 0 (i.e. that raltegravir is better than NRTI) was 89% in EARNEST and 70% in SECOND-LINE. Median estimates, 95% credible intervals from models, and pre-specified unacceptable differences are marked. ACE can be implemented using Bayesian analysis, which provides direct estimation of the probability one treatment is better or worse than another when the prior belief of the difference between treatments is added to the analysis. The degree of prior belief, termed priors, are based on existing data and/or expert opinion. Priors are specified as a distribution over the possible values that the difference can take, with non-informative priors essentially being a flat distribution, and strongly informative priors being very concentrated around the area of highest belief. ACE within a Bayesian framework is more consistent with the overall philosophy than within a frequentist framework, although frequentist analysis using confidence curves 13 14 gives very similar results to Bayesian analysis with uninformative priors.
An online tool enables ACE for up to three trials simultaneously using summary information from frequentist or Bayesian analysis https://m-clements.shinyapps.io/acceptability_curve_estimation/.
Further details of analyses, including statistical code, is available in the supplementary information.
Using ACE for reporting, EARNEST results would still conclude that superiority was not shown but could also include a statement such as 'ACE suggested that there was a 89% probability that the true treatment difference was greater than zero (i.e. that raltegravir was better than NRTI) and 100% probability that the true treatment difference was above -5, equivalent to a 0% probability that raltegravir was worse than NRTI by at least five percentage points. There was a 39% probability raltegravir was better than NRTI by at least five percentage points.' Similarly, reporting of SECOND-LINE results with ACE would still conclude that non-inferiority was shown but could also add a statement stating 'ACE suggested that there was a 70% probability of the true treatment difference being greater than zero, a 97% probability of the true treatment difference being above -5 percentage points, and a 16% probability that raltegravir was better than NRTI by at least five percentage points'.
Using ACE, stakeholders requiring clear benefit of raltegravir for implementation could use acceptability thresholds of zero and above, concluding that the probability of raltegravir being better than NRTI was 89% in EARNEST and 70% in SECOND-LINE, but the probability of being more than 5 percentage points better was much lower at 39% in EARNEST and 16% in SECOND-LINE. Other stakeholders more focused on other secondary benefits of raltegravir, such as lower toxicity, could use acceptability thresholds of zero and below, concluding almost certainty of the true treatment difference being more than -5 percentage points in either trial. This allows better comparison across trials than the primary analysis alone.
Interpretation through ACE has three main strengths. Firstly, it enables comparison between trials and trial types by harmonising reporting of results; the use of probabilities is straightforward, widely understood and reflects the uncertainty around the point estimate. Secondly, presentation of ACE acknowledges different acceptability values may exist in different situations. ACE allows clinicians, policymakers, and patients to make informed decisions based on their setting and individual circumstances if they feel the original choice of unacceptable difference is not appropriate for their context. Thirdly, ACE moves focus from comparison to the pre-specified unacceptable value to interpretation of the trial data. This may be especially useful for subgroup analysis where power is necessarily lower than the entire trial, when sample size is restricted, for example in uncommon conditions, and for trials pre-specified as non-inferiority where much of the focus is on the noninferiority margin.
Use of ACE does not remove all of the concerns that can arise with non-inferiority trials, which are caused by the unacceptable difference being less than zero. Non-inferiority trials cannot always provide assurance that the new treatment has a clinically relevant effect (greater than zero) relative to placebo and so it is important to carefully assess evidence about how much better the control treatment is than placebo when selecting the pre-specified unacceptable difference/non-inferiority margin to prevent biocreep. Non-adherence in clinical trials may bias the estimate of treatment differences towards zero, especially if treatment cross-over occurs, meaning that conclusions of noninferiority may be more likely with substantial non-adherence. Analysis of different trial populations (per protocol and intention to treat) or statistical adjustment must still be used to allow for this, but ACE can help improve interpretation.
ACE can be applied to historical trials or incorporated into statistical analysis plans for future analyses. ACE has been previously advocated for use in clinical trials reporting, but its use has not become widespread, perhaps due to lack of common language to discuss the analyses. Increased use of a variety of different trial designs means the time is right for unified design and interpretation though ACE.
We used weakly informative priors in the Bayesian analysis as truly uninformative priors on the logit scale is not possible.
The control event rate was assumed to be 75% in EARNEST and 80% in SECOND-LINE.
The control arm prior on the logit scale follows a Normal distribution with mean of the logit of the event rate and standard deviation 2.
The prior for the difference between the control and treatments arms on the logit scale follow a Normal distribution with mean 0 and standard deviation 8.