{'fraction_non_alphanumeric': 0.018733675583175718, 'fraction_numerical': 0.005674142123750338, 'mean_word_length': 5.301551267499054, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Statistical design considerations for trials that study multiple indications

Breakthroughs in cancer biology have defined new research programs emphasizing the development of therapies that target specific pathways in tumor cells. Innovations in clinical trial design have followed with master protocols defined by inclusive eligibility criteria and evaluations of multiple therapies and/or histologies. Consequently, characterization of subpopulation heterogeneity has become central to the formulation and selection of a study design. However, this transition to master protocols has led to challenges in identifying the optimal trial design and proper calibration of hyperparameters. We often evaluate a range of null and alternative scenarios, however there has been little guidance on how to synthesize the potentially disparate recommendations for what may be optimal. This may lead to the selection of suboptimal designs and statistical methods that do not fully accommodate the subpopulation heterogeneity. This article proposes novel optimization criteria for calibrating and evaluating candidate statistical designs of master protocols in the presence of the potential for treatment effect heterogeneity among enrolled patient subpopulations. The framework is applied to demonstrate the statistical properties of conventional study designs when treatments offer heterogeneous benefit as well as identify optimal designs devised to monitor the potential for heterogeneity among patients with differing clinical indications using Bayesian modeling.

Breakthroughs in cancer biology and immunology have defined new research programs emphasizing the development of non-cytotoxic therapies that target specific pathways in tumor cells or promote anti-cancer immunity. Innovations in clinical trials have followed with "master protocols," that are defined by inclusive eligibility criteria and devised to interrogate multiple therapies or histologies for a given therapy 1,2 . The use of master protocols for oncology has become more common with the desire to improve the efficiency of clinical research. The Lung-MAP study used a master protocol for a Phase II/III comparative trial designed to evaluate biomarker-matched therapies of patients previously treated with advanced squamous non-small-cell lung cancer (NSCLC) 3 . Shortly after initiation, the FDA approved a new immunotherapy for the same population. The design's flexibility facilitated subsequent modification of the standard of care, which preserved the relevance of the study 1 . The CREATE study master protocol evaluates the efficacy of crizotinib in patients with ALK or MET mutations and six different tumor histologies, where each histology constitutes a subtrial of ALK/MET+ or ALK/MET-tumors 2 . The NCI-MATCH trial uses a master protocol to enroll participants into drug-mutation-specific baskets while also incorporating genetic screening 4 .
Trial designs for "precision medicine" often partition study cohorts into putatively homogeneous subtypes on the basis of a shared genetic feature or common treatment target. Basket trials, for example, are designed with the hypothesis that the presence of a molecular marker predicts response to a targeted therapy independent of tumor histology 2,4,5 . While expedient conceptually, the assumption that an individual patient's expectation of treatment benefit is well characterized by the presence of a drug target alone may be limiting in practice. Proliferation rates of tumor cells are known to vary within and between tumor types 6 . Moreover, contemporary sequencing methodologies have revealed extensive genetic variation between different types of cancer. Treatment benefit heterogeneity may not be well characterized on the basis of a single molecular feature. As drug developers pursue drug approvals with broad labels (e.g. Pembrolizumab whose indication was expanded to include any unresectable or metastatic solid tumors with microsatellite instability in 2017) trials will evaluate multiple disease indications for "agnostic" effects. For these trials, subpopulation analysis becomes central to the study design and should be considered in the design and evaluation of trial operating characteristics.
This article first presents and reviews the statistical operating characteristics that need to be explicitly considered in the design stage. We then present a novel framework for use in the design stage of trials with inclusive enrollment for which subpopulation heterogeneity is central to the study's hypothesis in order to identify the optimal design given the potential uncertainty of the trial's outcomes. The proposed criteria facilitates the consideration of multiple scenarios based on proposed and accommodates all possible tradeoffs of type I error, marginal or family-wise, and power, extending to scenarios that are less extreme than those previously considered (e.g. 7 ). The criteria are then used to illustrate how to identify an optimal basket trial design that monitors the potential for heterogeneous treatment benefit among patients with differing clinical indications based on Bayesian modeling using the multisource exchangeability model 8,9 .

Multi-Indication Trial Statistical Considerations
Trials devised to enroll patients with a common drug target arising from multiple, potentially heterogeneous clinical indications pose additional complexities that require careful consideration. In particular, trialists need to decide whether to acknowledge subpopulation heterogeneity in the statistical analysis strategy as well as select an appropriate level of type I error control and discerning its implications for the design statistical power in the presence/absence of heterogeneity. The convention paradigm for trial design would suggest consideration of a single possible scenario or outcome, however this reduces the potential scope of possibilities when explicitly considering multiple indications, is impractical to select in advance, and fails to provide a comprehensive evaluation of the trial's properties across various scenarios of heterogeneity.

Analysis Strategies
For trials enrolling multiple indications or subpopulations, the consideration of how to incorporate subpopulations into the statistical analyses has traditionally existed on a spectrum that spans treating each indication as independent to one of naive pooling under an assumption of full statistical "exchangeability" of the subpopulations with one another. At the extreme of treating all indications as exchangeable and pooling them together, inference is based on the average population effect over all subpopulations and ignores the potential for any heterogeneity amongst indications. While power may be increased under this analysis strategy with the pooling of all data, in the presence of heterogeneity the indication-specific effect will not be identifiable. An example of this approach in practice was the single arm, open-label, Phase II basket trial of Imatinib in 40 different pathologic diagnoses that were hypothesized to have imatinib-sensitive kinases based upon encouraging results from an earlier study in Philadelphia chromosome positive chronic myelogenous leukemia patients 10,11 . The study was ultimately not designed to evaluate subpopulation-specific effects in each of the 40 diagnoses, and all indications were pooled together for one analysis population. On the other extreme of the spectrum, designs could evaluate each indication independently. This approach explicitly accommodates the potential for heterogeneity amongst indications, however it may suffer from limited power when there is low enrollment to an indication. For example, a basket trial was designed to determine the potential efficacy of vemurafenib on BRAF V600 mutations for non-melanoma patients after encouraging results were seen in melanoma patients 4,12,13 . However, the trial did not have equal accrual across all indications with some yielding few responses, with overall conclusions being difficult to ascertain on the basis of the trial alone 14 .
The development of statistical methods to facilitate sharing across exchangeable indications, while avoiding the sharing of information among non-exchangeable indications, provides a flexible approach relative to the strict dichotomy of analyzing the indications as independent or pooled that may increase statistical power while controlling bias 8,[15][16][17][18]24 . While these modeling strategies have encouraging results and have the potential to arrive at conclusions with fewer patients, other authors have encouraged caution when considering these methods for incorporation to a proposed trial with multiple indications 4,7,8 . One of the challenges noted, is that the decisions made for the analysis strategy can have profound implications on the expected enrollment distribution, the trial's operating characteristics, and the likelihood that a trial is robust to departures from the design assumptions in its ability to delineate truly effective indications.

Operating Characteristics
The fundamental consideration when designing trials with multiple indications is the method of type I error control and its expected strength. The potential for incorrectly identifying a null indication as effective can be summarized by marginal or family-wise type I error rates. The marginal type I error rate estimates the type I error rate within each indication separately, whereas family-wise type I error rates consider the entire trial in violation if a single indication falsely rejects the null hypothesis. The type I error rate that is most appropriate for a given study is partially determined by its objectives (i.e., exploratory versus confirmatory), with family-wise type I error rates reflecting more stringent control against false positives within individual indications when compared to marginal type I error control. In the case of pooling all indications together, the marginal and family-wise approaches are identical.
An additional consideration with regards to type I error control is the strength of control across expected scenarios. In the design stage for trials with multiple indications the treatment effect size no longer exists as a one dimension parameter, but may vary within each indication. To address this challenge, Dmitrienko, Tamhane, and Bretz describe two "senses" of family-wise error rate control, weak versus strong, that can be beneficial in helping to frame potential design considerations 19 . Weak control suggests that the family-wise type I error rate is only guaranteed for the global null scenario, where the null hypothesis is true for all indications, but may become inflated if other scenarios are encountered. In contrast, the definition of strong control would strictly require that for any configuration of null and alternative scenarios the family-wise type I error rate is maintained at the desired level. Given the challenge in evaluating this in practice and with simulation studies, we relax this definition to consider strong control so that the family-wise type I error rate is maintained across different reasonable scenarios. For studies with multiple indications, we generalize this concept to control of marginal or family-wise type I error and consider strength of control over a continuum.
Studies that only consider operating characteristics for the global null scenario by default use weak control.
Alternatively, strong control of the type I error accommodates mixed alternative scenarios. While many articles discuss various details of the trade-offs described above, none have provided generalized guidelines or criteria to inform the selection of optimal designs for studies with multiple indications across the spectrum of type I error control 4,7,8 . Additionally, for statistical models with hyperparameter specification, there is no prior framework to identify the optimal values a priori. One potential solution would be to evaluate the operating characteristics of a trial across a variety of assumptions and scenarios, however it is challenging to efficiently synthesize these potentially disparate results and one may end up implicitly averaging various scenarios to represent an overall summary of the operating characteristics. Ultimately this implicit averaging may not identify the optimal design or hyperparameter values to appropriately maintain the desired operating characteristics after the study is completed, potentially leading to a failed study.

Generalized Design Criteria
To address the lack of generalized guidelines for the design of trials enrolling multiple potentially heterogeneous indications, this section presents novel design criteria devised to accommodate the full spectrum of type I error control from weak to strong scenarios. To reflect the oncology setting and seminal simulation study 7 , we focus our discussion on single-arm trials (i.e., trials with no control group) for which response consists of a binary endpoint such as objective tumor response. To facilitate discussion of the criteria and methods presented, we first introduce some notation. Consider a trial devised to enroll a total of N patients with a common genetic mutation arising from a total of J clinical indications for which it is hypothesized, but not substantiated, that the intervention provides similar effects across all subpopulations.
Let i index individual patients such that Y i,j = 1 indicates the occurrence of a successful response for the ith patient within indication j, while = 0 indicates treatment failure. Let n j denote the number of patients observed with indication j and denote the total number of responses in basket j by Consider the context of a trial enrolling J = 5 indications such that the response rate for each indication can assume one of two values which correspond to the null and alternative hypotheses. This yields six possible scenarios which range between the global null and global alternative scenarios ( Traditionally, when designing a trial with multiple indications, only one of the aforementioned scenarios are chosen to summarize the power and type I error and to identify optimal hyperparameter selection. Depending on the analysis strategy, type I error, and strength of control chosen, the choice of a single scenario could present overly critical or optimistic estimates. In the case of basket trials, this traditional approach fails to reflect a reality wherein people inherently synthesize multiple scenarios over their uncertainty as to the true scenario that may be encountered when considering different modeling strategies or for determining how to optimize hyperparameters for a given model. For example, type I error rates are minimized for analysis strategies devised to share information across indications when all indications are ineffective (i.e., the global null scenario). The presence of one effective basket will increase the type I error rate when sharing information across indications. Thus, an analysis strategy may appear sub-optimal when compared to other strategies when evaluated on the basis of a single scenario. Further, despite the fact that preclinical evidence may suggest that a particular molecularly targeted therapy offers "tumor agnostic" effects, indication heterogeneity isn't predictable prior to implementation of human study. Thus, design selection based on trial operating characteristics should reflect the synthesis of many possible trial outcomes.
The methodology presented below is devised for use in the design stage of studies enrolling multiple indications to yield weighted summaries of operating characteristics for use in evaluating different modeling strategies and to identify the optimal hyperparameter specification for a given model. The approach extends optimal design selection to encompass numerous possible trial outcomes. Upon elicitation of null and alternative hypotheses, weights may be assigned to scenarios in relation to counts of indications that assume null versus alternative values. We propose an objective generalized approach based on the scenarios described in Table 1 which considers the number of null or alternative indications per scenario in the weighting, where a trial with J indications has J + 1 scenarios. Let b X denote the number of null indications (for type I error) or alternative indications (for power) in Scenario X, where X is the identifier for the scenario in Table 1, and s is a hyperparameter for the weighting scheme. We propose a standardized approach to defining the weight assigned Scenario X, ω X , as While the formula as presented is for the context where there are two possible outcomes, null or alternative, we note that it could be modified to consider scenarios with additional alternative outcomes or mixtures of scenarios that may be under consideration in the design stage of a study.
The proposed weighting scheme has some appealing properties. If s = 0, all scenarios are weighted equally.
With s > 0, scenarios with more indications of a given criteria (null or alternative) receive greater weight, while s < 0, increases the weight assigned to scenarios with fewer indications. Figure  For example, in identifying the optimal hyperparameter values and posterior probability thresholds for a Bayesian model to have a weighted type I error rate less than or equal to a desired level, one may wish to optimize these parameters under scenarios with fewer null baskets (i.e., negative values of s n ), but then wish to compare the impact of these potential hyperparameters for power across all scenarios equally (i.e., s a =0).
This potential for separate specifications provides further flexibility for summarizing operating characteristics and identifying optimal hyperparameter values across a variety of potential scenarios based upon the context of the given study.  With the introduction of a method to consider multiple scenarios simultaneously, we can define a sequence of steps required to identify an optimal design for any analytic strategy. First, the form of type I error control, marginal or family-wise, must be selected. Second, the hyperparameters s a and s n must be chosen based on the desired weighting of scenarios to be summarized for the given study in order to best evaluate the optimal    analysis strategies can be compared with respect to numerous possible trial outcomes and the optimal model (and potential hyperparameters) with a corresponding decision threshold can be selected for use in the proposed trial. In situations where multiple hyperparameters achieve the maximum observed weighted power, additional rules can be specified for which is selected for implementation (e.g., choose the smallest, largest, or average of eligible hyperparameter values). Returning to our example with J = 5, s n = −10 and s a = 0, the optimal hyperparameters would be selected based on the weighted type I error rate favoring scenarios with fewer null indications that maximize the power that is weighted from all scenarios equally.

Illustration of Proposed Design Criteria
In order to illustrate the proposed design criteria, we first describe potential analysis strategies, including approaches representing naive pooling of data or subpopulation specific analyses which treat indications as independent units and a Bayesian design that facilitates information sharing across indications with multisource exchangeability models (MEM) 9 . To facilitate direct comparisons with simulation studies previously reported by Freidlin and Korn, we present simulation results for scenarios with 5 or 10 indications 7 .

Analysis Strategies
In this subsection we define the analysis strategies compared in our simulation study, as well as describe the concept behind the Bayesian MEM. Lacking multivariate specification of relationships among subpopulations, conventional analysis strategies represent the extremes of either pooling all indications or analyzing each indication independently of the evidence acquired for other indications. More specifically, "pooling" in this context refers to an analysis strategy which is devised to evaluate evidence of effectiveness for all indications via statistical inference of a single response rate that is estimated using data combined from all indications.
Our simulation study uses the exact binomial test to evaluate the statistical significance of a pooled treatment effect, where the corresponding p-value from the test is simply compared to the predefined level of significance (e.g., α = 0.10). Relying on only one test, marginal and family-wise type I error rates are identical for the pooled approach.
At the other extreme, independent analyses infer a total of J response rates estimated from the data is rejected if the posterior probability is greater than a predefined threshold which is identified to maintain a desired type I error rate. We can further note that setting UB=0 represents a Bayesian implementation of the concept underlying the independent approach.

Simulation Design
Design evaluation using the proposed weighted operating characteristics framework is demonstrated using simulation. We consider the context of wanting to design a fixed sample size trial for two sets of scenarios  Table 1). The trial with 10 indications has 11 similarly defined scenarios.
For each scenario, 10 000 simulated trials were completed. For each simulated trial, the MEM prior specifications detailed below and the two frequentist approaches were applied. These simulated trials were then used for the calibration of a given (s n ,s a ) to identify the optimal MEM hyperparameter and the needed posterior probability thresholds to maintain the type I error rate. Finally, the operating characteristics were summarized from the simulated trials by scenario.
While, in practice, (s n ,s a ) should be defined in advance for the context of the specific study being planned, we illustrate the weighted operating characteristics over a grid of hyperparameters (s n ,s a ) ranging from -10 to 10 by increments of 1 to demonstrate a full range of possible conclusions under different weighting paradigms.
Posterior probability thresholds for each combination were identified to maintain weighted marginal or familywise type I error rates of 10%.
To illustrate the ability of the proposed framework to assist in hyperparameter selection, we consider the Bayesian symmetric MEM approach with a constrained EB prior. For this approach we consider UB hyperparameters over 29 values ranging from UB=0 (no borrowing) to UB=1 (all weight to the MEM maximizing the integrated likelihood). The optimal UB parameter is then selected to maximize power for each combination of s n and s a among admissible models for the specified extent of type I error control. When multiple UB parameters maximized power, the smallest value was chosen (e.g., if UB=0.05 and UB=0.10 both maximized power, UB=0.05 was selected as optimal).
We first present a comparison for the scenarios with 5 indications to demonstrate the improvement in calibration between the traditional approach of selecting one scenario to optimize hyperparameter values on versus the proposed weighted operating characteristics approach for the agnostic weighting where s n = s a = 0. We then present the results for weighting operating characteristics for the frequentist independence and pooling approaches and the Bayesian symmetric MEM approach across the grid of (s n ,s a ) values. With the previously described approach to calibration to maintain 10% type I error control, operating characteristics for the Bayesian MEM models based on optimal UB parameters were compared directly to frequentist independence and pooling approaches. Simulations were conducted in R 20 .

Hyperparameter Calibration for Optimal Models
In this section, we examine the different conclusions for hyperparameter calibration in our simulation study with 5 indications if the trial is optimized for only one scenario as compared to the proposed weighted operating characteristics approach. This means that during the design phase the optimal UB hyperparameter for the MEM assuming a constrained EB prior is calibrated either for only one scenario versus the proposed framework which utilizes all scenarios in calibrating the parameters. In either approach, the selected hyperparameter would be used for the analysis after trial completion. For our initial step, Table   2 represents the values of UB that maximize the power for each of the separate scenarios including at least one null indication, with many scenarios having a range of UB parameters that may be considered optimal with respect to power. Recall, we conservatively specify the lowest possible UB value is selected as optimal when multiple are eligible. We can see that only examining the global scenarios would result in aggressive borrowing with UB=1, whereas the intermediate scenarios are more conservative with UB≤0.12.
The estimated marginal and family-wise calibrations and type I error rates, with accompanying power, are presented in Tables 3 and 4, respectively, with the identified posterior probability thresholds as determined for the UB hyperparameter. These tables illustrate a few important messages. First, even when identical UB parameters are selected, different posterior probability thresholds are needed to maintain the desired type I error rate (e.g., Table 4 for the rows representing Scenarios 2 and 3 with UB=0.03). This has implications for the observed operating characteristics, since the posterior probability threshold directly affects how strongly type I error is controlled as well as its corresponding power. Second, simply averaging over the optimal UB or posterior probability threshold values is not equivalent to the proposed weighted operating characteristic calibration. For example, the optimal UB for s n = s a = 0 (equal weights among all scenarios) is UB=0.08 for marginal type I error rate control. Based on Table 3, however, the naive average would be 0.218, a value of UB that will more aggressively share information amongst indications than the optimal choices for 4 of the 5 scenarios. Third, the weighted operating characteristics (WOC) in Tables 3 and 4 are generally more consistent across scenarios when equally weighting each scenario with the proposed methodology, whereas any standalone scenario tends to have greater variability in statistical power and type I error. The findings demonstrate that calibrations that integrate multiple scenarios provide an improvement with respect to the current standard of calibrating based upon one scenario. Optimization with respect to weighted operating characteristics additionally improves upon implicit averaging across scenarios, as the scenario-wise statistical summaries fail to synthesize the myriad possible trial outcomes. Note, that statistical properties for the overall weighted summary, s n = s a = 0, are described in further detail in the next section.  Table 4. Minimum optimal UB for family-wise type I error control from Table 2 and posterior probability threshold (PP Thresh.), with resulting family-wise type I error rates and power presented for all scenarios, including the weighted operating characteristic (WOC) scenario where sn = sa = 0.  The results presented in this subsection reflect the weighted operating characteristics for the combinations of s n and s a for the optimal MEM model, the independence approach, and the pooling approach. Further, we note that a subset of results from the previous subsection in Tables 3 and 4 can be used to evaluate the performance of the proposed MEM model at s n = s a = 0 (WOC) with the Bayesian equivalent of the independence approach for Scenario 5 where UB=0 across all scenarios. While the operating characteristics in Table 3 for the marginal type I error rate comparing Scenario 5 and WOC are largely similar, the familywise type I error rate is controlled more strongly in Table 4 for WOC across scenarios versus the UB=0 in

Discussion
Subpopulation heterogeneity is intrinsic to evaluations of cancer therapies emerging within the evolving oncology landscape. Consequently, trials require explicit designs that take into account the potential for subpopulation effects. When implemented with adaptive and "seamless" decision rules, such master protocol designs with inclusive eligibility may conserve patient and financial resources when compared to the expense   which has demonstrated superiority with respect to other hierarchical modeling strategies 8,9 . When compared to frequentist analyses that either pool indications or assume independence, at least marginal, and sometimes substantial, improvements in the weighted power were evident for the Bayesian MEM approach. In fact, with inclusive scenario evaluations and weighted type I error control, our evaluations suggest it is never considered disadvantageous to leverage the Bayesian MEM approach.
There are important limitations to note. While we attempted to propose a simple formula for determining scenario weights, our weighting scheme will always favor one of the extremes of a singleton indication, except when s n = 0 or s a = 0. The singleton scenario results in a more conservative setting where sharing information may be more challenging, whereas the global null/alternative scenario represents the ideal setting for borrowing across indications. While it seems advantageous to simplify the summary of an entire set of scenarios to a single set of weighted operating characteristics, this may only represent a marginal improvement in the clarity of our approach relative to considering separate operating characteristics for each scenario.
However, even when presented with the results for more multiple scenarios, collaborators may still try to synthesize the results through informal weighting for a single takeaway, and model-specific hyperparameters may not necessarily be optimal by naively averaging over different scenarios. Our simulation considered a single treatment across all subpopulations, representing an easier context than may be encountered in designing complex real world trials. Another challenge is that use of statistical approaches to facilitate information sharing may be seen as too complex to incorporate without specialized knowledge. Recently, new software tools, such as the basket package in R to implement the symmetric MEM approach, have been created to facilitate use of these methods in future studies with multiple indications 23 . The proposed framework for identifying optimal trial designs with symmetric MEMs is being extended to sequential trials with multiple indications as a future research aim. With the increasing emphasis on precision medicine, master protocols offer a pivotal tool for characterizing treatment benefit heterogeneity. While the context of each individual trial is unique, further guidance pertaining to the statistical designs of such trials is needed from regulating agencies to promote best practices for each stage of the research process.

Declaration of conflicting interests
The last author receives research funds from Amgen, and serves as an advisor to Presagia.

Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of        Heat maps for 5 and 10 indication trials  Heat maps for the 10 indication trial simulations demonstrate similar results to the 5 indication trial considering marginal type I error rate control in Figure 5 or family-wise type I error rate control in Figure 6.
The 10 indication results also demonstrate that at no combination of s n and s a is it optimal to not borrow (UB=0), but with less borrowing favored when s n or s a are negative.  . Heat map of the optimal value of UB for 5 indication trial to maintain a weighted marginal type I error less than or equal to 10% for a given combination sn (s for null indications) and sa (s for alternative indications). Heat maps for weighted marginal type I error, weighted power, and change in power for the optimal UB compared to UB=0 (no borrowing).  . Heat map of the optimal value of UB for 5 indication trial to maintain a weighted family-wise type I error less than or equal to 10% for a given combination sn (s for null indications) and sa (s for alternative indications). Heat maps for weighted family-wise type I error, weighted power, and change in power for the optimal UB compared to UB=0 (no borrowing).  . Heat map of the optimal value of UB for 10 indication trial to maintain a weighted marginal type I error less than or equal to 10% for a given combination sn (s for null indications) and sa (s for alternative indications). Heat maps for weighted marginal type I error, weighted power, and change in power for the optimal UB compared to UB=0 (no borrowing).  . Heat map of the optimal value of UB for 10 indication trial to maintain a weighted family-wise type I error less than or equal to 10% for a given combination sn (s for null indications) and sa (s for alternative indications). Heat maps for weighted family-wise type I error, weighted power, and change in power for the optimal UB compared to UB=0 (no borrowing).