{'fraction_non_alphanumeric': 0.02961730449251248, 'fraction_numerical': 0.004925124792013311, 'mean_word_length': 5.075818843509907, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Computing the Expected Value of Sample Information Efficiently: Expertise and Skills Required for Four Model-Based Methods

Objectives: Value of information (VOI) analyses can help policy-makers make informed decisions about whether to conduct and how to design future studies. Historically, a computationally expensive method to compute the Expected Value of Sample Information (EVSI) restricted the use of VOI to simple decision models and study designs. Recently, four EVSI approximation methods have made such analyses more feasible and accessible. We provide practical recommendations for analysts computing EVSI by evaluating these novel methods. Methods: Members of the Collaborative Network for Value of Information (ConVOI) compared the inputs, analyst's expertise and skills, and software required for four recently developed approximation methods. Information was also collected on the strengths and limitations of each approximation method. Results: All four EVSI methods require a decision-analytic model's probabilistic sensitivity analysis (PSA) output. One of the methods also requires the model to be re-run to obtain new PSA outputs for each EVSI estimation. To compute EVSI, analysts must be familiar with at least one of the following skills: advanced regression modeling, likelihood specification, and Bayesian modeling. All methods have different strengths and limitations, e.g., some methods handle evaluation of study designs with more outcomes more efficiently while others quantify uncertainty in EVSI estimates. All methods are programmed in the statistical language R and two of the methods provide online applications. Conclusion: Our paper helps to inform the choice between four efficient EVSI estimation methods, enabling analysts to assess the methods' strengths and limitations and select the most appropriate EVSI method given their situation and skills.


Abstract Objectives
Value of information (VOI) analyses can help policy-makers make informed decisions about whether to conduct and how to design future studies. Historically, a computationally expensive method to compute the Expected Value of Sample Information (EVSI) restricted the use of VOI to simple decision models and study designs. Recently, four EVSI approximation methods have made such analyses more feasible and accessible. We provide practical recommendations for analysts computing EVSI by evaluating these novel methods.

Methods
Members of the Collaborative Network for Value of Information (ConVOI) compared the inputs, analyst's expertise and skills, and software required for four recently developed approximation methods.
Information was also collected on the strengths and limitations of each approximation method.

Results
All four EVSI methods require a decision-analytic model's probabilistic sensitivity analysis (PSA) output.
One of the methods also requires the model to be re-run to obtain new PSA outputs for each EVSI estimation. To compute EVSI, analysts must be familiar with at least one of the following skills: advanced regression modeling, likelihood specification, and Bayesian modeling. All methods have different strengths and limitations, e.g., some methods handle evaluation of study designs with more outcomes more efficiently while others quantify uncertainty in EVSI estimates. All methods are programmed in the statistical language R and two of the methods provide online applications.

Conclusion
Our paper helps to inform the choice between four efficient EVSI estimation methods, enabling analysts to assess the methods' strengths and limitations and select the most appropriate EVSI method given their situation and skills.

Introduction
Decisions on which research studies to fund are intrinsically economic in nature; that is, public, private and third sector (research charity) funders have finite resources and their decisions bear an opportunity cost. Therefore, it is important to prioritise research studies that are expected to yield the greatest benefit for every dollar or euro spent.
Typically, research priorities are set through consultation with experts, decision-or policy-makers and other key stakeholders (e.g., the James Lind Alliance [1]) [2]. Study outcomes are then selected for a number of reasons, such as ensuring timely collection of data. Most research designs focus on clinical outcomes and rarely consider economic implications. The sample size for most randomized controlled trials (RCTs) is determined using power calculations which explicitly manage type I and II errors for a statistical test of the selected primary outcome. However, this approach fails to take into account the opportunity cost of research: a large (and expensive) trial could be proposed while that knowledge could have minimal value to society. In contrast, Value of Information analysis (VOI) values an RCT and other types of research studies in terms of how much they reduce decision uncertainty (i.e., the probability of making a sub-optimal decision) about the best treatment for use in a population of interest. If a suboptimal decision is made, then potential health improvements are foregone. The probability of making the wrong decision is multiplied by the size of the loss incurred by that decision to generate the expected loss associated with making a sub-optimal decision [3]. New data are expected to reduce decision uncertainty, and so reduce the expected loss. The Expected Value of Sample Information (EVSI) measures the "expected reduction in expected loss" from a given research study. Scaled up to the relevant population this can be expressed in health terms of life years, or quality-adjusted life years (QALYs), or in monetary units [4,5]. The difference between the EVSI and the cost of the research is the Expected Net Benefit of Sampling (ENBS).
An increasing number of authorities and Health Technology Assessment (HTA) agencies acknowledge the importance of VOI as part of a cost-effectiveness analysis and are recommending it [6][7][8][9]. Thus, EVSI methods will soon be required from analysts who conduct cost-effectiveness analyses for HTA agencies.
While these methods have lowered the computational barriers to VOI analyses, they differ in their approach, requiring different expertise and skills. However, there is no structured comparison of the practical steps required to use them. Thus, it is challenging for analysts to determine which method is appropriate for their situation and expertise. Additionally, each method has different strengths and limitations that could make it more suitable for a given decision problem. The Collaborative Network for Value of Information (ConVOI) [21] is an international group of researchers with interests in the application and development of methods for VOI calculation. This manuscript provides practical guidance and good practice recommendations for computing EVSI using four recently developed approximation methods. For each method, we: 1) provide a step-by-step guide to its use, 2) compare the expertise and skills required to implement it, and 3) highlight its strengths and limitations. were selected as they place limited restrictions on the complexity of the underlying decision-analytic model and/or data collection exercise when calculating EVSI. The recommendations presented in this paper were developed and reviewed by ConVOI to aid analysts looking to compute EVSI.

Inputs Required for Efficient EVSI Calculation
The considered EVSI approximation methods have diverse requirements and make different assumptions.
However, all require a decision-analytic model on which a probabilistic sensitivity analysis is conducted.
The IS and MM methods also require calculation of the EVPPI for the parameter(s) to be evaluated and updated in the proposed study. If studies informing different parameters or groups of parameters are considered, then EVPPI would need to be computed for each. The following section briefly outlines these requirements and assumptions.

Decision-Analytic Model
VOI requires an objective function that should be optimized to determine the best course of action [22].
Economic evaluations in health care typically define this in terms of net health or monetary benefit for each of interventions [23], which uses a "willingness-to-pay" threshold to put the health consequences on the same scale as the costs for the different interventions [23]. The function that computes this usually takes the form of a decision-analytic model.
Decision-analytic models are mathematical models often used for cost-effectiveness analyses. They draw on set of inputs that we denote . These inputs could include information on the prevalence of a disease, the effectiveness of treatments, background mortality, health-related utility weights and costs. The model can take a number of forms, foremost among them are decision trees, Markov models and microsimulation models [24]. Conceptually, the decision-analytic model maps a set of inputs to the output, net benefit (either in monetary or health units). Assuming risk neutrality, the intervention with the highest expected net benefit should be implemented in the wider population.

Probabilistic Sensitivity Analysis
Probabilistic sensitivity analysis (PSA), sometimes known as probabilistic analysis or uncertainty analysis, is performed to propagate input parameter uncertainty to the model output under each decision alternative and thereby quantify decision uncertainty in a cost-effectiveness analysis [25]. PSA simultaneously varies all parameters for which there is meaningful uncertainty. Uncertainty in the model inputs is characterized using probability distributions, ( ) [26].
PSA is often conducted using Monte Carlo methods where parameter sets are drawn from ( ), for the whole set of model parameters = ( 1 , … , ). The decision-analytic model is evaluated at all , = 1, … , to estimate the costs and health outcomes of each of the strategies. This produces a distribution for the net monetary benefit for each strategy, which we denote NB , for = 1, … , . In non-linear models, PSA is required to generate the expected net monetary benefit of each treatment strategy, . PSA results can be presented in the form of a cost-effectiveness acceptability curve (CEAC), cost-effectiveness acceptability frontier (CEAF), expected loss curves (ELCs), and cost-effectiveness plane [3,6]. Furthermore, PSA is key to determining the value of potential future research using VOI

Expected Value of Partial Perfect Information
The EVPPI computes the value of eliminating all uncertainty about a subset of the model parameters [27][28][29]. Specifically, the model parameters are split into two subsets = ( , ), where we propose to gather further information about the model parameters . Typically, a proposed study does not collect information about all the underlying parameters in a decision-analytic model and therefore, indicates the parameters that will not be directly informed by the proposed data collection.
Mathematically, the EVPPI for the parameters is defined as where the inner expectation in the first term of equation (1) calculates the net monetary benefit for each intervention conditional on . The second term calculates the value of the decision made with current information, i.e., the expected net benefit of the treatment with the highest expected net benefit.
The PSA outcomes are simulated using probability distributions for and the EVPPI is estimated by computing the net monetary benefit for each intervention conditional = for = 1, … , . Several approximation methods are available to estimate the conditional net monetary benefit for each value [27,[30][31][32][33][34] and have been formally assessed in [28] and [29]. Software is also available to perform these calculations [35][36][37]. The MM and IS methods require a PSA simulation matrix with additional columns that contain Monte Carlo estimates of the expected net benefits E | [NB ], conditional on for = 1, … , , but averaged over the uncertain values of . We denote these simulations .

EVSI Methods
EVSI for a proposed research strategy that collects additional data, denoted , is defined as In this setup, are observable, but not yet observedand possibly never will be. At this stage, we consider the possibility of collecting data in the future and we compute EVSI to determine whether we should defer making a decision on the optimal intervention, among the possible options, and instead invest money and time collecting .
In line with a full Bayesian approach (which underpins the ideas behind the VoI analysis), the distribution of is defined by ( , ) = ( ) ( | ) where ( ) is the marginal distribution of and ( | ) is the sampling distribution of the data. In a full Bayesian setting, ( ) is a "prior" distributionin the sense that it represents the current level of uncertainty on the model parameters, before observing the new data . In reality, this distribution is defined by the PSA procedure and can represent the result of a Bayesian update given observed data that is used to construct the current economic model.
We note that will give information on the subset of model parameters , where could be the whole set . By definition, this means that and are independent given and that ( | ) = ( | ).
The second term of equation (2) can be estimated from the initial PSA. To obtain simulations for , we simulate potential study outcomes, , = 1, … , , for each row of the PSA dataset, i.e., we generate a single sample from ( | ) for = 1, … , . Traditionally, EVSI has been computationally demanding and methodologically challenging because a large number of simulations are needed to estimate the posterior mean of the net benefit for each . This requires a Bayesian model for the distribution of the data and the parameters, where the posterior mean is given by The EVSI approximation methods in this paper estimate ( ) with a reduced computational burden. For computational simplicity and stability, VOI calculations should be undertaken using incremental net benefit defined, without loss of generality, as for = 2, … , . We could also use the opportunity loss, defined as the incremental net benefit of all treatments from the optimal strategy * . By estimating the posterior mean of the incremental net benefit, we only need to add − 1 columns containing the posterior mean conditional on to our PSA dataset.

Expertise Required for EVSI Calculation
We have identified three skills that the analyst may require to compute EVSI using the approximation methods reviewed in this manuscript. None of the methods require all three skills but each method requires the mastery of at least one of the skills presented. Table 1 summarises the required skills and inputs for each method.

Regression Methods
Regression methods that model the relationship between a set of predictors and an outcome of interest are required for both the RB and GA methods. Both methods use regression metamodeling to model the incremental net benefit as a function of model inputs or quantities related to model inputs. The two methods differ as they require alternative covariates.
Both of these EVSI methods model relatively complex relationships using models that require little information about the functional relationship between the independent and dependent variables. To account for potential nonlinear relationships between these variables, flexible regression methods characterized using Generalized Additive Models (GAMs) [38] are the predominant regression methods for EVSI calculation. Standard software is available to fit GAMs [39][40][41], but the EVSI estimate can be affected by the structure of the GAM model and, thus, an understanding of these models is required.
Gaussian Processes [42] have also been suggested for more complex problems [13], so knowledge of these methods may also be required.
If regression methods are used to estimate EVSI, it is important to assess whether the regression model has correctly captured the relationship between the independent and dependent variables [13]. Regression methods assume that the residual error is uncorrelated with the fitted values from the regression, has zero mean and trivial covariance structure. These assumptions can be confirmed by observing no groups, systematic features, or outliers in the plot of the residuals against the fitted values [43].

Specification of data generating distribution
EVSI computes the expected value of collecting information in a future research study. The future research study would collect a potential data set, , with different clinical, health or economic outcomes from participants. For example, a clinical trial would collect the primary and secondary outcomes from each participant.
All EVSI computation methods require the simulation of potential data sets (the GA method requires study data simulation to compute the effective sample sizes of the priors of the only if these cannot be estimated directly, c.f. Bayesian Updating). As outlined above, this is achieved by specifying the assumed data generating distribution ( | ). Thus, analysts must have knowledge of probability distributions.
For the RB method, the simulated data sets for = 1, … , must be summarized to reflect how the data would be analyzed following the trial. This requires expertise in standard methods for the statistical analysis of study results, for example, maximum likelihood procedures. A variation of the GA method can be also be applied if a summary statistic is available.
For the IS method, the analytic likelihood for must be specified and coded as a function. Likelihood is used in its proper statistical meaning and, therefore, this specification requires statistical expertise.
Typically, this likelihood specification will require an analyst who has knowledge of statistical theory, function development and coding.

Bayesian Updating
Bayesian methods are a statistical paradigm in which conclusions are formally updated as evidence accumulates [44]. A full Bayesian model consists of probability distributions specified for the model inputs ( ), the prior distribution defined using the PSA distributions, and the sampling distribution for the proposed data collection. Based on these distributions, Bayesian methods update the distribution of the parameters conditional on the data to produce a posterior distribution. In practice, this is undertaken using specialized software such as BUGS [45], JAGS [46] or Stan [47]. The MM and, in some cases, GA methods are explicitly based on Bayesian analysis, which requires expertise in these programs and in the specification of Bayesian models. This is advantageous if the final trial analysis will be performed according to Bayesian principles as the statistical analysis plan has been developed as part of the design process.
Additionally, the GA method is based on the prior effective sample size [48]. The prior effective sample size, denoted 0 , is the number of participants that would need to be studied to obtain the level of information in the prior. In some Bayesian models, typically based on conjugate prior-likelihood pairs, the prior effective sample size can be estimated directly from the parameters of the distributions [48]. If this is not possible, then Jalal and Alarid-Escudero (16) present two algorithms to compute the prior effective sample size (see the Supplementary Materials). The first of these is based on summary statistics and the second on a full Bayesian analysis.

Calculating EVSI
To highlight how these skills and inputs are used to compute EVSI, we have presented step-by-step guides to each of the four methods considered. These algorithms are provided in Boxes 1-4.
Step-by-step guide for regression-based method INB, incremental net benefit; PSA, probabilitic sensitivity analysis; θ, set of inputs in a decision-analytic model; p(θ), probability distributions to characterize uncertainty in the model inputs; S, number of parameter sets that are drawn from p(θ) in PSA; T, number of considered interventions; θ=(φ,ψ), φ are model parameters for which we are aiming to collect further information and ψ are parameters that will not be directly informed by the proposed data collection; X, new data proposed to be collected; μ, posterior mean; W(X), a summary measure for the data. to collect further information and ψ are parameters that will not be directly informed by the proposed data collection; X, new data proposed to be collected; μ, posterior mean; η, PSA simulations used to compute EVPPI; Q, number of times PSA is performed.

Strengths and Limitations
As each method is different, the most suitable method for EVSI calculation will depend on the decisionanalytic model, the sampling distribution for the data, the expertise of the analyst and the amount of computation time available. The following section highlights the strengths and limitations of these methods to help analysts select the most appropriate method. A summary of these strengths and limitations is presented in Table 2.

Strengths
The

Limitations
For complex studies where the number of collected outcomes is greater than five or six, it can be challenging to fit a sufficiently accurate regression model. The simulated study data must also be correctly summarised using ( ) to obtain accurate EVSI estimates. This can be challenging and time-consuming in more complex studies. The relationship between the incremental net benefit and ( ) must also be well-approximated by the regression model to ensure accurate EVSI estimation.

Strengths
This method only requires the PSA results to compute the EVSI (i.e., access to the original decisionanalytic model is not necessary). Furthermore, the IS method can be used irrespective of the number of study outcomes with similar complexity, once the likelihood function for the potential data to be collected has been defined.

Limitations
The accuracy of the EVSI estimation relies heavily on the appropriate specification of the analytic likelihood and the accurate estimation of the EVPPI. Because the most efficient EVPPI estimation methods are based on regression methods [29,31], the analyst must be confident assessing the accuracy of regression models before proceeding with EVSI estimation. Furthermore, the IS method can have computational issues for large sample sizes of the future data as the likelihood tends to 0 with increasing sample size, leading to inaccurate EVSI estimation [10]. Finally, uncertainty in the EVSI estimation procedure cannot be estimated using this method.

Strengths
Estimation of EVSI with this method does not require to rerun the decision-analytic model; thus, access to PSA results is sufficient. This method estimates EVSI using flexible non-parametric (typically GAM) linear regression models. Although GA method requires an estimate of the prior effective sample size 0 for the proposed data collection, 0 can be estimated directly from previous studies. Furthermore, this method estimates EVSI across different sample sizes of the proposed data collection exercise at minimal computational cost, once the prior effective sample size has been estimated. Thus, the optimal sample size for proposed data collection can be obtained cheaply. Finally, the method can estimate the uncertainty in the EVSI estimation procedure using the same approach as in [13] based on [49].

Strengths
This method uses the same nested simulation structure as the gold standard nested Monte Carlo methods.
This means that if an analyst has already developed the nested Monte Carlo method for EVSI estimation, it can be easily adapted for this method. Furthermore, this method has been extended [19] to estimate EVSI for multiple alternative sample sizes with a fixed additional computational cost. This extended method also provides a measure of uncertainty around the final EVSI estimate. The MM method can estimate EVSI irrespective of the number of outcomes considered.
The EVSI package in R has been developed to implement the MM method based on a Bayesian decision-analytic model. The manual is available [50] and the EVSI package can be installed in R using the command: devtools::install_github("annaheath/EVSI")

Limitations
The MM method requires simulated study data and relies on performing a PSA times, with 30 < < 50. Thus, the decision-analytic model must be rerun a significant number of times. Therefore, in computationally expensive decision-analytic models, such as microsimulation models, this method requires significant computational power. Next, if the original PSA simulation size is small, this method is inaccurate as it is based on an estimate of the variance of NB , which must be sufficiently accurate. The MM method can also lead to inaccurate EVSI estimates when the sample size of the proposed study is less than 10. Finally, this method requires an accurate estimate of EVPPI and is more accurate for studies that will have significant impact on the underlying uncertainty in the decision-analytic model, i.e., the EVPPI of needs to be high compared to the value of reducing all model uncertainty (i.e., EVPI), ideally greater than 40%.

Presenting EVSI results
Once EVSI results are available, the EVSI package in R contains several graphical displays to present EVSI and related quantities to practitioners and stakeholders. Results of an EVSI analysis can be loaded into the EVSI package, irrespective of the computation method used. These graphics can then be displayed in R directly or explored using a dynamic graphical display launched from within R. For analysts unfamiliar with R, EVSI results can be loaded into an online interface and these graphics can be explored online at https://egon.stats.ucl.ac.uk/projects/EVSI/Test/. The optimal sample size estimated using VOI methods can also be presented in form of a curve of optimal sample size (COSS) [52].

Conclusion
VOI analysis has the potential to guide policy-makers in the prioritization and design of future research studies, thereby improving decision-making. Increasingly, HTA agencies are acknowledging this potential and are recommending VOI analyses to determine whether/what potential future research is needed. In this study, members of ConVOI have provided a practical guide and good practice recommendations to facilitate the implementation of these methods. Our recommendations outline the inputs, analyst skills, and software required to use each of the EVSI approximation methods. We have also highlighted the strengths and limitations of each method. These recommendations are supported by a recent review that compared properties of these four recently developed, efficient computation methods across three decision-analytic models [10].
An accompanying GitHub repository includes R code demonstrating all four EVSI methods in real-world examples. Thus, this guide helps analysts choose the method that is the most suitable for their application and skills, and our code repository provides practical support to aid the implementation of these methods.
This increases the feasibility and accessibility of the EVSI methods as they become an important and required tool.

Supplementary Materials
Accompanying the manuscript:

PSA Matrix
We provide an example of a PSA dataset in Table S1 and an example of an augmented dataset in Table S2.  Table S2. An example of an augmented PSA dataset θ, set of inputs in a decision-analytic model; S, number of parameter sets that are drawn from p(θ) in PSA; T, number of considered interventions; θ=(φ,ψ), φ are model parameters for which we are aiming to collect further information and ψ are parameters that will not be directly informed by the proposed data collection; nmb.t, net monetary benefit for strategy t=1,…,T ; evppi.t, expected value of partial perfect information for strategy t=1,…,T.

Prior Effective Sample Size Algorithms
This section proposes three alternative methods for calculating the prior effective sample size. The effective sample size 0 is the number of patients that would have to provide data to generate the "amount" of information in the prior.

Direct Estimation
In some settings, 0 can be found from the parameters prior distribution. For example, 0 for a beta prior ( , ) coupled with a binomial likelihood ∼ ( , ) is equal to + . This comes from conjugacy as the posterior distribution is ( + , + ( − )). This is the case for specific conjugate pairs listed in Table S3.

Calculating a Summary Statistic
This method uses a Gaussian approximation to compute 0 based on a summary statistic ( ).
Specifically, the data must be simulated and summarised following the method described in the Specification of data generating distribution section. We can assume that this summary statistic is approximately Gaussian and that the prior for the parameter of interest is also approximately Gaussian. In Therefore, if we have calculated the summary statistic by summarising a dataset of size , we can compute the 0 using this formula.
However, in these cases, we need to compute a summary statistic that is "on the same scale" as the parameter. For example, the prior in a beta-binomial is typically a probability between zero and 1, while the binomial likelihood is generally the distribution of the number of successes. Therefore, the summary statistic must convert the data into the proportion of successes, which can be easily achieved by dividing the number of successes by the total sample. In some setting this may be non-trivial. For these settings, we propose an indirect Markov chain Monte Carlo (MCMC) approach.

Using MCMC
In this case, we calculate the posterior mean for the parameter of interest for each simulated value of , denoted . The prior effective sample size can then be estimated from the variance of using the following formula: More information about calculating 0 can be found in [1] alongside code to estimate 0 within R.

S1.
Jalal H, Alarid-Escudero F. A Gaussian approximation approach for value of information analysis.