{'fraction_non_alphanumeric': 0.029164630030491353, 'fraction_numerical': 0.00805344133324023, 'mean_word_length': 5.062367715535488, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Stakeholders' feedback on the proposed recommendations for updating the patented medicine prices review board (pmprb) budget impact analysis guidelines.

INTRODUCTION
The present study aimed to obtain Canadian stakeholders' feedback on a list of proposed recommendations for updating the Patented Medicine Prices Review Board (PMPRB)'s 2007 budget impact analysis (BIA) guidelines.


METHODS
A mixed-methods study was designed to obtain feedback from two stakeholder perspectives-(public and private) payers and manufacturers-on the proposed BIA recommendations. We obtained policymakers' opinion through one-on-one interviews and collected feedback from manufacturers and their consultants using a survey. The interview guide and the survey were developed based on the list of recommendations related to BIA key elements, which were either not discussed or addressed differently in the PMPRB 2007 BIA guidelines. The list was derived from 16 Canadian or other national and transnational BIA guidelines. A thematic analysis was applied for analysis of the qualitative (interview) data.


RESULTS
Thirty-five policymakers and manufacturers participated in the study. Stakeholders supported the inclusion of 56% of the proposed recommendations into the guidelines pertaining to the use of expert opinions, data extrapolated from the payers' database, scenario analysis, and dynamic population. Inclusion of indirect costs, and cost transfers from other jurisdictions, were not approved. There was no consensus regarding the inclusion of patients' adherence/compliance and cost offsets.


CONCLUSIONS
The present study has provided sufficient insights to enable the creation of a penultimate version for updating the PMPRB BIA guidelines. This penultimate version will be subject to a broader consultation among stakeholders prior to a final revision and approval. Further Canadian stakeholder feedback is required for reaching consensus on inconclusive recommendations.


INTRODUCTION
In Canada and other developed countries, spending on pharmaceuticals is expected to increase significantly with the population aging and the introduction of highly specialized, expensive medicines. 1 Health technology assessment and budget impact analysis (BIA) are important components to determine whether a drug will be approved by the drug benefit programs. In Canada, the fourth version of the guidelines for the economic evaluation (EE) of healthcare technologies was updated in 2017, and manufacturers are required to comply with these guidelines when submitting cost-effectiveness analyses to support the public or private reimbursements of their products. Similarly, manufacturers have to provide a 3-year BIA to reimbursement authorities. The first and the only Canadian BIA guidelines were published in 2007 by the Patented Medicine Prices Review Board (PMPRB). 2 These BIA guidelines provide recommendations on BIA analytical model structure, data input, and sources and reporting format, and have been approved and adopted by most of the Canadian provincial drug plans. 3 Since 2007, many BIA guidelines have been published or updated in several jurisdictions, but contrary to the Canadian guidelines for EE, which have been updated thrice to incorporate new developments in the conduct of EEs, 4 the Canadian BIA guidelines have never been updated. To this end, we conducted a systematic review of BIAs [5][6][7][8][9][10][11] to identify recommendations from the literature that were either not included or discussed differently in the 2007 PMPRB BIA guidelines. In this paper, we present the results of a qualitative and quantitative (mixed) analysis of Canadian stakeholders' views and feedback on the proposals for updating the Canadian PMPRB BIA guidelines. 11,12

Study Design
This study was premised upon the central research question, "What is the Canadian stakeholders' perception about the practicality and relevance of the newly proposed BIA recommendations for updating the Canadian BIA guidelines?" This mixed-methods study obtained recommendations related to BIA key elements, which were either not discussed or addressed differently in the PMPRB 2007 BIA guidelines. The list was derived from 16 Canadian or other national and transnational BIA guidelines. A thematic analysis was applied for analysis of the qualitative (interview) data. research ethics approval from Hamilton Integrated Research Ethics Board (project number: 2923). The study includes both qualitative and quantitative data collection.
For the qualitative analyses, a semi-structured interview guide was developed collaboratively by the research team to ensure that the questions would address the research objectives of the study. 13,14 The interview guide was developed based on the discordance that was observed between the PMPRB 2007 and the other BIA guidelines that were reviewed 5,7,9 (Table 1). These include, for example, proposing "dynamic" (vs. closed) population, 1 including catch-up effect in the case of chronic conditions, scenario analysis for managing uncertainty, in the PMPRB 2007 BIA guidelines. Based on this information, we developed a semi-structured interview guide, which included 10 interview questions around major themes (i.e., BIA usage in drug reimbursement decisions and price adjustments in Canada, BIA usage in disinvestment decisions, and linking of incremental cost-effectiveness ratio [ICER] with BIA [affordability] and BIA key elements). Appendix Table 1 presents the semi-structured questionnaire. In addition, it was agreed that as interviews progressed, interview probes and follow-up questions could be amended given the completion and content of previous interviews, and in order to elicit the most comprehensive information possible from the research participants. The interview guide also included 14 closed questions (Appendix Table 2) for which a Likerttype ordinal scale was used to rate the responses, ranging from 1 to 5 (1 = "strongly disagree," 3 = "neither agree nor disagree," 5 = "strongly agree"). 1 Patients could be added to or removed from the analysis based on whether they meet the inclusion criteria or not over time. In some cases, when a drug applies to a well-defined group of patients, the BIA may require a defined closed population. All interviews were conducted by the lead author (NF), who had extensive knowledge of the BIA context. The methodological principles of interpretive description were applied to sampling, data collection, and analysis procedures for the interviews. 15 Interviews were mostly conducted in a single meeting using the Google Hangout application (n = 6) except for three participants in whose case a telephonic interview was conducted. The 30-minute (average) interviews were recorded, and they were completed between March 2018 and September 2018.
In parallel with the interviews, an online written survey consisting of 30 questions was developed using SurveyMonkey (Appendix 2). We converted each BIA recommendation that is not considered or discussed differently in the Canadian BIA guidelines (Table 1) to a question in order to get participants' opinion to assess whether they agreed, disagreed, or neither to include a recommendation in the Canadian BIA guidelines. It was mandatory to answer all the questions, and there was no option to move backward through the survey. The survey was open from May 10, 2018 to December 31, 2018.  The initial email included brief information about the project, the identity of the interviewer (NF), the purpose of the study, the number of questions, the expected interview duration, and that the study was voluntary. Recipients were also told that the interview would take place through an online meeting at a time convenient to them. It was stated that their agreement to be interviewed was inferred as their consent to participate in the study. A second reminder was sent 2 weeks after the initial request, followed by a third reminder, which was sent a month later. Those who did not respond to the third reminder were contacted by phone.
For the online survey, we sought participation from pharmaceutical manufacturers and industry reimbursement consultants in order to obtain a pharmaceutical industry perspective. Recruitment was conducted with an invitation and a URL link that was sent by email to these stakeholders. Similar to the interview, a second reminder was sent 2 weeks after the initial request, followed by a third reminder, which was sent a month later.
With consent obtained prior to the interviews, all interviews were audiotaped and stored in an MP3 format. Two authors (NF and BJ) transcribed the data. The interviews, transcription, and analyses were conducted concurrently, allowing an opportunity for new themes to emerge across participants and for further exploration of these themes throughout the remaining interviews. 15 A deductive content analysis that is based on previous knowledge and framework was deemed appropriate for this study. 16 Given that the present study was one of four studies for the author's (NF) doctoral dissertation, NF independently coded all of the transcripts. Specifically, NF completed iterative readings of each transcript. This allowed her to gather, label, and compare keywords from the text that captured key thoughts and concepts described by the participants, referred to as "codes." The author generated overarching themes among the codes through a process of identifying patterns of coding within and across participants. She generated an initial codebook with definitions of each code and the linking themes. The online survey data were analyzed using the "results analysis" feature of the SurveyMonkey application.

RESULTS
We conducted nine interviews with policymakers, including public and private payers (62% response rate), and collected 27 online surveys from reimbursement experts in the pharmaceutical industry (51% response rate). The audio data for one participant was lost due to an unexpected error in the recording process (although interview notes were still available). Thus, transcripts from eight interviews entered the thematic analysis. The results are reported below in three sections: (1) feedback from policymakers (BIA reviewers), (2) feedback from manufacturers or their consultants (BIA producers), and (3) comparative analysis between policymakers and manufacturers/consultants.

Feedback from Policymakers
Through a thematic content analysis, we identified the following major themes in the interview results: (1) BIA usage in drug reimbursement decisions and price adjustments in Canada, (2) BIA usage in disinvestment decisions, (3) linking ICER and BIA (affordability), and (4) BIA key elements (e.g., time horizon) including additional recommendations for improving the guidelines.

BIA Usage in Drug Reimbursement Decisions and Price Adjustments in Canada
Most interviewees believed that BIA could be useful in drug reimbursement decisions and price adjustments. This was captured by the following comments: "I would say they are very useful" or "it is the crucial part of assessing the affordability to pay for new technology, …, very important, the most important [part is] when it comes to the reimbursement because it shows the … how the new technology may impact the budget…" The remaining interviewees did not have a strong idea about it due to the fact that they were representing organizations that do not actually review BIAs or do not usually use BIA for price adjustments (e.g., private payers). In the private drug benefit programs, BIA could be helpful in setting insurance premiums; however, according to an interviewee, premiums tend to be set about [12][13][14][15][16][17][18]

BIA Usage in Disinvestment Decisions
Most participants believed that, at least theoretically, BIA could be helpful in disinvestment decisions (delisting drugs); however, in practice, there are many other factors that would need to be taken into account in delisting a drug (e.g., clinical efficacy and safety, cost-effectiveness, ethical considerations, patient access), which makes it a rare occurrence in the formulary management process. "The way we use it right now is when we look at the BIAs we just generally look at it when we do listing decisions for our formularies at the very front end, when we decide whether or not we list that drug, so we don't ever look at them later to make disinvestment decisions pretty much, but I assume they [BIA] could come in handy" was mentioned by one of the provincial drug plan's representative. According to the latter interviewee, BIA could be useful in disinvestment decisions in the circumstances that they do consider disinvestment such as when a new more cost-effective drug becomes available and is associated with a considerable cost saving.

Linking Incremental Cost-Effectiveness Ratio and Budget Impact
In a constrained budget, the higher the budget impact of a new drug, the lower the ICER threshold for appraising the drug for reimbursement. 17 We asked the Canadian stakeholders for their opinion about the use of BIA (affordability) in price adjustments through defining the ICER thresholds. Some found the idea of linking ICER and BIA helpful in the sense that they are complementary to each other, and having both pieces together could provide policymakers with a better understanding of value for money and affordability in assessing a new medication for reimbursement and price adjustments. A few stakeholders preferred keeping them separate (as they are), for example, "I think keeping it [BIA] separately would make more sense … so it gets down to what the purpose is, and that is to determine the BIA for the drug plan."

BIA Key Elements and Additional Comments for Improving the PMPRB BIA Guidelines
With regard to the perspective of adopting in a BIA, in addition to the public or private payer perspective, some interviewees believed that asking manufacturers to include the patients' perspective as a complementary component to the BIA base-case analysis would not be practical or feasible. The main concern was that in some jurisdictions, "there are so many different scenarios/plans that they could have for co-payments and co-insurance, which makes it really hard for the manufacturers to capture any of that in their submission, or in their BIA." Participants were asked for their opinion about the advantages and limitations of the current 3-year time horizon in the Canadian BIA guidelines. Some participants believed that a longer time horizon (≥3 years) could be more helpful in Canada (e.g., 5 years). The advantages and disadvantages of a 3-year or a longer time horizon are summarized in Table 2. Increasing the uncertainty, especially for market size estimation and lack of real-world information, was the main concern of most participants on using a longer time horizon. Cost analysis is directly related to the perspective of the adopted budget holder. If the new drug represents one of a class of drugs, the least cost alternative (LCA) within the class as defined by the drug plan could be used to set the price of the new comparator. Among public plans, "Alberta and British Columbia [use LCA]… Ontario doesn't" and "it varies by the provinces based on what they deem to be interchangeable." Private payers have programmed their system "to recognize what the lowest cost is, and [the] price for that [which] generally is the generic [version of the drug], but sometimes it might not be." From the policy-making standpoint, there may be an opportunity cost associated with introducing new technology, as it may use additional resources that must be taken from the existing services. 18 Having this definition in mind, participants believed that manufacturers could not estimate the opportunity cost, "obviously in terms of specific decisions that provinces make, it's literally impossible for the drug company to know what we would do with the money instead "three year period is easier to project because it's a kind of short-term projection or forecast. " "If it's less than three years, you start losing relevance, and if you make it longer than three years, there's just way too much uncertainity. " "…three years is good. " Disadvantages (n = 1) "often the three-year budget impact is assumed as a long-term time horizon; however; it usually takes at least 3 years before the drug reaches steady state (plateau) in the market. So the three-year budget impact 'grossly' underestimates the long-term budget hit, which provinces will be ultimately settled with. " Time horizon > 3 years (e.g., 5 years) Advantages (n = 2) "For 5 or 6 years, it is harder to project what will happen in the market, especially it is hard to project the market trends and what would happen. " "[One could] see when the drug will reach its peak sales. For example, many drugs today, they might not reach their peak sales after three years, they will go further and further. So the impact on the budget may not be complete… because it shows lower shares of market uptake after three years, after that, it doesn't go up and show the full extent of the impact. " Disadvantages (e.g., 5 years) (n = 2) "For a longer TH, many manufacturers don't feel that they can secure market assumptions based on what they submit. So as you increase the time horizon, you increase the likelihood that something is going to be generic or [there will be] struggle through the market…I think the past three years is a little bit too much for us [in Canada]. " "The one limitation I think that went for [a] longer BIAs is, right now a lot of things can change during that time period, so three years is good. " n = the frequency of discussions about that specific issue (e.g., advantages of TH = 3). of funding this so… that's not really something they can answer." Modeling may be needed to calculate the budget impact for bringing together the best available data from different sources. 1,3,[18][19][20][21][22][23] If there is an EE in advance to the BIA, assumptions should be consistent with EE. 3,[18][19][20]22,23 All participants believed that modeling could be helpful as long as it is as simple as possible. Usage of complicated models (e.g., Markov) is not required. One of the participants mentioned that, "certainly we try to model out the condition or population as much as we can, to get to a level that gives us confidence while providing say enough certainty, so that's an extra level of complexity that usually is not needed." One participant brought up the discussion related to BIA models in the United States in comparison with the Canadian approach. "I think in the US there's less of a reliance on cost-effectiveness analysis (CEA). So there's a lot of emphasis on BIA, and so they might not be getting some of this additional information regarding some of the complexity of the condition with BIA alone. So I guess it really depends on the purpose of the BIA and from what perspective it is being conducted. … if you already have a CEA that accounts for some of this already, and the BIA is completely from a pharmacy perspective or a drug plan perspective, then maybe there's not the same level of need to have components of the CEA [as] part of the BIA. But … if you don't have a CEA or the perspective of the BIA needs to be broader, [then it is another story]… it really depends on the question that the BIA is intended for." From a private payer's perspective, sensitivity analysis is a better way to take into account the complexity of the disease (e.g., acute plus chronic conditions). In general, participants believed that the extent of required modeling complexity depends on the disease condition and payers' perspective (e.g., in one province, they are sometimes interested in a more dynamic model if the treatment reduces the disease mortality, disease rate of complications, and when it changes the duration of treatment). Moreover, the idea of reflecting both chronic and acute conditions is more about using incidence and prevalence-based approaches in BIA. A "re-assessment approach" was also recommended by a payer to determine the long-term consequences of medications in chronic conditions. All but one participant had read the 2007 PMPRB BIA guidelines (that individual was familiar with the US guidelines for BIA), and all believed that there is a need to update the PMPRB BIA guidelines mainly because they are considered out of date. All participants provided comments for improving the new version of the PMPRB BIA guidelines, which are summarized in Appendix Table 3.
Based on the results of the survey component of the interview guide (Appendix Table 4), a majority of interviewees believed that (1) treatment switches and (2) changes in the rate of mortality and disease progression are important to be captured over the time horizon. Most of them welcomed the idea of providing a list of acceptable databases as reliable references for input data in BIA calculations in the updated version of the Canadian BIA guidelines. Most of them also agreed with a cap or threshold for the budget impact of new drugs to signal the need for negotiation with manufacturers for lowering the price. They thought it is important to build in a reassessment process of BIAs in a future real-world, post-market environment. The opinions regarding the inclusion of patient adherence and compliance in the target population assessment, including the cost of adverse events, clinical outcomes, and disease complications in the BIA cost analysis, were inconclusive.

Feedback from Manufacturers or Their Consultants
The results from the online written survey of 27 participants with an industry perspective are   addition to an aggregated analysis for the whole population. Thirteen participants believed that the target population should be dynamic (open) in BIAs, meaning that patients could be added to or removed from the analysis based on whether they meet the inclusion criteria or not over time. Including off-label indications in the target population assessment was not supported by the majority of participants (74%; n = 20/27 disagree).
Pertaining to the costs, either of the comparators or those included in the base-case, using the LCA in the cost analysis, was acknowledged as appropriate by the majority of the survey participants. Approximately 50% of survey participants disagreed with the inclusion of either indirect or non-healthcare-related costs (e.g., training or introduction cost, transportation, productivity, and caregiver-related costs) or the taxes, for example, harmonized Sales Tax (HST) in BIA. Including the cost transfer from other jurisdictions (where there is a lack of real-world data for the proposed medicine) was not acceptable to 70% of the survey participants. A majority (62%; n = 16/27) of the participants supported including the total and incremental impact on the budget (cost analysis for all new and currently covered indications) in the BIA, but opined that the effects of inflation and discounting should not be included in the BIA.
Most participants disagreed with using probabilistic sensitivity analysis (PSA) in BIA. Scenario or deterministic sensitivity analyses were highly recommended (96 and 85%, respectively). There was support for describing the direction and magnitude of the impact of uncertainty on the overall estimates, but risk-sharing agreements and longer introduction phase were not favored for decreasing model and data uncertainty. The majority of respondents felt that data from manufacturers, clinical data from other jurisdictions, expert opinions, and extrapolating data from similar (or proxy) drug experiences on the payers' database could all be considered reliable sources of data in BIA.
Over 80% of the respondents supported reporting the gross and the net impact on the budget based upon the anticipated sales of the drug of interest for each of the first 3 years after the coverage is granted. There was less support for a schematic representation of the uncertainty analysis (e.g., Tornado diagram). Two-thirds felt that aggregated and disaggregated budget impact results should be reported for each year of the time horizon.
The support was 52% for both cost outcomes being presented separately for different payers and for cost outcomes being presented in monetary units. There was no consensus on whether some cost outcomes should be in natural units (e.g., number of unpaid working days) or not.

Comparative Analysis of Stakeholders' Feedback (Policymaker vs. Pharmaceutical Manufacture Perspectives)
We performed a comparative analysis between the two groups of stakeholders for seven survey questions. Table 3 summarizes the questions that are common in the data obtained from both groups. Figure 1 illustrates all survey results, including the comparative results, between the two groups.
Both groups did not support the inclusion of staff training or introduction costs, nonhealthcare-related costs or taxes in a BIA. Both groups believed that expert opinions and data extrapolation from similar drug in the payers' database are reliable sources of data to be used in BIAs where there is a lack of real-world data for the proposed drug. While 62% of industry participants supported that reporting both the total and incremental impact on the budget for a new medication is important in BIA, the results from policymakers were indecisive (38% neither agreed nor disagreed).

DISCUSSION
In the present study, the Canadian stakeholders' feedback on the BIA recommendations, obtained through qualitative and quantitative methods, provides additional insight to help define BIA guidelines from a Canadian perspective. This information may also be of value for updating or creating BIA guidelines worldwide. The present study arose from the results of the literature reviews describing Canadian, international (e.g., France, Australia, Belgium, Ireland, Brazil, and the United Kingdom) and transnational (e.g., ISPOR) BIA guidelines, and was designed to capture feedback and expert inputs from the stakeholders (policymakers and manufacturers) in the field of pharmaceutical pricing and reimbursement in Canada. 5,24 The authors identified discrepancies between the PMPRB 2007 BIA guidelines and the more recently published or updated BIA guidelines either in Canada or outside. This generated a list of BIA recommendations, which were not included or discussed differently in the PMPRB 2007 guidelines. While the PMPRB BIA guidelines are clearly intended to serve the interests of the people who will be using them, that is, the policy makers, we also obtained inputs from the stakeholders who are tasked with creating the BIA documents for submission to the policymakers. On a positive note, for the items where we were able to obtain common data, there was a  general agreement between these two groups. Nevertheless, had there been disagreement, the BIA guidelines would have to reflect the needs of the policymakers or the BIA submissions; otherwise the guidelines would be unhelpful in the drug reimbursement regime. While there was consensus on many recommendations, some recommendations will need further inputs to determine whether they should be included in an updated version of the PMPRB BIA guidelines, especially involving policymakers from both public and private perspectives (e.g., 3-year or longer time horizon, patient adherence and compliance, cost offsets, reporting total and incremental impact on the budget, and providing a list of reliable databases as data sources). Obtaining stakeholders' feedback was part of the process to create many of the previously published national BIA guidelines. 4,5,9,11 The PMPRB 2007 BIA guidelines were initially developed based on a needs assessment and a literature review and then improved upon with inputs from the NPDUIS Advisory Committee, including drug plan managers from multiple provinces in Canada and a representative from the CADTH. 4 In Poland, the BIA guidelines were initially conducted internally within the Agency for Health Technology Assessment and Tariff System, and then within the Guidelines Update Team. The Guidelines were submitted for public comment and for review by the Minister of Health. 11 In Belgium, the preliminary BIA guidelines were developed based on a literature review and then stakeholders' feedback was obtained involving the Belgian Health Care Knowledge Centre (KCE) and different Belgian stakeholders from both government and industry. 9 In France, as a part of the French BIA guidelines development, a public consultation process was conducted including international expert reviews and approval from the HAS Board and the Economic and Public Health Evaluation Committee of HAS. 5 Unfortunately the stakeholder analyses that were conducted as part of the above-mentioned guidelines were not published with any methodological detail. Our study is unique in terms of: (1) the rigorous study design (mix methods), (2) the scope (including policymakers from both public and private drug plans), (3) inclusion criteria (clear definitions for selecting stakeholders), (4) the one-on-one semi-structured interviews providing a rich description of the stakeholders' opinion on improving the PMPRB BIA guidelines, and (5) publishing the stakeholder analysis in the public domain.
There are a number of conclusions that arise from the results of the feedback. Similar to Pearson and Ghabri et al., we found that using the BIA as an affordability factor for ICER 7 threshold adjustments and price cap estimation, especially in chronic conditions such as hepatitis C virus drugs, is a practical benefit of using BIA in a real-world scenario. 1,25,26 Moreover, considering a BIA cap or threshold received positive feedback in our study. There are some examples for using BIA threshold internationally such as a budget impact test of £20million for NHS England since 2017 27 and a budget impact threshold linked to the growth in the national economy GDP 8 in the United States. 25 Similarly, a real-world reassessment process for BIA results (in the post-market surveillance phase) would be recommended, as already used in the United Kingdom.
One should note that in the Irish 18 and ISPOR 23 BIA guidelines, opportunity costs are defined as the costs that arise when implementing the technology or clinical guidelines that might not be reflected in the "actual costs" at the time of doing a BIA, and that they are different from the opportunity cost definition in policymaking of "whether the improvement in health outcomes that the proposed new drugs offer exceeds the improvement in health that would have been possible if the resources had, instead, been made available for other health care activities." 28 Based on the results, we concluded that, it is not feasible to ask manufacturers to calculate in BIA the opportunity cost of investing in a new drug. Therefore, there is a need for a method to calculate the opportunity cost without relying on the BIA, and an alternative method has been proposed by Ochalek et al. where the opportunity costs in the healthcare expenditures are represented by the threshold value for the CEA. 28 Regarding the time horizon, a minimum of 3 years is favorable in Canada and a time horizon beyond 3 years should be justified by the manufacturer (e.g., for a specific drug, a disease area, or patent duration). In some BIA guidelines, such as in France, 1 ISPOR, 23 and Brazil, 29 there is a range for time horizon, for example, 3-5 years, whereas in the British and Irish guidelines, they introduce a punctual time horizon of 5 years. Further stakeholder feedback is required to reach a consensus on the most favored approach for a BIA time horizon in Canada.
The results show that usage of complicated modeling techniques (e.g., Markov models) is not recommended in BIA, but it is advised that the disease condition should be modeled as much as possible to capture the long-term consequences (at least within the adopted time horizon) associated with using the proposed medication in a chronic condition. Providing either incidence-or prevalence-based (or both) models would help to better understand drug costs related to the acute and chronic conditions. This is especially important for responding to a methodological gap, which was highlighted by Mauskopf et al. 30 Different terminologies are used in different guidelines/countries to define comparators. In the Canadian context, the multidrug treatment strategy for defining comparators is called "strategy-based treatment," which is different from France 1 (treatment set), ISPOR 23 (treatment mix or set), and Australia 19 (treatment mix). In the new version of the PMPRB BIA guidelines, usage of a "treatment mix" is suggested to be the most consistent with international terminology.
One should note that comparator mix does not necessarily always match the comparator mix in the utilization (real world); for example, the treatment mix that private payers and public plans see in their database maybe different because of inherent differences in their formularies. This type of difference between public and private payers should be addressed in BIA. It is also recommended that the choice of comparators in BIA should be consistent with the health EE (e.g., cost effectiveness study) unless there are clear justifications for not taking the same treatment strategy as the health technology assessment (e.g., in the case of non-drug/surgical alternatives). Similar to the PMPRB 2007 BIA guidelines, our study participants confirmed that off-label indications should be considered only in sensitivity analysis (not in base-case analysis). As it is recommended in the Irish 18 and ISPOR 23 BIA guidelines, the catch-up effect (treatment switch) is also recommended in BIA in Canada. Mauskopf et al. 30 raised a methodological gap relating to the "treatment switch" or "drug discontinuation," in the chronic conditions, which might not be appropriately addressed in many published BIAs in the United States.
Similar to the PMPRB 2007 BIA guidelines, cost offsets, that is, costs associated with changes in clinical outcomes, costs associated with clinical consequences/complications (e.g., adverse drug reactions), and resource utilization (e.g., hospitalization, emergency room admission) are still excluded from the analysis. The impacts on indirect costs (e.g., productivity, transport, capacity, and workforce) are not included in a BIA basecase analysis, and cost data from other jurisdictions are not acceptable.
Uncertainty in input data is a general concern in BIA. PSA is used in the Irish and Belgian guidelines, 9,10 whereas it is not recommended in Canada. In contrast, scenario (for structural uncertainty) or deterministic (for input data uncertainty) sensitivity analyses are highly recommended in Canada. A methodological review of US budget impact models 30 showed that sensitivity and scenario analyses presented in published BIAs are "typically too limited to allow a budget holder to assess the likely budget impact for their health plan." We would expect that the gap would be covered by recommending scenario and sensitivity analyses being included in the PMPRB BIA guidelines for dealing with uncertainty. Risk-sharing agreements and longer introduction phase for decreasing the uncertainty in new drug submissions, as they are recommended in Australia 19 and the United Kingdom, 20 are not favored in Canada.
There are a number of limitations to the present study. There was a limited sample size for interviews providing the qualitative data. Nine policymakers agreed to participate in an interview, and the audio data for one participant was lost. Also, two participants from the same province were interviewed at the same time. While the sample size was limited, we were able to obtain data from representatives of different jurisdictions across Canada, and a similarity in their responses was noted suggesting that new themes or ideas might not have been forthcoming even if the sample size was larger. Nevertheless, in order to make any meaningful comparison between the opinions of public and private payers' (subgroup analysis), we would have required substantially larger numbers.

CONCLUSION
We obtained Canadian stakeholders' opinion on a list of recommendations prepared based on a comparative literature review of national and ISPOR BIA guidelines using a mixed-methods approach. A mandate for submitting a companion CEA/CUA along with a BIA in a new drug application in Canada could be a reason for some observed differences between the Canadian stakeholders' perspective and recommendations from other jurisdictions (e.g., ISPOR), especially with respect to the inclusion of cost offsets (e.g., clinical outcomes) and complicated modelling techniques in BIA.
The present study is an integral step towards creating a proposal for updating the PMPRB BIA guidelines. The present study aimed to gain initial Canadian stakeholders' feedback and opinion on potentially new recommendations. A penultimate revised PMPRB BIA guidelines will be developed based on the results of the present study. In Canada, when guidelines are proposed by a government agency or board, there is a mandatory comprehensive consultation process with stakeholders. The results generated by the current study will form the template for the new draft guidelines document that the PMPRB will produce. PMPRB will then conduct a broader consultation with stakeholders than could be achieved in this study. After that step, there will be a final revision and subsequent adoption of updated BIA guidelines.

COMPLIANCE WITH ETHICAL STANDARDS FUNDING
This work was supported by the Mitacs Accelerate research award (IT09958).

CONFLICTS OF INTEREST
Naghmeh Foroutan, Jean-Eric Tarride, Feng Xie, and Bismah Jameel have no conflicts of interest that are directly relevant to the contents of this article. Mitchell Levine is the chair of the PMPRB, and Fergal Mills is a director at Innomar Consulting.

DATA AVAILABILITY STATEMENT
We have provided the required data as much as possible (in a summarized format), and there are no additional data to be shared.

4.
Do you think BIA should take into account the complexity of the disease/condition under study or of the treatment (e.g., acute plus chronic treatments)? Do you prefer to do it through more complex modeling techniques (e.g., Markov models) or sensitivity analysis?

5.
What is your opinion about increasing the time horizon to more than 3 years (e.g., 6 years)? We used Likert-type ordinal scales to rate the responses, ranging from 1 to 5 (where: 1 = "strongly disagree, " and 5 = "strongly agree"), with a middle neutral category (3 = "neither agree nor disagree"). Please answer the questions by choosing a number between 1 and 5.

BIA key elements Comments
Analytical model structure One of the participants believed that "the [BIA] methodology is quite comprehensive in the PMPRB BIA guidelines, but for some parameters, such as time horizon, the guidelines could be updated to a longer period, or just given opportunity to basically consider it for some of the technologies. "

Time horizon
The comment raised by the private payers highlighted the fact that the time horizon should be long enough to capture seasonality in pharmacare provinces where there is a deductible season-a part of the year where there is basically no claims as people in that province are working toward the deductible.

Target population assessment
Two participants mentioned that the target population assessment is always a challenge (e.g., in the hospital setting in Quebec, the billing data for the comparators are not always available for manufacturers and thus it is hard to validate the population size for the BIA calculations for RAMQ 9 ). So, there are always more data available for public drug plans, but not manufacturers, which result in differences in the BIA estimations. Also, there is heterogeneity in the epidemiologic data coming from different sources of literature from different jurisdictions. Altogether, the final population could be uncertain, and sometimes it is hard to find good, relevant Canadian data.
Another participant raised the issue that manufacturers can predict the eligible population. However, they cannot know how many people would actually start that medication even if their physicians have prescribed them the medication.
Moreover, it is good to make it clear in the BIA guidelines as to in which cases the analysis should be incidence-based or prevalence-based or both to make the analysis more consistent.
Furthermore, a concern regarding new (vs. old) indications highlighted the fact that many drugs are indicated in different disease categories, which are all considered in the BIA. The consideration for continuing (or discontinuing) the coverage for some indications should be highlighted in the guidelines. For example, if a drug is commonly used as the second line in a disease category and in the new application, it becomes the first line in a new indication, and then it may not be covered as a second-line therapy anymore.

Comparators
Choice of the comparators is important, which is sometimes consistent with the economic evaluations and sometimes not. Comparator mix does not necessarily always match the comparator mix in the utilization [real world] of the different comparators that private payers see in their database, and that could be due to differences in the public versus private formularies.

Costing
Markups and professional fees are different in public and private plans, and they are recommended to be considered in an interactively updated model template for BIA.
Modeling techniques All submitted models should be transparent, simple, and include confidential prices at the same time. Excel-based electronic models would be better if they have the ability to express results in either contract years or fiscal years. In addition, it was recommended that the incident and prevalent patients and patients coming off the patient excess (if there is one) be shown in the model separately (e.g., in the case of biologics).
Input and data sources It is very difficult to come up with solid parameter estimates, particularly around market growth and market penetration of the new drug, and target population assessment (market size estimation). Uncertainty is a big issue related to input data according to the participants: "there's not really good evidence of what it's going to be so, it is just judgment [subjective]. "

Uncertainty
Robust methods for sensitivity analysis are required to adequately address the issue of uncertainty.
Reporting format Currently, most BIA reports are providing results in a disaggregated format for each year of the time horizon, whereas provinces need BIA reports in fiscal years. It should be helpful to be reflected in the guidelines.