{'fraction_non_alphanumeric': 0.024945295404814005, 'fraction_numerical': 0.014573304157549235, 'mean_word_length': 5.174277222372332, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Impact of the pan-Canadian Oncology Drug Review on provincial concordance with respect to cancer drug funding decisions and time to funding

Background The pan-Canadian Oncology Drug Review (p codr ) was implemented in 2011 to address uneven drug coverage and lack of transparency with respect to the various provincial cancer drug review processes in Canada. We evaluated the impact of the p codr on provincial decision concordance and time from Notice of Compliance ( noc ) to drug funding. Methods In a retrospective review, Health Canada’s Drug Product Database was used to identify new indications for cancer drugs between January 2003 and May 2014, and provincial formulary listings for drug-funding dates and decisions between 1 January 2003 and 31 December 2014 were retrieved. Multiple linear models and quantile regressions were used to evaluate changes in time to decision-making before and after the implementation of the p codr . Agreement of decisions between provinces was evaluated using kappa statistics. Data were available from 9 provinces (all Canadian provinces except Quebec), identifying 88 indications that represented 51 unique cancer drugs. Two provinces lacked available data for all 88 indications at the time of data collection. Interprovincial concordance in drug funding decisions significantly increased after the p codr ’s implementation (Brennan-Prediger coefficient: 0.54 pre-p codr vs. 0.78 post-p codr ; p = 0.002). Nationwide, the median number of days from Health Canada’s noc date to the date of funding significantly declined (to 393 days from 522 days, p < 0.001). Exploratory analyses excluding provinces with incomplete data did not change the results. , concordance in cancer drug funding decisions between provinces and decreased time to funding decisions were observed.


BACKGROUND
The advent of new and effective cancer treatments has contributed to declining cancer death rates 1,2 . Cancer care costs are rising, and new pharmaceuticals are an important contributor to that rise 1 . An evidence-based drug review process is a critical step in ensuring access to effective and cost-effective drugs. The recommendations of regional evidence-based cancer care programs affect funding decisions by government organizations 3 . Integrating cost effectiveness analyses into the review process further ensures that taxpayers acquire access to drugs that are cost-effective 4 .
Before 2007, Canadian provinces and territories had separate regional drug review processes to inform their local funding decisions 5,6 . Provincial funding decisions were further affected by individual provincial budgets and priorities 6 . The rising cost of cancer treatments and the increase in financial pressures on the health care system contributed to concerns about cost-effectiveness. Concerns also arose about the variation in coverage from province to province and the appearance of a lack of transparency in how funding decisions were made. For example, bevacizumab for metastatic colorectal cancer was funded starting in January 2006 in British Columbia, but not until April 2009 in Alberta 7 . Furthermore, quick access to effective new treatments is always a concern. Lengthy approval times can result in access delays that cumulatively translate to a loss in patient life-years 8 .
Given similarity in the governance and accountability structures of provincial cancer systems, such as cancer agencies, a collaborative interprovincial initiative for drug evaluation was thought to be possible. Furthermore, given limited access to a small pool of experts and resources and the inefficiency associated with duplication of effort in multiple drug reviews, a pan-Canadian effort was supported 6 . In 2007, the interim Joint Oncology Drug Review (ijodr) was created to facilitate the creation of a single drug review process. The ijodr represented an interim, evaluative process in which one province conducted reviews and shared the results with the other provinces. The other provinces had no obligation to follow the recommendations. Additionally, not all drugs funded by the provinces during the ijodr period were reviewed through the ijodr process.
After an evaluation of the initial interim process, the Conference of Deputy Ministers of Health in 2010 approved the creation of a permanent body. Named the pan-Canadian Oncology Drug Review (pcodr), this formalized national body conducts reviews on behalf of all provinces and territories except Quebec 6 . The pcodr began accepting drug submissions for review in July 2011. On 1 April 2014, administration of the pcodr was assigned to the Canadian Agency for Drugs and Technologies in Health 9 .
The pcodr was one strategy intended to resolve uneven cancer drug coverage across Canada. We hypothesized that, since the creation of the pcodr, the time from a Notice of Compliance (noc) issued by Health Canada for market authorization of the cancer drug to the making of a decision about drug funding in provincial drug programs has shortened, and that more consistent decisions have been made from province to province. Here, we report on how, compared with the pre-pcodr period, the pcodr has affected time from noc to time of funding by provincial drug programs and provincial decision concordance.

METHODS
In a retrospective review, we identified all anticancer drugs (excluding supportive care drugs) and distinct indications with a noc date issued by Health Canada between 1 January 2003 and 31 May 2014. For each province, drug funding decisions and dates of those drug funding decisions between 1 January 2003 and 31 December 2014 were extracted. Indications were categorized as pre-pcodr or pcodr depending on the noc date. Indications with a noc date within the ijodr period were categorized as pre-pcodr because the interim process did not require provinces to follow ijodr recommendations, and many provinces used their existing provincial drug-review processes during the interim period. Submission dates for drug funding consideration to provincial drug-review processes were not used because of heterogeneous data and a lack of consistent data.

Data Sources
Health Canada identified and provided a list of all approved cancer drugs and indications. The publicly available Drug Product Database Online Query maintained by Health Canada (http://webprod5.hc-sc.gc.ca/dpd-bdpp/ index-eng.jsp) was used to manually corroborate all drug and indication entries. Individual provincial ministries of health, cancer agencies, and the pcodr were contacted to obtain the date that funding decisions were made and the specific funding decisions by individual provinces. Contact with the provincial ministries of health and cancer agencies was facilitated through the pcodr and the pcodr Provincial Advisory Group. To confirm accuracy, all available data were corroborated with two separate data extractions by the provinces.

Data Collection Process
Health Canada provided a list of all drugs and associated indications classified as antineoplastic and immunomodulatory. Veterinary entries were excluded. That dataset was manually reviewed against the Drug Product Database Online Query database to extract the noc dates for each unique indication.
A prospectively defined electronic data extraction sheet was used by two authors (AS, NP) to independently extract the required data. A third author (HM) resolved any discrepancies. Duplicate entries were removed so that the final data set included one unique noc date per drug and unique indication. The electronic drug monograph available through the online Drug Product Database was used to corroborate drug indications. Duplicate noc entries were removed for entries involving various strengths of drugs, non-cancer drugs and indications, discontinued drugs, and replicated entries for changes in manufacturer or manufacturing processes (or both). It was not possible to link an individual indication to a specific noc date for all older drugs. Therefore, drugs with a first noc date before 1 January 2003 were excluded.

Variable Definitions
Extracted variables included the generic drug name; indication; province; noc date; submission period (pre-pcodr: January 2003 to June 2011; pcodr: July 2011 to present); tumour group; route of administration; review committee recommendation date (initial); review committee recommendation date (final); review committee recommendation (recommend funding, recommend funding with conditions, do not recommend funding); notice to implement date (for pcodr-reviewed drugs only); provincial funding status (funded, not funded, under provincial consideration, under manufacturer negotiation); provincial decision date; provincial funding status date; if applicable, revised decision dates; provincial funding criteria; and the review process used for the drug and indication of interest.

Statistical Analyses
Provincial identifiers were removed and results were anonymized before analysis. Anonymization was necessary to obtain interprovincial collaboration. Descriptive statistics are used to summarize characteristics of evaluated drugs and indications. The times from the noc date to funding dates were calculated in calendar days and are summarized using descriptive statistics. A mixed-effects multiple linear model was constructed to examine the effect of the pcodr on time to decision-making, adjusting for the effect of drugs and the effect of provinces. The effect of drugs was modelled as a random intercept, because that effect could be correlated between the provinces; the association between time to funding and province was examined in a fixed-effect model, because the provinces included in our study represented all the provinces that were participating in the pcodr process at the time of the study. Assumptions of normality were assessed based on residual plots, and a slight positive skewness was observed. Because time to funding was expected not to be perfectly symmetrically distributed and might show a "right tail" for some decisions that might take substantially longer to make, median regressions and quantile regressions were also performed to examine the effect of the pcodr on time to decision-making over a range of the percentile time to decision-making (for example, 25th percentile, median, 75th percentile). The usefulness of quantile regression is that it allows for an examination of whether the effect of the pcodr on time to funding was restricted to the extreme outliers with a very long time to funding (the extreme end of the quantile range) or was generalizable to a broad range of drugs that might have had shorter times to funding. Missing data were handled with list-wise deletion.
Agreement of decisions in the provinces before and after implementation of the pcodr was examined by using the methods of Brennan-Prediger to account for multiple raters (that is, multiple provinces) in computing agreement kappa statistics 10 .
In exploratory analyses, provinces with incomplete datasets were excluded. Statistical analyses were performed using the SAS (version 9.4: SAS Institute Cary, NC, U.S.A.) and R software applications (version 3.2.1: The R Foundation for Statistical Computing, Vienna, Austria). Statistical significance was 2-sided and defined as p < 0.05, with confidence intervals (cis) provided when relevant. Given the relatively small number of predefined outcomes that were analyzed, no corrections were made for multiple significance testing.

Drugs Included in the Analysis
Health Canada identified 2211 noc dates and cancer drugs. Excluding duplicate entries, non-cancer indications, discontinued drugs, and drugs with a first noc date before 1 January 2003, 88 cancer indications comprising 51 unique drugs remained (Figure 1). Table i lists the characteristics of the drugs.

Characteristics of Available Provincial Data
In the 9 provinces, the availability of funding data (Table ii) and funding dates varied. Two provinces (id 6 and 8) lacked data for all 88 indications identified at the time of data collection, because a centralized process for recordkeeping was not available. Missing data for those two provinces all came from the pre-pcodr period and predominantly related to intravenously administered drugs.

Impact on Provincial Funding Status Over Time
Since the introduction of the pcodr, the concordance of drug funding decisions between the provinces has significantly increased [Brennan- With missing data censored, 14 of 52 indications in the pre-pcodr period were unanimously funded (27%), and 5 indications were unanimously not funded (10%). After implementation of the pcodr, 19 of 36 indications (53%) were unanimously funded, and 2 indications (6%) were unanimously not funded. With the implementation of the pcodr, the proportion of unanimous decisions increased (37% vs. 60%, p = 0.048)-a result that was corroborated with exploratory analyses excluding provinces 6 and 8 (38% vs. 94%, p < 0.001).
Although greater concordance was observed during the pcodr period, discrepancies between the provinces in the percentage of drugs funded remained (Table ii). During the pcodr period, the proportions of funded drugs were relatively similar in most provinces, except for one outlier (province 7). Excluding the outlier, the proportion of drugs funded before the pcodr varied from 58% to 85%. After the pcodr, the proportion of drugs funded varied from 50% to 72%. Taking into consideration indications still in the review process or under consideration in the provinces after drug-review recommendations, no statistically significant change was observed in the proportion of drugs funded at a provincial level (66% pre-pcodr vs. 73% pcodr, p = 0.10).

Impact on Funding Timelines Over Time
The availability of drug funding dates was limited for some provinces (Tables iii). Large variations between the provinces were observed for the time from the noc date to the funding date (Table iii). Extreme values were more likely to be identified for provinces with limited data availability. Despite the variations, a statistically significant reduction in the number of days to funding was observed after introduction of the pcodr (Table iv). After adjustment for province and indication, the mean reduction in time from noc date to funding date was to 497 days from 768 days, a reduction of 270 days (95% ci: 89 to 453 days; p = 0.004). Interaction testing confirmed that the effect of the pcodr on reduction in time to funding did not uniformly affect all provinces (p = 0.01). Exploratory analyses excluding provinces 6 and 8 produced no change in the results (mean reduction in time from noc date to funding date: to 489 days from 752 days, a reduction of 263 days; 95% ci: 40 days to 446 days; p = 0.005) and a similar interaction effect.
Nationwide, a significant decline occurred in the median number of days from Health Canada's noc date to the date of provincial funding (to 393 days from 522 days, p < 0.001), which remained present in exploratory analyses excluding provinces 6 and 8 (to 432 days from 587 days, p < 0.001).
The results from quantile regressions suggested that establishment of the pcodr had the most influence on submissions that traditionally tended to take a relatively longer time to fund, but also had a meaningful influence on submissions that took about the median time to fund and on some submissions that took less than the median time to fund (Figure 2). For example, the 75th percentile number of days (that is, for submissions that tended to take relatively longer to fund) was reduced by 143 days (p < 0.001) from the noc date to the funding date. In contrast, the 25th percentile number of days (that is, for submissions that tended to have relatively shorter time to fund) was not reduced from the noc date to the funding date (p = 0.4). The median (that is, the 50th percentile) number of days from the noc date to the funding date was reduced by 155 days (p < 0.001).
For 52 of the indications identified in the analysis, provincial funding decisions were made before a noc was issued for the indication: 50 in the pre-pcodr period and 2 in the pcodr period.

DISCUSSION
After implementation of the pcodr, Canada experienced greater concordance in cancer drug funding decisions in the provinces and a decline in the time to funding by about 9 months overall.
In some instances, the time to funding was negative. Those instances included n decisions by provinces, based on emerging evidence, to fund a specific drug for a new indication if a noc already existed for the drug for a different indication 6 ; n overwhelming clinical need for the drug based on evidence for the new indication; and n expanded eligibility not considered in an original drug submission.
In most provinces, time-to-funding decisions exceeding 4 years were identified; however, such decisions occurred before implementation of the pcodr. Those very long times could reflect n varying committees and review processes in place at the provincial level to approve drug funding; n a choice by some tumour site groups potentially not to put forth a drug review submission until the provincial fiscal climate was one that would be able to fund the drug of interest; n the possibility of a resubmission after the initial review was negative; and n the negotiation process that might take place with manufacturers with respect to cost.
To increase accessibility and equity of access to cancer drugs throughout Canada, strategies to facilitate faster price negotiation processes and greater concordance between the provinces with respect to time to funding continue be explored by provincial ministries of health and cancer agencies. Such strategies include n ongoing development of mechanisms to increase the efficiencies of the pcodr process, such as early conversion mechanisms (a step in the pcodr process that allows assessment of feedback on an initial recommendation to determine whether, in limited  circumstances, it is eligible for conversion to a final recommendation without reconsideration by the committee, which could shorten the time to recommendation by about 30 days) and checkpoint meetings between manufacturers and the pcodr (to ensure that data are complete and clear, preventing unnecessary administrative delays). n clear prioritization, in which indications classified as "conditional recommendation" should be considered first, because no guidance is provided to the provinces at present. Most "conditional recommendations" were related to conditions of "improvement in costeffectiveness," but had varying levels of supportive evidence and of magnitude of net clinical benefit. Ideally, submissions that have high level of evidence and a high magnitude of net clinical benefit would be streamed first to optimize benefits to patients earlier, instead of a "first in, first out" process.
Despite the improvements seen after the pcodr, the average time to funding is still more than a year from the noc date, and variability between the provinces remains. The publicly available annual metrics from the pcodr confirm that the time from submission date to notice to implement final recommendation date is a median of 141 days 11 . Although the drug review process provides funding recommendations, final funding approval is a provincial decision that accounts for each province's needs and financial constraints at a given time. The variations in funding decisions that persist throughout the country are reflective of the different resources available to each province and of the independence and autonomy that provinces have with respect to funding decisions. Although the provinces show regional differences (variations in size, resources, patient populations, and process), the pcodr has provided a mechanism to offer evidencebased information to all provinces and has enabled more consistent funding decisions.
Because provinces are obligated to keep the prices and details of agreements with drug manufacturers confidential, product listing agreements and prices paid by provinces are not available publicly. Therefore, it is unknown whether a concurrent increase in affordability has occurred, or indeed how cost-effective the decisions made by the provinces about cancer drugs are. Given that evidence demonstrates a decreasing absolute benefit of cancer treatments over time with increasing price, cost-effectiveness might have decreased 12 . The pan-Canadian Pharmaceutical Alliance (http://www.pmprovincesterritoires.ca/en/ initiatives/358-pan-canadian-pharmaceutical-alliance) was established in 2010 to lower drugs costs and to increase affordability. How the negotiations conducted by the Alliance have affected time to funding is unclear.
Our study has limitations, including a lack of complete datasets from all provinces. However, exploratory analyses excluding the two provinces with missing data demonstrated consistent results. The rationales for discordance in funding status and review recommendations were not available. A separate study has evaluated the causes for discrepancies 13 . Systematic identification of the rationales for discordant funding decisions can help to tailor strategies to improve equitable access throughout Canada. Furthermore, a lack of access to drug prices and cost-effectiveness analyses at the time of funding decisions prevented an evaluation of the associations between decisions about cost and funding and assessments of cost-effectiveness and provincial economic impact. The establishment of the pan-Canadian Pharmaceutical Alliance might also be a confounder. Lastly, the lack of complete dates for funding status limited our time-to-funding analyses; we could not fully analyze time to decisionmaking (either positive or negative) without the dates of negative decision-making for each of the provinces before and after the establishment of the pcodr (as opposed to the negative recommendations by the pcodr). However, most of the available funding status dates reflected positive funding decisions.
Although a limited number of studies have looked at cancer drug approval timelines outside Canada, no similar studies have been conducted within Canada. Given that other jurisdictions make health care decisions nationally, our study addresses the unique challenge of harmonizing the cancer drug funding decision-making process in the context of multiple provincial or local jurisdictions.

CONCLUSIONS
Implementation of the pcodr has resulted in greater concordance in cancer drug funding decisions between the provinces and a decline in the time to funding decisions.

CONFLICT OF INTEREST DISCLOSURES
We have read and understood Current Oncology's policy on disclosing conflicts of interest, and we declare that we have none. MS was the executive director of the pcodr at the time of the study.