{'fraction_non_alphanumeric': 0.023115741494383852, 'fraction_numerical': 0.007868034076726899, 'mean_word_length': 4.952842377260982, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Cost-effectiveness analysis of PET/CT surveillance imaging to detect systemic recurrence in resected stage III melanoma: study protocol

Introduction In the new era of effective systemic therapies for advanced melanoma, early detection of lower volume recurrent disease using surveillance imaging can improve survival. However, intensive imaging follow-up strategies are likely to increase costs to health systems and may pose risks to patients. The objective of this study is to estimate from the Australian health system perspective the cost-effectiveness of four follow-up strategies in resected stage III melanoma over a 5-year period following surgical treatment with curative intent. Methods and analysis A decision-analytic model will be built to estimate the costs and benefits of (1) 12 monthly, (2) 6 monthly, (3) 3–4 monthly positron emission tomography/CT imaging for 5 years, compared with (4) no imaging follow-up. The model will be populated with probabilities of disease recurrence, test performance measures using data from >1000 consecutive resected stage III melanoma patients from Melanoma Institute Australia diagnosed between 2000 and 2017. Healthcare resource use, including surveillance imaging, doctor’s visits, subsequent tests and procedures to investigate suspicious findings, will be quantified from detailed patient records and valued using Australian reference pricing. Economic outcomes include cost per new distant melanoma recurrence detected and cost per diagnostic error avoided, for no imaging compared with the other strategies. Deterministic sensitivity analyses will examine the robustness of model results. Ethics and dissemination This study was approved by the Sydney Local Health District, Sydney Local Health District Ethics Review Committee (RPAH Zone), AU/1/830638 and the Australian Institute of Health and Welfare (EO2019-1-454). The results of this study will be published in peer-reviewed medical and health economics journals and will inform melanoma management guidelines.

7. Introductionparagraph 1please define age corresponding to the term "young" 8. Introductionparagraph 2please provide reference for sentence three regarding risk of relapse highest within first 3 years after diagnosis/treatment. 9. Page 5, paragraph 1, sentence 1 consider adding "one functionally and anatomically detailed imaging sequence" 10. Consider specifying that "downstream tests", paragraph 2, page 5, include both image-guided biopsies and short-term additional cross-sectional imaging follow-up. 11. Please correct punctuation and grammatical errors.

REVIEWER
Lucy Abel University of Oxford, UK REVIEW RETURNED 24-Apr-2020

GENERAL COMMENTS
This is an interesting study protocol for a decision model that will evaluate the cost-utility of several PET/CT surveillance schedules for patients with stage III melanoma in Australia who have undergone surgical treatment with curative intent. This is an important population, and the authors discuss the need to balance the benefits and harms of surveillance concisely and clearly. The introduction is well written and the decision problem is clearly outlined.
However, I have a number of concerns relating to the proposed model structure that I would urge the authors to think about.
Firstly, It is not clear to me how the varying diagnostic accuracy (DA) associated with the different follow up strategies will be captured in the decision tree. Figure 1 suggests that the reported diagnostic accuracy of each PET/CT performed will be combined to ascertain a single DA for the entire 5 year follow up period, but no detail is provided on how this will be calculated. I have strong reservations about the statistical validity of this approach.
It is also unclear to me how the timing of the recurrence will be captured in the model. A recurrence at 1 year will have different cost-effectiveness implications than one at 4 years in, where a number of TN PET/CTs will already have been conducted. I strongly suggest they make use of a model structure that can account for time effects. The simplest way to do this would be to use a markov model with a 3 month cycle.
The model structure proposed is appropriate for a static diagnostic pathway, but is not appropriate for a screening programme with a strong time component, which is what this amounts to.
My second concern relates to how the harms of follow up are evaluated. The authors reference the potential harm of incidental findings, but this is not included as an outcome in the model. An incidental finding that is not attributable to melanoma will presumably be investigated further, incurring further costs and potential adverse events. The other harm mentioned is the long-term risk of radiation exposure. This is not included in the model, potentially underestimating the harms of imaging.
Likewise adverse events from further investigation are not described in the model, and need to be, along with how they will be incorporated, where applicable, into the QALY outcome.
Finally, the QALY outcome for a false negative result is missing, so I have not been able to evaluate this.
Data: the parameters will be derived from an observational dataset, but how they will be derived is not reported. Reference 42 Is a conference abstract, which I was not able to find. An updated reference is necessary, but additionally a detailed description of the dataset and the analysis that will be used (or has been used) to obtain the diagnostic accuracy, prevalence and survival statistics is also needed.
Minor comments: Abstract: State the country and perspective in the abstract.
Page 5: "incidental findings that are later found to be harmless, causing unnecessary investigations and anxiety" -note that not all incidental findings will later be found to be harmless. Some will be treated and the patient will get better, and it is only by looking at population-level mortality data that we are able to ascertain that some of these cancers must have been harmless. This is of course what makes tackling overdiagnosis so challengingthere is no natural feedback to patients or clinicians. page 6: "Currently there is no high level evidence that early detection and management of recurrence from surveillance imaging improves patient outcomes." define high level.
Page 6: International guidelines refer to different follow-up durations as well as screening intervals. Why is follow up fixed at 5 years?
Model assumptions: Assumption h needs justifyingif the aim is to increase the proportion of surviving patients then this is a key cost How will the model be validated?
Probabilistic SA: Test accuracy typically uses a beta distribution to bound the results between 0 and 1 (being a probability), rather than a normal.
What threshold will be used to judge cost-effectiveness?

Points raised by Reviewer 1
The investigators describe their retrospective study to evaluate the effect of different surveillance strategies on outcomes of melanoma patients. The aim of this work is timely, important and has the potential to have a strong impact on both healthcare economics and melanoma patient outcomes. Understanding that methodology is unlikely to be altered based on this review, I have a few questions for the authors as follows:

1.Please address limitations and how you plan to address bias as the surveillance strategy was not randomized. Do you plan to match patients in each surveillance group? Please clarify.
Response: Thank you for this comment. The reviewer is correct in that data inputs for this model are based on a cohort study rather than a randomised controlled trial. However these are the best available, and contemporary real-world data. We have added this as a limitation in the discussion section, see page 14, paragraph 3 of the track change revision: "The comparison of the three imaging schedules was not based on data from randomised trials and some selection bias might be introduced." 2.Do you plan to assess correlation of imaging with disease burden? Specifically, notably with immunotherapy, pseudo progression is well known to confound PET/CT imaging of melanoma patients and further, imaging not infrequently underestimates disease burden. Please comment. Response: It is unlikely that that data for test performance of the imaging modalities is affected by immunotherapy treatment, as very few in our cohort study were on these systematic treatments between the years 2000-2017. Our cohort underwent surveillance before the widespread use of the more effective systemic treatments now available for patients with metastatic melanoma. In an era where immune-checkpoint inhibitors may induce long-lasting responses in a subset of patients with metastatic disease, inflammation in the short term may interfere with true signs of recurrent disease. Response: Yes the test accuracy of imaging is measured against a reference standard that includes the findings from clinical follow up. We agree with the reviewer that nodal metastasis might be found by PET/CT and physical exam. However, in this model we only focus on distant metastatic disease which is the primary indication for PET/CT imaging. We have changed the term "asymptomatic" and replaced it with "resected" throughout the manuscript.

3.Do you
Patients in all imaging regimens have a clinical visit with a melanoma specialist at the time of each scan. Patients in the "no imaging follow-up" group will have clinical visit every 4 months for the first 3 years, every 6 months in years 4-5.
4. Are you evaluating detection of regional relapse or focusing just on systemic? Throughout you are focusing on "systemic" recurrence but frequently these stage III patients relapse in same or proximal or other additional nodal basin for which detection and treatment might be very relevant to improving melanoma outcomes. Consider modifying this language throughout. Response: In this study we limited the outcome to distant recurrence only, as surveillance imaging is primarily used to identify systemic recurrence. Regional melanoma recurrence (e.g. in the lymph node basin) is often identified by physical examination in contrast to distant disease, which is difficult to detect by physical examinations. Thus PET/CT imaging surveillance is predominantly undertaken to identify distant metastatic disease. We have changed the language to "distant recurrence" throughout the manuscript. imaging for patients who developed recurrence, they did not carry out a formal cost analysis, they estimated the average cost of PET/CT-surveillance strategy and there was no comparison with other strategy. Our study is one of the first cost-effectiveness analyses that compares three different imaging strategies with PET/CT compared to no imaging in resected stage III melanoma patients. We have reworded the statement in the abstract and in the discussion section see page14, paragraph 3.

Response:
We have now added the definition "The value of information analysis is Value of information analysis is a method to estimate the return on investment in proposed research projects." for VOI, see page 13, paragraph 2.

Introductionparagraph 1 -please define age corresponding to the term "young".
Response: We have added the corresponding age and the sentence now reads: "Its incidence is continuing to rise slowly in both sexes, and while mortality and five-year survival (95%) after melanoma diagnosis are stable, the disease remains a leading cause of cancer death in young adults aged under 40 years."

11.Please correct punctuation and grammatical errors.
Response: Thank you, punctuation and grammatical errors have been attend to throughout.

Points raised by Reviewer 2
This is an interesting study protocol for a decision model that will evaluate the cost-utility of several PET/CT surveillance schedules for patients with stage III melanoma in Australia who have undergone surgical treatment with curative intent. This is an important population, and the authors discuss the need to balance the benefits and harms of surveillance concisely and clearly. The introduction is well written and the decision problem is clearly outlined. However, I have a number of concerns relating to the proposed model structure that I would urge the authors to think about.

Firstly, It is not clear to me how the varying diagnostic accuracy (DA) associated with the different
follow up strategies will be captured in the decision tree. Figure 1 suggests that the reported diagnostic accuracy of each PET/CT performed will be combined to ascertain a single DA for the entire 5 year follow up period, but no detail is provided on how this will be calculated. I have strong reservations about the statistical validity of this approach. Response: PET/CT sensitivity and specificity over a 5-year period will be derived from a cohort study we conducted among 332 resected stage III melanoma patients (median follow-up 61 months). This analysis has provided data regarding the diagnostic accuracy of PET/CT at each 3, 6 and 12-monthly follow-up time points. This is mentioned on page 10, paragraph 2. Results from this cohort study will be published and cited in the economic model.

2.
It is also unclear to me how the timing of the recurrence will be captured in the model. A recurrence at 1 year will have different cost-effectiveness implications than one at 4 years in, where a number of TN PET/CTs will already have been conducted. I strongly suggest they make use of a model structure that can account for time effects. The simplest way to do this would be to use a markov model with a 3 month cycle. The model structure proposed is appropriate for a static diagnostic pathway, but is not appropriate for a screening programme with a strong time component, which is what this amounts to. Response: In the first decision analytic model, we are estimating the cost-effectiveness of three imaging follow up strategies at 5-years compared to no imaging follow-up strategy for the outcomes of distant recurrence appropriately diagnosed and treated and diagnostic error avoided. This a static diagnosis pathway of the commonly used PET/CT imaging for the purpose of diagnosing distant melanoma, and information presented in this way is helpful to clinicians when discussing the pros and cons of a surveillance strategy with their patients .
In addition, as suggested by the reviewer, we will build a second longer-term model with a Markov structure containing a 3-month cycle length that allows the accrual of costs and benefits over time taking into account the time of distant recurrence. This model will have quality-adjusted survival as the economic outcome.
3. My second concern relates to how the harms of follow up are evaluated. The authors reference the potential harm of incidental findings, but this is not included as an outcome in the model. An incidental finding that is not attributable to melanoma will presumably be investigated further, incurring further costs and potential adverse events. Response: We agree and in this context the literature around the costs of incidental findings related to routine follow-up imaging and care is scarce. 1 To account for the cost of incidental findings, our model will include the costs of further investigations (e.g. extra scans, diagnostic biopsies) following a false positive result. This is mentioned on page 10, paragraph 3. The adverse events that occur from these extra investigations will be reflected in the short-term utilities assigned to that health state in the Markov model.

The other harm mentioned is the long-term risk of radiation exposure. This is not included in the model, potentially underestimating the harms of imaging.
Response: Radiation exposure from long-term surveillance imaging can cause new primary cancers. Generally these occur beyond 5 years and therefore will not feature in the shorter-term decision analysis. In a sensitivity analysis for the longer term model we will include the costs and disutility associated with a second primary cancer diagnosis and its subsequent treatment.
Likewise adverse events from further investigation are not described in the model, and need to be, along with how they will be incorporated, where applicable, into the QALY outcome. Response: As mentioned above, in the new Markov model that will be built for the finding of the cost per QALY gained, we will include adverse events from further investigations for false positive results, as reported in our previous cohort study. 1 Finally, the QALY outcome for a false negative result is missing, so I have not been able to evaluate this.

Response:
The disutility associated with a false negative result was omitted accidently, however will now be added to the Markov model inputs.
Data: the parameters will be derived from an observational dataset, but how they will be derived is not reported. Reference 42 Is a conference abstract, which I was not able to find. An updated reference is necessary, but additionally a detailed description of the dataset and the analysis that will be used (or has been used) to obtain the diagnostic accuracy, prevalence and survival statistics is also needed. Response: Data for the model parameters (test accuracy, prevalence of distant recurrence and survival) will be taken from a retrospective cohort study evaluating the 5-year test accuracy and survival of 3-monthly, 6-monthly and 12-monthly surveillance imaging. This has been submitted for publication, is currently under review and will be cited in the model manuscript. As soon as the findings from the cohort study have been published we will cite them, for now the findings exist in abstract form as presented at the International Health Economics Association (iHEA) and the ARCS Australia Annual Conference conferences.
Minor comments: 1.Abstract: State the country and perspective in the abstract.
Response: This has been added to the abstract.
2. Page 5: "incidental findings that are later found to be harmless, causing unnecessary investigations and anxiety" -note that not all incidental findings will later be found to be harmless. Some will be treated and the patient will get better, and it is only by looking at population-level mortality data that we are able to ascertain that some of these cancers must have been harmless. This is of course what makes tackling over diagnosis so challengingthere is no natural feedback to patients or clinicians.