{'fraction_non_alphanumeric': 0.02374765410711708, 'fraction_numerical': 0.00960011549011116, 'mean_word_length': 4.97176724137931, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Models predicting survival to guide treatment decision-making in newly diagnosed primary non-metastatic prostate cancer: a systematic review

Objectives Men diagnosed with non-metastatic prostate cancer require standardised and robust long-term prognostic information to help them decide on management. Most currently-used tools use short-term and surrogate outcomes. We explored the evidence base in the literature on available pre-treatment, prognostic models built around long-term survival and assess the accuracy, generalisability and clinical availability of these models. Design Systematic literature review, pre-specified and registered on PROSPERO (CRD42018086394). Data sources MEDLINE, Embase and The Cochrane Library were searched from January 2000 through February 2018, using previously-tested search terms. Eligibility criteria Inclusion required a multivariable model prognostic model for non-metastatic prostate cancer, using long-term survival data (defined as ≥5 years), which was not treatment-specific and usable at the point of diagnosis. Data extraction and synthesis Title, abstract and full-text screening were sequentially performed by three reviewers. Data extraction was performed for items in the CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies checklist. Individual studies were assessed using the new Prediction model Risk Of Bias ASsessment Tool. Results Database searches yielded 6581 studies after deduplication. Twelve studies were included in the final review. Nine were model development studies using data from over 231 888 men. However, only six of the nine studies included any conservatively managed cases and only three of the nine included treatment as a predictor variable. Every included study had at least one parameter for which there was high risk of bias, with failure to report accuracy, and inadequate reporting of missing data common failings. Three external validation studies were included, reporting two available models: The University of California San Francisco (UCSF) Cancer of the Prostate Risk Assessment score and the Cambridge Prognostic Groups. Neither included treatment effect, and both had potential flaws in design, but represent the most robust and usable prognostic models currently available. Conclusion Few long-term prognostic models exist to inform decision-making at diagnosis of non-metastatic prostate cancer. Improved models are required to inform management and avoid undertreatment and overtreatment of non-metastatic prostate cancer.

The authors aimed to propose a systematic revue of prognostic models for newly diagnosed primary nonmetastatic prostate cancer. I have major concerns.
1) The choice of the authors is to focus the review on tools for predicting patient survival. Nevertheless, this restrictive choice is not explained in the title or in the last paragraph of the introduction related to the objective of the study. I expected that all the predictive models in patients with prostate cancer will be considered. The title and the introduction should be clarified and justified to better understand this restriction.
2) I doubt on the usefulness to predict the mortality for medical decision making in the context of prostate cancer. Indeed, for this population with a large part of elderlies, an important question is to predict the risk of disease progression before death. When this risk is low, active surveillance may be better compared to the other possible treatments/surgeries. Therefore, I believe that the inclusion of all the manuscripts related to predictive models in this review should have been important, not only the manuscripts related to patient survival.
3) The authors concluded that two models are usable: the UCSF CAPRA score and the Cambridge Prognostic Groups. Nevertheless, the word "usable" is confounding. A score should be used if it is useful for the physicians and/or the patients. But the utility of a predictive model is not proportional to its prognostic capacities (discrimination and calibration). It has been recently reported in prostate cancer for the CAPRA scoring system (Lorent et al. Meta-analysis of predictive models to assess the clinical validity and utility for patient-centered medical decision making: application to the CAncer of the Prostate Risk Assessment, BMC Med Inform Decis Mak. 2019 Jan 7;19(1):2). The discussion should integrate this important difference.
4) The section related to results is very short, especially compared to the discussion. More details on the methodological aspects will be interesting, especially the pitfalls (interaction testing, overfitting, technique for internal validation, sample size, censoring consideration in the C-index computation, etc.).

GENERAL COMMENTS
This is an interesting review, though quite closely related to a plethora of others which have been published in this area.
In regards to the research question, this is not clearly articulated in the introduction of the paper (though it appears a little clearer in other parts of the paper/protocol).
If the objective is to identify models which predict if active treatment will be more effective (in regards to survival) compared with surveillance, this should be clearly stated.
If the objective is to identify models which predict which active treatment is most effective (in regards to survival) for a particular patient, this should be stated. Each of these scenarios will include/exclude different papers in the review process. As it stands, the authors appear to be addressing the second question above, yet include models which do not allow for such a prediction.
The protocol has been revised, and it is not clear to me is the document attached matches that uploaded to PROSPERO. The changes appear to have taken place very late in the piece, possibly indicating a post-hoc revision.
The paper is generally well written, but a recurring grammatical error relating to the use of numerals could be corrected throughout.
A number of statements in the discussion could be more fully referenced. e.g. page 14, lines 17, 21. The statement on line 42 'most recent' seems incorrect -there are many other reviews.

Alyson Mahar
University of Manitoba, Canada REVIEW RETURNED 11-Feb-2019

GENERAL COMMENTS
This is a well-written systematic review describing the landscape of prognostic tools for non-metastatic prostate cancer. The approach was originally published as a protocol and the methods are clear and appropriate and cite guidance documents for best practices. The results are comprehensively reported. Many systematic reviews have been performed on this specific topic and many others in cancer prognostic tools in general. The added value of this study to other reviews is not evident for tool developers or tool users, especially in light of new guidelines for tool development and reporting have recently been published and likely have not yet had time to take effect and the lack of clinical discussion to guide users of these tools Major Comments: The potential impact of this paper and its target audience are unclear. Many systematic reviews have already been performed on this topic, as summarized by the authors (both in prostate cancer and cancer in general), and is unclear what this paper adds to the existing literature documenting the poor methodological quality of prognostic and predictive tools for prostate cancer, and cancer in general. The authors do not report how many new studies are identified that have yet to be summarized in previous reviews. The authors conclusions appear to be similar to those of the previously completed reviews. The authors state that this review better focuses on the clinical decision prostate cancer patients and clinicians face, yet this is unsubstantiated in the contents of the introduction and discussion. Therefore how this study's findings will do better than the existing literature in improving the quality of work performed or increasing/decreasing the uptake of tools (whichever is appropriate) in clinical practice is unclear. The article represents a systematic review of the available tools predicting survival endpoints from localized or locally advanced PCa. With a review of all models published since 2000, Authors concluded that only few long-term prognostic models exist, especially taking into account treatment effect, which is supposed to be the most interesting prediction in the current era of active surveillance vs minimally invasive prostatectomy. The article is clear and well depicted from a statistical point of view, conclusions are adequate.
We are grateful for these positive comments from Reviewer 1.
Only few points should be further addressed or commented: R1.2 Since diagnostic (i.e. mpMRI) and therapeutic advances are extensively evolving, I believe that the survival endpoint is evolving as well: which could be the role of previous models in the current scenario?
We agree with this point, and have expanded upon it in the discussion (page 16 lines 370-374). However, we also make the point that it is a necessity to use relatively historic cohorts if we are to use long-term survival outcomes, which do provide a less equivocal endpoint of survival itself.

R1.3 Tables 3 and 5 lack of CI
Wherever they were available 95% confidence intervals have now been added to both tables 3 and 5.

R1.4 When writing "
Heterogeneity between studies meant that attempts to meta-analyse data would not be appropriate" does it mean that you attempted a meta-analysis? Why did you find this heterogeneity? Have you considered the meta-analysis the whole models or the one predicting OS apart from the ones predicting prostate cancer specific mortality?
Thank you for this query. There was heterogeneity in composition of models, modelled endpoints, and statistical assessment of models -for example some studies used the concordance statistic whereas others used area under the ROC curve, and others failed to report any measure of discrimination. It would not have been possible to meta-analyse these results, and on seeing these data, no attempts were made to do so. This has been clarified in the text also (page 7 lines 241-243).
Reviewer: 2 The authors aimed to propose a systematic revue of prognostic models for newly diagnosed primary nonmetastatic prostate cancer. I have major concerns.
R2.1 The choice of the authors is to focus the review on tools for predicting patient survival. Nevertheless, this restrictive choice is not explained in the title or in the last paragraph of the introduction related to the objective of the study. I expected that all the predictive models in patients with prostate cancer will be considered. The title and the introduction should be clarified and justified to better understand this restriction.
We agree the title and objective could have made this point more clearly. The title, and the objectives (lines 140-142) have been amended to clarify this point.
R2.2 I doubt on the usefulness to predict the mortality for medical decision making in the context of prostate cancer. Indeed, for this population with a large part of elderlies, an important question is to predict the risk of disease progression before death. When this risk is low, active surveillance may be better compared to the other possible treatments/surgeries. Therefore, I believe that the inclusion of all the manuscripts related to predictive models in this review should have been important, not only the manuscripts related to patient survival.
The reviewer makes a very valid point, that many prostate cancer patients are elderly and therefore avoiding treatment may be preferable in many cases. However, using disease progression as an endpoint for a prognostic model is problematic, as it is a relatively equivocal endpoint. Biochemical recurrence is recognised to be a poor surrogate for mortality, and metastasis as an endpoint relies upon imaging usage which is highly variable. Other reviews have previously sought to catalogue all predictive models, which we cite to in our discussion (page 15 lines 311-315) for the readers reference. This review is deliberately focussed on the unequivocal endpoint of mortality as explained in the manuscript.
R2.3 The authors concluded that two models are usable: the UCSF CAPRA score and the Cambridge Prognostic Groups. Nevertheless, the word "usable" is confounding. A score should be used if it is useful for the physicians and/or the patients. But the utility of a predictive model is not proportional to its prognostic capacities (discrimination and calibration). It has been recently reported in prostate cancer for the CAPRA scoring system (Lorent et al. Meta-analysis of predictive models to assess the clinical validity and utility for patient-centered medical decision making: application to the CAncer of the Prostate Risk Assessment, BMC Med Inform Decis Mak. 2019 Jan 7;19(1):2). The discussion should integrate this important difference.
Thank you for highlighting this very important point. We agree this point could have been made more clearly, and that at times through this publication we had confused the terms clinical usability and availability. We have now amended the wording throughout, and have added a section differentiating clinical validity from utility(page 17 lines 419-424). This particular recent publication is very relevant to this review, and we have incorporated it accordingly here -we thank the reviewer for highlighting this recent publication.
R2.4 The section related to results is very short, especially compared to the discussion. More details on the methodological aspects will be interesting, especially the pitfalls (interaction testing, overfitting, technique for internal validation, sample size, censoring consideration in the C-index computation, etc.).
Much of the results are currently summarised within tables 3-6, such that the prose section on results has been kept relatively short. At the reviewer's suggestion we have now expanded the results section slightly, and unpacked some of methodological aspects they mention.
Reviewer: 3 This is an interesting review, though quite closely related to a plethora of others which have been published in this area.
R3.1. In regards to the research question, this is not clearly articulated in the introduction of the paper (though it appears a little clearer in other parts of the paper/protocol).
Our objective at the end of the introduction has been amended to match the main research question described elsewhere in the paper and the study protocol. As per our response to R2.1 the study title and introduction have been amended also.
R3.2. If the objective is to identify models which predict if active treatment will be more effective (in regards to survival) compared with surveillance, this should be clearly stated.
If the objective is to identify models which predict which active treatment is most effective (in regards to survival) for a particular patient, this should be stated.
Each of these scenarios will include/exclude different papers in the review process. As it stands, the authors appear to be addressing the second question above, yet include models which do not allow for such a prediction.
The intention in this study was to identify and review models that are usable at the point of diagnosis -i.e. before a patient has committed to a single treatment strategy. This is the reason treatmentspecific models were excluded. Pre-treatment survival models, with or without incorporated treatment effects will help patients and clinicians to consider whether treatment may be of benefit. As the reviewer will be aware, it remains uncertain whether one treatment is more effective than another in the management of non-metastatic prostate cancer. Particularly in low-and intermediate-risk disease, following the results of the ProtecT study -which is mentioned in the discussion. Ultimately, randomised trial data should be used to help answer this question, or treatment coefficients from randomised studies incorporated into prognostic models.
R3.3. The protocol has been revised, and it is not clear to me is the document attached matches that uploaded to PROSPERO. The changes appear to have taken place very late in the piece, possibly indicating a post-hoc revision.
Thank you for highlighting this point and prompting us to re-check the online records. A comparison has been made between the latest version of the protocol (v0.6 November 2018) and the information in PROSPERO. Please note that there is an updated PROSPERO entry for 27 th November 2018 (viewable at the very base of the page) -by default the information presented on PROSPERO is the very original submission dated 22 nd January 2018. The protocol submitted herewith matches the November update. The amendments to the protocol, made following its original upload, are recorded transparently within the amendments section of the Protocol. These were made prior to full-text screening, and informed by the abstract screening process. Note also that the inclusion dates for studies was extended from mid-January to end of February 2018, given the delays to the review's progress. We apologise for the confusion caused here. Please be assured, all of the full-text screening has taken place subsequent to the final protocol amendments in November 2018, without subsequent post-hoc changes.
R3.4. The paper is generally well written, but a recurring grammatical error relating to the use of numerals could be corrected throughout.
We thank you for this positive comment. The grammatical error with regards to numerals is not made clear. We believe it may be an inconsistency in use of numerals or words to describe values. We have now gone through the manuscript to ensure numbers are written as text if they start a sentence, but otherwise the numbers are used.
R3.5. A number of statements in the discussion could be more fully referenced. e.g. page 14, lines 17, 21. The statement on line 42 'most recent' seems incorrect -there are many other reviews.
Additional references have been added against these statements, citing relevant publications. This is a well-written systematic review describing the landscape of prognostic tools for non-metastatic prostate cancer. The approach was originally published as a protocol and the methods are clear and appropriate and cite guidance documents for best practices. The results are comprehensively reported. Many systematic reviews have been performed on this specific topic and many others in cancer prognostic tools in general. The added value of this study to other reviews is not evident for tool developers or tool users, especially in light of new guidelines for tool development and reporting have recently been published and likely have not yet had time to take effect and the lack of clinical discussion to guide users of these tools Thank you for these positive comments. We would make the additional point that this review has a particular focus on survival models available before any treatment decision, which is unusual in the literature. To our knowledge, it is also the first review in the field of prostate cancer to apply the newly described PROBAST risk assessment tool -specifically designed for prognostic models.
Major Comments: R4.2. The potential impact of this paper and its target audience are unclear. Many systematic reviews have already been performed on this topic, as summarized by the authors (both in prostate cancer and cancer in general), and is unclear what this paper adds to the existing literature documenting the poor methodological quality of prognostic and predictive tools for prostate cancer, and cancer in general. The authors do not report how many new studies are identified that have yet to be summarized in previous reviews. The authors conclusions appear to be similar to those of the previously completed reviews. The authors state that this review better focuses on the clinical decision prostate cancer patients and clinicians face, yet this is unsubstantiated in the contents of the introduction and discussion. Therefore how this study's findings will do better than the existing literature in improving the quality of work performed or increasing/decreasing the uptake of tools (whichever is appropriate) in clinical practice is unclear.
We would disagree with some of the comments in this point. Additional explanation of how this review is usefully focussed at the pre-treatment point in the cancer pathway has been added (Page 15 lines 338-341). The review includes two new models by Kerkmeijer et al., and Gnanapragasam et al., which have not been summarised in previous reviews, and this point has now been made clear. Indeed, neither the CPG development or validation paper have been summarised in previous reviews, and are very contemporary with the validation published in 2018 (page 16 lines 360-363).
Minor Comments: R4.3. The authors have selected tools that report >5 year survival. It is not clear how that cut-point was selected, as most studies would look for 10-year survival in this particular cancer site, as 5-year survival in prostate cancer is nearly 100% and likely not a meaningful outcome.
The reviewer makes a very valid point that early prostate cancer survival outcomes are very good, up to and in excess of 5 years. Indeed this point is made on page 16 lines 376-378 of the discussion. The minimum threshold was set this low to identify the maximum possible number of publications.
R4.4. Similarly, as the authors hope to address patient and physician decision-making, it is unclear how this information would be used/weighted against other outcomes such as complications, recurrence, etc. Are these tools even needed/useful for clinicians? Would they be used even if there were more of them, considering the outcome?
This is a similar point to that made in R2.2. and the answer would be similar. Shorter term outcomes are problematic for a number of reasons. Whether prognostic tools are needed or useful for clinicians is a slightly separate issue, which does not appear to be very well explored in the literature in relation to prostate cancer models. Anecdotally, we find that local clinicians are keen to find web-based tools that help inform the clinical consultation. If a high quality, well-validated, model were to be demonstrated I suspect it would be widely used.
R4.5. The inclusion and exclusion criteria of the search are not clear. Why were studies of single treatment arms excluded? Why weren't screened or unscreened populations included? There is redundancy in the listed inclusion/exclusion criteria (eg included non-metastatic, exclude metastatic).
As per the answer to R3.2 models exploring outcomes following a single treatment were excluded as these are of less value to a patient diagnosed with prostate cancer deliberating between treatment or management strategy. Models exclusively exploring post-prostatectomy or post-radiotherapy outcomes have also been explored thoroughly previously, as are referred to in the discussion -with additional references now included as per R3.5. Models derived from both screened and unscreened cohorts were included. We agree that in retrospect there is redundancy within the eligibility criteria -if the editor wishes we could remove the duplication within these, as per the reviewers' suggestion. However, we would ordinarily prefer to publish the criteria as per the protocol, rather than making retrospective changes.
R4.6. Reporting whether or not a tool was available online or not is not meaningful without understanding first if it was well done (includes the right prognostic factors, addresses an important clinical question, is validated and of high quality) and should be available online, and vice versa.
This is an important distinction and ties in to the comment made in R2.3 -as answered above.
R4.7. Do the tools do better/worse in particular sub-populations? How good are they at predicting in stage III vs stage I? In those who receive prostatectomy vs watchful waiting? This information might be helpful.
Further detail has been added to the results section, unpacking the results summarised in Tables 3-6. Some lines have been added to the results to address some of these points (Page 8 lines 252-254).
The relative performance of one model compared to another within treatment sub-groups would be better-answered by one of the referenced reviews that summarised models for these clinical situations.

FORMATTING AMENDMENTS (if any)
Required amendments will be listed here; please include these changes in your revised version: F1. Patient and Public Involvement: We have implemented an additional requirement to all articles to include 'Patient and Public Involvement' statement within the main text of your main document. Please refer below for more information regarding this new instruction: Authors must include a statement in the methods section of the manuscript under the sub-heading 'Patient and Public Involvement'. This should provide a brief response to the following questions: How was the development of the research question and outcome measures informed by patients' priorities, experience, and preferences?
How did you involve patients in the design of this study?
Were patients involved in the recruitment to and conduct of the study?
How will the results be disseminated to study participants?
For randomised controlled trials, was the burden of the intervention assessed by patients themselves?
Patient advisers should also be thanked in the contributorship statement/acknowledgements.
If patients and or public were not involved please state this.
A short paragraph describing our patient and public involvement has been included on page 6 lines 197-200 F2. Please re-upload your Supplementary files in PDF format.
Thank you for this reminder. These have now been uploaded in PDF format accordingly.
F3. The in text citation for " Figure 2" is missing in your main text of your main document file. Please amend accordingly.
An in text citation has been added on page 13 line 279 accordingly.

GENERAL COMMENTS
The authors aimed to propose a systematic revue of prognostic models for newly diagnosed primary nonmetastatic prostate cancer. Regarding the answers, I have one remaining minor comment.
1) The choice of the authors is to focus the review on tools for predicting patient survival. Nevertheless, this restrictive choice is not explained in the title or in the last paragraph of the introduction related to the objective of the study. I expected that all the predictive models in patients with prostate cancer will be considered. The title and the introduction should be clarified and justified to better understand this restriction.
We agree the title and objective could have made this point more clearly. The title, and the objectives (lines 140-142) have been amended to clarify this point Your title is now "Survival models …" Survival model refers to regression, not to prediction. You should prefer a title like: "Models for predicting survival …"

GENERAL COMMENTS
The paragraph "Principal findings" at the beginning of the discussion is slightly misleading. This seems to state that the tools identified can help predict which treatment will have the best outcome, but this is not possible with the studies identified. and/or similar topics and how this publication will help move the field forward or contribute to clinical practice. However, there are no major methodological concerns with the paper.
The reviewer raises no additional amendments. We thank them for their time and help on this publication.