{'fraction_non_alphanumeric': 0.019722814498933903, 'fraction_numerical': 0.006716417910447761, 'mean_word_length': 5.27072192513369, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Building Reality Checks into the Translational Pathway for Diagnostic and Prognostic Models

There has been a significant increase in the number of diagnostic and prognostic models published in the last decade. Testing such models in an independent, external validation cohort gives some assurance the model will transfer to a naturalistic, healthcare setting. Of 2,147 published models in the PubMed database, we found just 120 included some kind of separate external validation cohort. Of these studies not all were sufficiently well documented to allow a judgement about whether that model was likely to transfer to other centres, with other patients, treated by other clinicians, using data scored or analysed by other laboratories. We offer a solution to better characterizing the validation cohort and identify the key steps on the translational pathway for diagnostic and prognostic models.


Background
The last decade has seen a marked increase in diagnostic and prognostic prediction models in medicine. The growth of interest in biomarkers, personalized and stratified medicine, coupled with the rise of machine learning and statistical algorithms aiding the building of such diagnostic and prognostic tools 1-3 has led to a proliferation of prediction models in recent years -see Supplementary Figure 1. However, the quality and reporting of studies to support such prediction models is often poor 4 prompting the development of reporting guidelines for studies of observational data 5 , tumour markers 6 , diagnostic accuracy 7 , genetic risk prediction 3 , and multivariable prediction models 4 -the STROBE, REMARK, STARD, GRIPS and TRIPOD guidelines respectively. In particular, the TRIPOD guideline offers specific recommendations around studies developing or validating prediction models 4 .
As a result, most studies now use some kind of validation cohort to evaluate the model. Usually some portion of the data is retained for testing models. Internal validation tools -such as splitsampling, cross validation, and bootstrapping -are an essential part of the model building process. Internal validation directly addresses sampling variability: minimizing the risk of over-fitting models; minimizing the impact of outliers or other influential data; and guarding against chance partitions of the data impacting upon the model. Unfortunately, they tell us little or nothing about variation in the study population or clinical setting. They tend to be overoptimistic giving good "apparent" performance while giving little or no assurance on how the prediction model is likely to perform in real life 8 . Internal validation is a necessary, but not sufficient, condition to ensure the robust transfer of diagnostic and prognostic tools. However, in medicine it is critically important to know if the model generalizes to other patients, assessed and treated by other clinicians, at other sites, using assays performed in other laboratories.

Main Text
This problem is essentially one of technology transfer or 'transportability' 9 . And internal validation methods tell us little or nothing about how the technology -the prediction model -performs when transferred to another setting. For this reason, the performance of a model must be tested using data other than that used for the model development 10,11 12 . An independent, external validation cohort provides much stronger evidence that a prognostic or diagnostic model will perform in real life 13 . Without such a cohort, the "ecological validity" of the model may be severely limited. Bleeker et al 8 for example give an early paediatric infection example demonstrating how disappointing such prediction models can be when transferred to populations other than those in which they were initially developed. More recently, Sahami et al report on the limitations of prognostic models for proctocolectomy 14 , and Damen et al 15 report on the limitations of cardiovascular risk models.

Building Reality Checks into the Translational Pathway for Diagnostic and Prognostic Models
While external validation studies may be critical to understanding the performance of such models in real life, unfortunately such studies are relatively rare [16][17][18] . We searched the PubMed database for diagnostic and prognostic models published in medicine through to September 2016 (see Supplementary Table 1). The majority of studies employed some, often limited, form of internal validation. Of 2,147 published papers on prognostic or diagnostic modelling, 323 claimed to have tested, verified or validated the model using an independent OR external validation cohort -see Table 1. And of these studies, just 120 claimed to have tested, verified or validated the model using a separate independent AND external validation cohort. However, upon further review, not all of these 120 studies can be considered truly independent validation studies. Some confuse internal and external validation; others employ relatively 'soft' validation methods. For example, timeseparated cohorts from the same centre, using the same protocols, examined and treated by the same clinicians, using test results from the same laboratory, may not capture all the sources of variability in a truly independent cohort. Even when models are validated using groups of patients from other centres, the laboratory analysis, diagnosis or scoring of results may be performed by the same team from a single centre -see Figure 1. As part of the validation process it is essential to be clear what sources of variability -patient, clinician, clinical setting -are captured by the validation cohort. We need to know whether the validation cohort gives us any reassurance that a decision tool will transfer to other centres, other patients, other clinicians, in other clinical settings. Without such assurance it is most unlikely that model performance will be reproducible. To this end, we offer the Newcastle 'Validation Cohort' Checklist to assist reviewers in reality checking the validation and identifying those sources of variability captured by the validation cohort ( Table 2). The goal is to provide greater assurance that a model will perform in a real world healthcare setting. Building Reality Checks into the Translational Pathway for Diagnostic and Prognostic Models While diagnostic and prognostic models may be of some value even without such external validation, it is not unreasonable for funding agencies to expect clarity around the generalizability of those models, proposed validation plans and the prospective translational pathway. We urge researchers to be explicit about the sources of variability captured by the external validation cohort. The validation cohort needs to be sufficiently well documented that we can tell whether the model is likely to transfer to other centres, with other patients, treated by other clinicians, using data scored or analysed by other laboratories.
Of course, model validation and impact studies are but one step on the translational pathway. For our models to translate to real patient benefit, we need to build qualification and monitoring processes into that pathway -see Figure 2. Ultimately, to apply a model we need to know whether that model is valid in a given healthcare setting, healthcare trust or even an individual practice. Even an externally validated prognostic or diagnostic model is no more than a plausible candidate for routine use in the clinic. Without some kind of qualification study, it does not establish that the model is valid in an individual healthcare setting. Even after the validation and qualification of a prospective model, we need 'production' systems in place to monitor the performance of that model in a naturalistic healthcare setting. This permits models to be reviewed and recalibrated to reflect changes in the patient population, clinical protocols and training. The analogy we draw is that of the clinical trial process. The movement from development, through recalibration and validation, to qualification and monitoring parallels that of drug development: from early proof-ofconcept studies, through dose finding to Phase 3 clinical trials, followed by post-marketing studies and surveillance. The emergence of national registries and access to 'big data' health records open up exciting possibilities for the external validation, recalibration and 'real-time, real-world' updating of diagnostic and prognostic tools 19 . Currently, it is questionable whether we have the national systems and data access permitting routine monitoring of such models. But that must be our goal.

Conclusions
While the internal validation cohort is an important part of the model building process, the external validation cohort gives greater assurance that a diagnostic or prognostic model will translate to the clinic. However, researchers need to be explicit about the sources of variability captured by the external validation cohort. The validation cohort needs to be sufficiently well documented that we can tell whether the model is likely to transfer to other centres, with other patients, treated by other clinicians, using data scored or analysed by other laboratories. Building Reality Checks into the Translational Pathway for Diagnostic and Prognostic Models Building Reality Checks into the Translational Pathway for Diagnostic and Prognostic Models