{'fraction_non_alphanumeric': 0.024711490215755142, 'fraction_numerical': 0.012826141495233317, 'mean_word_length': 4.977319587628866, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
How little data do we need for patient-level prediction?

Objective: Provide guidance on sample size considerations for developing predictive models by empirically establishing the adequate sample size, which balances the competing objectives of improving model performance and reducing model complexity as well as computational requirements. Materials and Methods: We empirically assess the effect of sample size on prediction performance and model complexity by generating learning curves for 81 prediction problems in three large observational health databases, requiring training of 17,248 prediction models. The adequate sample size was defined as the sample size for which the performance of a model equalled the maximum model performance minus a small threshold value. Results: The adequate sample size achieves a median reduction of the number of observations between 9.5% and 78.5% for threshold values between 0.001 and 0.02. The median reduction of the number of predictors in the models at the adequate sample size varied between 8.6% and 68.3%, respectively. Discussion: Based on our results a conservative, yet significant, reduction in sample size and model complexity can be estimated for future prediction work. Though, if a researcher is willing to generate a learning curve a much larger reduction of the model complexity may be possible as suggested by a large outcome-dependent variability. Conclusion: Our results suggest that in most cases only a fraction of the available data was sufficient to produce a model close to the performance of one developed on the full data set, but with a substantially reduced model complexity.


Background and significance
Physicians infer diagnoses, prognoses, and treatment pathways based on the available medical history of their patients and the current clinical guidelines. Clinical prediction models can support this process by providing risk information on disease presence and progression. [1,2] Well-known prognostic prediction models are QRISK and the Framingham Risk Score, which predict the 10-year risk of developing cardiovascular disease, or the CHA2DS2-VASc Score, which predicts one-year ischemic stroke risk in patients with atrial fibrillation. [3][4][5] The common approach to optimally develop such prediction model has been to include as much data as possible, because data set could often be small. Small datasets may describe model performance far beyond what would have been possible if additional data were accumulated.
In recent years, networks of large observational health databases have started to reverse this practice and opened up possibilities to develop clinical prediction models on massive amounts of patient data. For example, IBM MarketScan Research Databases provide de-identified longitudinal patient-level data of more than 245 million individuals since 1995. [6] This amount of data creates opportunities to develop and validate models for many populations at risk and all their relevant health outcomes.
However, models developed on these large amounts of observational health data run the risk of having more complexity (e.g. include more features) without achieving substantially better discrimination. As a result, these models may become harder to interpret, more difficult to implement in clinical practice, and more susceptible to overfitting. [7] In addition, developing prediction models on such large data sources can put strong demands on computing resources and may require computation times that can become prohibitive. Reducing the sample size of a large and unwieldy dataset to an "adequate" sample size that is still sufficient to achieve nearly the same performance as the full dataset, may facilitate the development of less complex clinical prediction models.
An often used heuristic to establish an adequate sample size is to require a minimum number of events per variable (EPV). Various studies proposed different rules for the minimum EPV, ranging from 5 to 50. [8][9][10][11] However, these EPV rules originated from studies that were limited by sample size themselves and based on only a handful of prediction problems, which may restrict their generalizability. Another concern is that these rules may be too simplistic and that the adequate sample size depends on many aspects, such as the data structure or the model complexity. [12][13][14] Van Smeeden et al. concluded that the current evidence supporting EPV rules for binary logistic regression is weak and that there is an urgent need for guidance on sample size considerations. [15] Moreover, to estimate an adequate sample size based on EPV rules requires a priori knowledge of the number of variables in the model. This information is not available for data-driven approaches that automatically select predictors from an extensive set.
Alternatively, the adequate sample size can be established empirically through a learning curve, which shows model performance as a function of sample size. Learning curves can be generated by training models on training sets of increasing size and assessing their performances on a test set. The sample size at which the plateau phase of the learning curve begins, which indicates convergence towards a maximum performance, can then be considered the adequate sample size.
In this study, we have performed one of the largest real-world data study of predictive modeling to date, fitting 17,248 models through 81 prediction problems in three observerational health databases. We report the findings of this study with an aim to provide guidance on sample size considerations for future prediction work.

Objective
Provide guidance on sample size considerations for developing predictive models by empirically establishing the adequate sample size, which balances the competing objectives of improving model performance and reducing model complexity as well as computational requirements.

Data sources
We used three longitudinal IBM MarketScan Research Databases: [6] • Commercial Claims and Encounters Database (CCAE). This database contains data from more than 90 million individuals in the United States, for example, employees, their spouses, and dependents enrolled in employer-sponsored insurance health plans. The data include inpatient and outpatient medical claims, diagnosis codes, and outpatient pharmacy dispensing claims. The patients in CCAE are between 0 and 65 years of age.

Prediction
We define patient-level prediction as a modeling process wherein a health outcome is predicted within a time-at-risk period relative to the index date of a target cohort. Prediction is performed using a set of predictors derived from individual patient data in an observation window before the index date. The target cohorts are defined by an index rule specifying the index date, accompanied by a set of inclusion criteria. Criteria can reference one or more clinical concepts that arise from standardized vocabularies in the OMOP CDM. [17] For our analysis, we used 81 previously-defined prediction problems: 23 health outcomes predicted in a depression cohort, [17] and 58 health outcomes predicted in a hypertension cohort. [18] The detailed target cohort definitions can be found in Appendix I. Among the various health outcomes that are predicted in a time-at-risk of 365 days after the index date are, for example, gastrointestinal bleeding, stroke, dementia, and chronic kidney disease.
Appendix II provides a complete list of all prediction problems, target cohort sizes, number of outcomes, and outcome rates in the three IBM MarketScan Research Databases.

Candidate predictors
We considered a large group of candidate predictors, including demographics (gender, five-year age groups, race, ethnicity, and index month), condition diagnoses, drug exposures, procedures performed, measurement occurrences, observations, and device exposures.
For all candidate predictors, we determined their occurrence in a long-term window of 365 days before the index date. All non-demographic predictors were additionally considered in a short-term window of 30 days before the index date. This approach allows for distinguishing long-term and short-term trends that could be predictive of a health outcome. Table 1 shows the number of candidate predictors and the median target cohort size, number of outcome events, and outcome rate, for each target cohort and database. Target cohort size could differ slightly for different health outcomes as detailed in the supplemental material in Appendix I.
Numerical predictors were binned, for example, age into age groups, and one-hot encoded into binary variables for each category value. We omitted candidate predictors that were either true or false throughout all patients in a target cohort.

Learning curve
To generate a learning curve, we trained prediction models on successively larger subsets of a training set and measured their performance on a test set.
The test set of each prediction problem consisted of a random selection of 20% of the full data set. We used the same test set to assess the performance of all models of a learning curve. The remaining 80% of the data was used as the training set. From this set, we defined a sequence of proper subsets, where each larger subset contained all data from the smaller subsets.
We report our subset sizes in terms of the number of events they contain. This is analogous to EPV rules, which regard the number of events (or, more generally, samples of the minority class) as the determinant for model performance. Indeed, our experiments show events being more strongly correlated to the learning rate of a learning curve than the sample size, which becomes instrumental when comparing across different datasets as this study aims to do. A detailed explanation of the experiments can be found in the supplemental material in Appendix III.
Therefore, our sampling strategy takes each subset by stratified sampling to maintain the same outcome rates as in the complete training set and sample size recommendations will be in terms of the number of events. The size of the subsets increases in steps of 100, from 100 events to 20,000 events (if available).
Previous learning curve studies used accuracy as performance metric. [7,21,22] However, since accuracy does not allow comparison between datasets with different outcome rates and is not a good measure of performance on imbalanced classes we evaluate the model's discrimination performance with the area under the receiver operating characteristic curve (AUROC) as is common practice in patient-level prediction studies.
To draw the learning curves, we plotted model performance (AUROC) on the test set against the number of events used for training.

Model complexity curve
The property of LASSO regularization to optimize models to contain fewer predictors is used to assess the change in model complexity across different sample sizes, prediction problems, and databases. In our learning curve analysis we also record the number of predictors selected by LASSO for each model.
To draw the model complexity curve, we plotted number of predictors on the test set against the number of events used for training.

Power-law fit
Generally, learning curves will exhibit some degree of noise due to random effects in the data and model fitting process. To diminish this noise and reliably apply convergence criteria, comparative studies have shown that learning curves are best fit with an inverse power law function. [21,[23][24][25] The performance P as a function of the number of events x is then given in which the parameters %, ', and ( represent the minimum achievable error, learning rate, and decay rate, respectively. The fitted curve provides a smoothed, strictly monotone version of the original learning curve.
For learning curve fitting, we used the Levenberg-Marquardt algorithm, which requires at least as many observations as the number of variables to fit. For three variables and a step size of 100 events, this implied that a prediction problem needed at least 300 events in the training set to be included in our experiments. The model complexity curves were not fitted during this analysis.

Determination of the adequate number of events and sample size
To detect learning curve convergence, slope-based and threshold-based criteria have been proposed. Slope-based criteria determine the slope of the learning curve for different sample sizes and take the smallest sample size for which the slope of the learning curve is "close to zero."[26-30] However, none of the studies that proposed slope-based criteria quantify when the slope should be considered sufficiently close to zero.
Threshold-based criteria are based on the performance difference between a model trained on all available samples and a model trained on a reduced sample size. The minimum sample size for which this difference becomes smaller than a user-specified threshold is considered adequate. [22] Previous studies that used accuracy as the performance metric considered threshold values varying from 0.001 to 0.02. [7,21,22] These studies were performed for a single prediction problem using relatively small samples.
Our approach to determine the adequate number of events relies on the threshold-based method and is exemplified in Figure 1Error! Reference source not found. for a single prediction problem. In this example, 40 prediction models were generated for 40 training sets, with number of events ranging from 100 to 4,000. The performances of the models were fitted with a power law function. The performance according to the fitted curve at the maximum number of events was taken as the maximum achievable performance. The number of events where the performance decreased by a prespecified threshold value was then considered the adequate number of events. The corresponding adequate sample size was computed by dividing the adequate number of events by the outcome rate of the prediction problem. We repeated this exercise for 81 target-outcome pairs, within 3 different datasbases. We

Number of samples and predictors shown for the adequate model pertain to the model at the closest data point.
From the 81 prediction problems, we excluded those that had fewer than 300 events in the training set, resulting in 80, 75, and 72 prediction problems for CCAE, MDCD, and MDCR, respectively. Using these datasets, a total of 17,248 prediction models were trained and evaluated to generate the learning curves for the 227 included prediction problems. Figure 2 visualizes the true and fitted learning curves. While many of the learning curves appear to be  To evaluate the goodness of fit, we computed the residuals as the AUROC performance difference between the true learning curve and the fitted learning curve at 1,000, 2,000, 5,000, 10,000, and 20,000 events ( Table 2). The median and interquartile range of the residuals show that the fit is excellent. It is noteworthy that increasingly fewer data sets could provide these larger number of events apparent from the number of included learning curves. Appendix V provides some more detailed goodness of fit results for included learning curves.   The relative reduction of the number of events at the four thresholds is shown in Figure 4. Of note, since this reduction is a proportion, the relative reductions in number of events and in full sample size are the same. There is an upward trend for larger number of events, which indicates that the reduction is more substantial for larger data sets. For the thresholds of 0.001, 0.005, 0.01, and 0.02 we achieve a median reduction (interquartile range) of 9.5% (6.7% -13.6%), 37.3% (28.4% -48.0%), 58.5% (46.1% -68.8), and 78.5% (67.8% -85.1%) respectively.
There is a large variability of the relative reduction across the different prediction problems and databases indicated by the vertical spread of the data points.

Model complexity
We assessed how model complexity changes at the threshold-based adequate number of events as compared to the full models based on all available of events. Because at the adequate number of events no datapoint was generally recorded (and we did not fit the model complexity curve as we did for the learning curves), we decided to report the complexity of the model trained on the next smallest subset as the adequate model. Prediction problems were omitted from this plot, if the adequate model had less than 100 events.

Discussion
Our extensive analysis of learning curves on large observational health datasets showed that a considerable reduction in sample size and model complexity paired with a minimal loss of performance is possible.
Using a threshold-based convergence criterion we defined the adequate sample size, which achieves a median reduction of the number of events between 9.5% and 78.5% for thresholds between 0.001 and 0.02. Because, the outcome rate in the learning curve's subsets was held constant, this reduction in the number of events can be translated to a reduction in the absolute sample size of the same proportion. Remarkably, also for small datasets often large reductions are feasible. However, in general larger datasets saw larger reductions as indicated by the trend lines in Figure 4.
An additional benefit of using the adequate sample size is a less complex model. We could achieve a median reduction of the number of predictors between 8.6% and 68.3% for the thresholds between 0.001 and 0.02. Oates et al. have previously reported that larger datasets lead to overly complex decision tree models. [7] Our experiments show that this phenomenon also holds true for logistic regression models fitted using LASSO. Therefore, using the adequate sample size we can develop less complex models, which has implications for applying these models in practice. For example, a large number of predictors can be a barrier to clinical implementation when a database cannot provide such features. Fewer predictors may also become more interpretable for clinicians and easier to apply manually. Future research should be dedicated to assessing the adequate model's transportability compared to a model developed on the complete data through external validation.

Use of learning curves
The threshold-based approach to finding an adequate number of events relies strongly on the monotone exponential function fitted to the learning curves. Both slope-based and thresholdbased criteria are sensitive to variability in the learning curves. Several approaches have been proposed to smooth the curve before a criterion is applied, including averaging of data points and curve fitting. [21] Comparative studies have shown that learning curves are best fit with a power-law. [23][24][25] In our experiments, this power-law fit was found to be excellent, with median residuals smaller than 0.001. However, the interquartile range of the fit is in the order of our smallest threshold value of 0.001, indicating that our results for this threshold may need to be interpreted with caution.
Future work is required to deepen our understanding of learning curves, for example, by comparing how the adequate sample size behaves for different metrics (e.g. AURPC) and different prediction algorithms (e.g. random forest).
Note that this study exclusively focuses on reducing the sample size. Another research question is 'how much performance can be gained?', if more data were available. This is an extrapolation question for which learning curves may also be useful. Although the power-law fit was excellent for interpolating the adequate sample sizes, further research is necessary to assess whether the power-law function can be used to establish the achievable performance by extrapolation.

Recommendations
The possibility of reducing sample size and model complexity can have implications for prediction studies. Depending on their goals, researchers may decide for one of two scenarios: (1) If a researcher's primary goal is to reduce computational resources with the additional benefit of reduced model complexity, we recommend sample size reduction based on the minimum expected relative reduction in the number of events as shown in Figure 4.
For example, if a researcher accepts a threshold of 0.01, the median reduction for a given data set with 20,000 events would be 73%, and thus sample size could be reduced with 73%, corresponding to a median complexity reduction of 59% ( Figure 6). The advantage of this approach is that no learning curve needs to be generated, which saves on the overhead associated with fitting the various models.
(2) If a researcher's goal is to reduce model complexity for a given data set as much as possible, we recommend that a learning curve is generated to determine the adequate sample size for this data set. Clearly, generating a learning curve will increase the use of computational resources but the complexity reduction is potentially much larger than in the previous scenario. For example, for a threshold of 0.01 and a data set with 20,000 events, Figure 4 shows that a sample size reductions of up to 84% may be possible.

Limitations
Although various prediction problems were included in this study, they originated from only two target cohorts (depression and hypertension), in which the different health outcomes were predicted, and three databases, which exclusively held U.S. administrative claims data.
Moreover, only a single prediction algorithm was assessed, which was logistic regression. As a result one should be aware that there may be limitations for scenario (1) in terms of generalizability to other target populations, data sources and prediction algorithms. More problem specific sample size recommendations can be received through generating a learning curve as proposed in scenario (2).

Final thoughts
We believe sample size considerations are fundamental to move forward the notion of a learning healthcare system. Observational health databases not only offer massive datasets for individual prediction problems, but also create the opportunity to develop prediction models for many populations at risk and all their relevant health outcomes. Infrastructure for such prediction analyses at scale is readily available. For example, the Observational Health Data Sciences and Informatics (OHDSI) initiative has designed and implemented a prediction framework for observational health data standardized to a common data model. [17] This framework can be used to develop and validate clinical prediction models across disparate observational health databases using a standardized pipeline, which allows to serve large patient population in a timely manner. We believe that the requirement to develop and update hundreds, or even thousands, of prediction models at a time on massive datasets can greatly benefit from the adequate sample size.

Conclusion
In this study we empirically derived the adequate sample size for patient-level predictive modeling using learning curves. Our results suggest that in most cases only a fraction of the available data was sufficient to produce a model close to the performance of one developed on the full data set, but with a substantially reduced model complexity.

COMPETING INTERESTS STATEMENT
The authors have no competing interests to declare.

Appendix I: Cohort definitions
For this study we defined prediction problems across the two target cohorts of hypertension and dementia.

Initial Event Cohort
People having any of the following: • a drug exposure of First-line hypertension drug 1 o for the first time in the person's history and limit initial events to earliest event per person.

For people matching the Primary Events, include:
Having all of the following criteria: • exactly 0 occurrences of a drug exposure of Hypertension drug 2 where event starts between all days Before and 1 days Before index start date • and at least 1 occurrences of a condition occurrence of Hypertensive disorde 3 where event starts between 365 days Before and 0 days After index start date Limit cohort of initial events to earliest event per person.

Custom Drug Era Exit Criteria
This strategy creates a drug era from the codes found in the specified concept set. If the index event is found within an era, the cohort end date will use the era's end date. Otherwise, it will use the observation period end date that contains the index event.
Use the era end date of First-line hypertension drugs 2 • allowing 30 days between exposures • adding 0 days after exposure end

Additional population settings
The target cohort was further specified using two attrition steps: (1) patients were included if they had continuous observation for at least 365 days prior to index date. Within this washout period predictors were recorded. And (2), the time-at-risk period during which a health outcome was predicted was defined as 365 days after index date. Patients who had not experienced the health outcome were only included if they were observed for the full 365 days of time-at-risk. By contrast, patients who had experienced the health outcome were always included regardless of whether they were observed during their full time-at-risk period. In this way, we ensured that all patients who experienced a health outcome that resulted in the end of their observation period during their time-at-risk, for example due to death or change of care provider, were still included in the study. This approach also meant that target cohort size could differ depending on the health outcome. We defined the time-at-risk period to start on the first day after the index date. This was done to exclude patients who experienced the health outcome on the index date, because the succession of medical observations on a single day could not be reconstructed from the available observational health data.

Concept sets
The concept sets below reference concept ids defined in the Observational Medical Outcomes Partnership Common Data Model (OMOP CDM) version 5. The "Exclude" column is a checkbox, which will prevent that concept from being used in the concept set. The "Descendants" column is a check box, which will use the vocabulary relationships to automatically select all descendants. If this option is used in conjunction with the "Exclude" option, it will exclude the current concept and all its descendants. The "Mapped" column is a check box, which will use the vocabulary relationships to automatically select all source concepts mapped to the selected concept in the OMOP CDM.

Initial Event Cohort
People having any of the following: • a condition occurrence of Depression 4 o for the first time in the person's history and limit initial events to earliest event per person.

For people matching the Primary Events, include:
Having any of the following criteria: • at least 1 occurrences of a condition occurrence of Depression 4 where event starts between 1 days After and all days After index start date • or at least 2 occurrences of a drug exposure of Drugs to treat depression 5 having all of the following criteria (where event starts between 0 days Before and all days After index start date): o exactly 0 occurrences of a condition occurrence of Other indications for drugs used to treat depression 6 where event starts between 30 days Before and 7 days After index start date • or at least 2 occurrences of a procedure of Procedures for depression 7 where event starts between 0 days Before and all days After index start date Limit cohort of initial events to earliest event per person.

End Date Strategy
No end date strategy selected. By default, the cohort end date will be the end of the observation period that contains the index event.

Additional population settings
The target cohort was further specified using two attrition steps: (1) patients were included if they had continuous observation for at least 365 days prior to index date. Within this washout period predictors were recorded. And (2), the time-at-risk period during which a health outcome was predicted was defined as 365 days after index date. Patients who had not experienced the health outcome were only included if they were observed for the full 365 days of time-at-risk. By contrast, patients who had experienced the health outcome were always included regardless of whether they were observed during their full time-at-risk period. In this way, we ensured that all patients who experienced a health outcome that resulted in the end of their observation period during their time-at-risk, for example due to death or change of care provider, were still included in the study. This approach also meant that target cohort size could differ depending on the health outcome. We defined the time-at-risk period to start on the first day after the index date. This was done to exclude patients who experienced the health outcome on the index date, because the succession of medical observations on a single day could not be reconstructed from the available observational health data.

Concept sets
The concept sets below reference concept ids defined in the Observational Medical Outcomes Partnership Common Data Model (OMOP CDM) version 5. The "Exclude" column is a checkbox, which will prevent that concept from being used in the concept set. The "Descendants" column is a check box, which will use the vocabulary relationships to automatically select all descendants. If this option is used in conjunction with the "Exclude" option, it will exclude the current concept and all its descendants. The "Mapped" column is a check box, which will use the vocabulary relationships to automatically select all source concepts mapped to the selected concept in the OMOP CDM.

Appendix III: Relevance of events
The inherent idea of events per variable (EPV) rules is that outcome events are imperative for learning (with the general assumption being that the minority class is to be predicted). In Figure   1 we compare learning curves for logistic regression as a function of the number of observations with the same learning curves as a function of the number of events for all prediction problems. Note that some learning curves end due to using up the complete available training set data, while others extend beyond the plotted area.
Learning curves as a function of the number of observations converge inconsistently across prediction questions. Most of the models appear to converge between 100,000 -300,000 observations. The remaining models, primarily those with low outcome rates, show large instability in their learning curves even beyond 300,000 observations or have insufficient observations to converge. In contrast, the convergence of learning curves as a function of the number of events appears more consistent. This suggests that learning is correlated more strongly to the number of events than to the number of observations. To quantify the correlation observed in Figure 1, we fitted the function !(#; %, ', () = (1 − %) − ' • # !" as presented in Equation 1 of the main document to each learning curve. We then filtered those fitted curves that had unreasonable parameter value, including minimum achievable errors below 0 and decay rates above 1.5. Figure 2 compares the parameter distribution for curves fitted as a function of observations and curves fitted as a function of events. While minimum achievable error and decay rate are the same (slight variation due to parameter convergence during fitting), we observe a significantly wider spread of the learning rate for curves fitted against the sample size (note the logarithmic vertical axis). This suggests that the outcome rate acts as a scaling factor, which becomes relevant once comparing across prediction questions with varying outcome rates, as in this study. It is, thus, reasonable to give more weight to events when compiling data sets for clinical prediction across prediction problems.