{'fraction_non_alphanumeric': 0.042415930102610994, 'fraction_numerical': 0.024179620034542316, 'mean_word_length': 4.744674642544499, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Preterm Birth Prediction: Deriving Stable and Interpretable Rules from High Dimensional Data

Preterm births occur at an alarming rate of 10-15%. Preemies have a higher risk of infant mortality, developmental retardation and long-term disabilities. Predicting preterm birth is difficult, even for the most experienced clinicians. The most well-designed clinical study thus far reaches a modest sensitivity of 18.2-24.2% at specificity of 28.6-33.3%. We take a different approach by exploiting databases of normal hospital operations. We aims are twofold: (i) to derive an easy-to-use, interpretable prediction rule with quantified uncertainties, and (ii) to construct accurate classifiers for preterm birth prediction. Our approach is to automatically generate and select from hundreds (if not thousands) of possible predictors using stability-aware techniques. Derived from a large database of 15,814 women, our simplified prediction rule with only 10 items has sensitivity of 62.3% at specificity of 81.5%.


Introduction
Every baby is expected at full term. However, still 10-15% of all infants will be born before 37 weeks as preemies (Barros et al., 2015). Preterm birth is a major cause of infant mortality, developmental retardation and long-term disabilities (Vovsha et al., 2014). The earlier the arrival, the longer the baby stays in intensive care, causing more cost and stress for the mother and the family. Predicting preterm births is highly critical as it would guide care and early interventions.
Most existing research on preterm birth prediction focuses on identifying individual risk factors in the hypothesis-testing paradigm under highly controlled settings (Mercer et al., 1996). The strongest predictor has been prior preterm births. But this does not apply for first-time mothers or those without prior preterm births. There are few predictive systems out there, but the predictive power is very limited. One of the best known studies, for example, achieved only sensitivity of 24.2% at specificity of 28.6% for first-time mothers (Mercer et al., 1996). Machine learning techniques have been used with promising results (Goodwin et al., 2001;Vovsha et al., 2014). For example, an Area Under the ROC curve (AUC) of 0.72 was obtained in (Goodwin et al., 2001) using a large observational dataset. This paper asks the following questions: can we learn to predict preterm births from a large observational database without going through the hypothesis testing phase? What is the best way to generate and combine hypotheses from data? To this end, this work differs from previous clinical research by first generating hundreds (or even thousands) of potential signals and then developing machine learning methods that can handle many irrelevant features. Our goal is to develop a method that: (i) derives a compact set of risk factors with quantified uncertainties; (ii) estimates the preterm risks; and (iii) explains the prediction made.
In other words, we derive a prediction rule to be used in practice. This demands interpretability (Freitas, 2014;Rüping, 2006) and stability (Yu et al., 2013). While interpretability is self-explanatory (Rüping, 2006), stability refers to the model that is stable under data resampling Yu et al., 2013) (i.e., model parameters do not change significantly when re-estimated from a new data sample). Thus stability is necessary for reproducibility and thus must also be enforced.
Our approach is based on 1 -penalized logistic regression (Meier et al., 2008), which is stabilized by a graph of feature correlations , resulting in a model called Stabilized Sparse Logistic Regression (SSLR). Bootstrap is then utilized to estimate the mode of model posterior as well as to compute feature importance. The prediction rule is then derived by keeping only top k most important features whose weights are scaled and rounded to integers. For estimating the upper-bound of prediction accuracy, we derive a sophisticated ensemble classifier called Randomized Gradient Boosting (RGB) by combining powerful properties of Random Forests (Breiman, 2001) andStochastic Gradient Boosting (Friedman, 2002).
The models are trained and validated on a large observational database consisting of 15,814 women and 18,836 pregnancy episodes. The SSLR achieves AUCs of 0.85 and 0.79 for 34-week and 37-week preterm birth predictions, respectively, only slightly lower than those by RGB (0.86 and 0.81). The results are better than a previous study with matched size and complexity (Goodwin et al., 2001) (AUC 0.72). A simplified 10-item prediction rule suffers only a small loss in accuracy (AUCs 0.84 and 0.77 for 34 and 37 week prediction, respectively) but has much better transparency and interpretability.

Cohort Selection
We acquired a large observational dataset from Royal North Shore (RNS) hospital located in NSW, Australia. The RNS has 18,836 pregnancy episodes collected during the preriod of 2007-2015. The data statistics are summarized in Table 1.

Preterm Birth Definition
A birth is considered preterm if it occurs before 37 full weeks of gestation. There are two types of preterm births -spontaneous and elective (or indicated). Spontaneous births occur naturally without clinical interventions, and this accounts for about 60-70% of all preterm births. In our database, the rules for figuring out spontaneous and elective, as specified by our senior clinician, are: • Spontaneous: PPROM ∨ (Preterm∧ UterineActiv.Adm. ∧ ¬PrelaborInterv.) • Elective: Preterm ∧ ¬PPROM ∧ ¬UterineActiv.Adm. ∧ (PrelaborInterv. ∨ Caesare-anSect.) where PPROM stands for Preterm Premature Rupture of Membranes. This may leave a small portion of births not falling into either category (∼ 1.5%).
Sub-cohort Selection Our initial analysis revealed a critical, undocumented issue: covariate shifting over time, probably due to the evolution of data collection protocol. Using features extracted in the next subsection, we embedded data into 2D using t-SNE (van der Maaten and Hinton, 2008) and visually examined the shifting. For this reason, we will work exclusively on the data collected between 2011-2015 as they are more recent.

Feature Generation
We operate under a hypothesis-free mode -the data is collected in hospital operations. The biggest challenge in generating features from an observational database is to detect and prevent the so-called "leakage" problem (Kaufman et al., 2012). This happens when recorded information implicitly indicates the outcome to be predicted. In pregnancy databases, it could be procedures and tests that are performed late in the course of pregnancy. When it happens, it may already indicate a full-term birth. With our clinicians and local database experts, we verify the generated feature list to prevent this from happening. We explicitly extracted features that occurred before week 25 of gestation. During a pregnancy, a woman typically visits the hospital several times before labor. At 20-25 weeks, it is critical to estimate the risks, one of which is the risk of preterm birth. A care model may be decided by clinician depending on the perceived risks. The care allocation contains important information not present in other risk factors, and thus we will treat them separately. A part from real-valued measurements (such as BMI and weights), many features are discrete, for example, a medication name under the data field "medication". Thus discrete features are counts of such discrete symbols. The database also contains a certain amount of free text, documenting the findings by clinicians. For each text field, we extracted unigrams after removing stop words. For robustness, rare features that occurs in less than 1% of data points were removed. Finally, we retained 762 features without textual information, and 2,770 features with textual information.

Stabilized Sparse Logistic Regression
For a feature vector x ∈ R p , we focus on building a linear prediction rule: .., w p are feature weights. For binary classification, we use logistic regression to define the probability of outcome: P (y = 1 | x) = 1/ 1 + e −f (x) . To work with a large number of features, we derive a sparse solution by minimizing the following 1 -penalized loss (Meier et al., 2008): where d denotes data points, and λ > 0 is the penalty factor responsible for driving weights of weak features toward zeros.
However, sparsity invites instability, that is, the selected features vary greatly if we slightly perturb the dataset (Gopakumar et al., 2014;Zou and Hastie, 2005). One reason is that when two features are highly correlated, the 1 -penalty will pick one randomly. In healthcare practice, multiple processes and data views are often recorded at the same time, causing a high level of redundancy in observational data. This results in multiple models (and feature subsets) of equal predictive performance. This behavior is undesirable because the derived models are unstable to earn trust from clinicians.
An effective way to reduce instability under correlated data is introduced in . More precisely, we assert that correlated features will have similar weights. This is realized through the following objective function: , and S ij > 0 is the similarity between features i and j, subject to j =i S ij = 1. This feature-similarity regularizer is equivalent to a multivariate Gaussian prior of mean 0 and precision matrix (1 − α) (S − I) (S − I) where I is the identity matrix. Hence minimizing the loss in Eq. (2) is to find the maximum a posterior (MAP), where the prior is a product of a Laplace and a Gaussian distributions. In this paper, the similarity matrix S is computed using the cosine between data columns (each corresponding to a feature). We will refer to this model as Stabilized Sparse Logistic Regression (SSLR).

Deriving Prediction Rule and Risk Curve
The prediction rule and risk curve are generated using the following algorithm. Given number of retained features k and number of bootstraps B, the steps are as follows: 1. Bootstrap model averaging: B SSLR models (Sec. 3.1) are estimated on B data bootstraps. The feature weights are then averaged. Together with the MAP estimator in Eq.
(2), this model averaging is closely related to finding a mode of the parameter posterior in an approximate Bayesian setting. This procedure is expected to further improve model stability by simulating data variations (Wang et al., 2011).
2. Feature selection: Features are ranked based on importance, which is averaged weight × feature standard-deviation (Friedman and Popescu, 2008). This measure of feature importance is insensitive to feature scale, and encodes the feature strength, stability and entropy. Top k features are kept.
3. Prediction rule construction: Weights of selected features are linearly transformed and rounded to sensible integers. For example, the weights range from 1 to 10 for positive weights, and from -10 to -1 for negative weights. The prediction rule has the following form: f * (x) = k j=1 η j x j where η j are non-zero integers. We shall refer to features with positive weights as risk factors, and those with negative weigths as protective factors.

Risk curve:
The prediction rule is then used to score all patients. The scores are converted into risk probability using univariate logistic regression. This produces a risk curve.

Randomized Gradient Boosting
State-of-the-art classifiers are often ensembles such as Random Forests (RF) (Breiman, 2001) and Stochastic Gradient Boosting (SGB) (Friedman, 2002). To test how simplified sparse linear methods may fare against complex ensembles, we develop a hybrid RF/SGB called Randomized Gradient Boosting (RGB) that estimates the outcome probability as follows: where β t ∈ (0, 1) is a small learning rate, x t ∈ x is a feature subset, and each h t (x t ) is a regression tree, which is added in a sequential manner as in (Friedman, 2002). Following (Breiman, 2001), and each non-terminal node is split based on a small random subset of features.

Evaluation Approach/Study Design
The data is randomly spitted into two parts: 2/3 for training and 1/3 for testing. To be consistent with the practice of clinical research, for the test data we maintain balanced classes through under-sampling of the majority class. The parameters for Stabilized Sparse Logistic Regression (SSLR) in Eq. (2) are set as α = 0.5, and λ = 5. The Randomized Gradient Boosting (RGB, Sec. 3.3) had 500 decision trees learnt from a learning rate of 0.03, and each tree had 256 leaves at most. Each tree is trained on a random subset of m = p/3 features, and each node split is based on a random sub-subset of m/3 features.
For performance measures, we report sensitivity (recall), specificity, NPV, PPV (precision), F-measure (2×recall×precision/(recall + precision)) and AUC. Except for AUC, the other measures depend on the decision threshold at which the prediction is made, that is we predictŷ = 1 if P (y = 1 | x) ≥ τ for threshold τ ∈ (0, 1). We chose the threshold so that sensitivity matches specificity in the training data.

Visual Examination
To visually examine the difficulty of the prediction problem, we embed data points (episodes) into 2D using t-SNE (van der Maaten and Hinton, 2008). Fig. 2a plots data points coded in colors corresponding to preterm or full-term. There is a small cluster mostly consisting of preterm births, and a big cluster in which preterm births are randomly mixed with term births. This suggests that there are no simple linear hyper-planes that can separate the preterm births from the rest.

Prediction Results
We investigate multiple settings: observed features only (Obs), observation with booking & care allocation (Obs+care), and observation with textual information (Obs+care+text).       Table 3 reports the AUC for different settings (for spontaneous births only and all cases). For spontaneous births, the highest AUC of 0.773 is achieved by RGB using all available information. For all births, the highest AUC is 0.807, also by RGB with all information. Overall, RGB fares slightly better than SSLR in AUC measure. Care information, such as booking and allocation of care, has a good predictive power. Table 4 reports the AUC for 34-week prediction. For spontaneous births prediction, the largest AUC of 0.849 is achieved by RGB using care information. For both elective and spontaneous births, the largest AUC is 0.862.

Prediction Rules
Prediction rules are generated using the procedure described in Sec. 3.2. Table 5 reports the predictive performance of generated rules with 10 items. Generally the performance drops by several percent points. Using protective factors (those with negative weights) is better, suggesting that they should be used rather than discarded. Table 6 lists the items and their associated weights (with standard deviations) for the case without care information. The top three risk factors are multiple fetuses, cervix incompetence and prior preterm births. Other risk factors include domestic violence, history of hypertension, illegal use of marijuana,     Table 7 reports for the case with care information, which plays an important roles as risk factors. Fig. 2b shows the risk curve estimated from the first prediction rule in Table 7 (without protective factors). When the score is 0, there is still a 5.3% chance of preterm. That says that the risk factors here only account for 50% of preterm births. When the score is 10 (e.g., with twins), the risk doubles.

Discussion and Related Work
We have presented methods for predicting preterm births from high-dimensional observational databases. The methods include: (i) discovering and quantifying risk factors, and (ii) deriving simple, interpretable prediction rules. The main methodological novelties are (a) the use of stabilized sparse logistic regressions (SSLR) for deriving stable linear prediction models, and (b) the use of bootstrap model averaging for distill simple prediction rules in an approximate Bayesian fashion. To estimated the upper-bound of model accuracy for given data, we also introduced Randomized Gradient Boosting, which is a hybrid of Random Forests (Breiman, 2001) and Stochastic Gradient Boosting (Friedman, 2002).

Findings
For 37-week preterm births, the highest AUC using RGB is in the range 0.78 using only observational information, and in the range 0.80-0.81 with care information (booking + allocation decision). The SSLR is slightly worse with the AUC in the range of 0.76 with only observational information, and AUC of 0.79 with care decision.Thus, care information has a good predictive power. This is expected since it encodes doctor's knowledge in risk assessment. It is also likely to be available later in the course of pregnancy. The results are better than a previous study with matched size and complexity (Goodwin et al., 2001) (AUC = 0.72). Simplified prediction rules with only 10 items suffer some small loss in accuracy. The AUCs are 0.74 and 0.77 with and without care information, respectively. The payback is much better in transparency and interpretability.

Related Work
Preterm birth prediction has been studied for several decades (de Carvalho et al., 2005;Goldenberg et al., 1998;Iams et al., 2001;Macones et al., 1999;Vovsha et al., 2014). Most existing research either focuses on deriving individual predictive factors, or builds prediction model under highly controlled data collection. Three most common known risk factors are: prior preterm births, cervical incompetence and multiple fetuses. These agree with our findings (e.g., Table 6). Data mining approaches that leverage observational databases have been attempted in (Goodwin et al., 2001) and (Vovsha et al., 2014) showing a great promise.
Clinical prediction rules have been widely used in practice (Gage et al., 2001). A popular approach is logistic regression with scaled and rounded coefficients. Model stability has been studied in biomedical prediction (Austin and Tu, 2004;He and Yu, 2010;Gopakumar et al., 2014;. The machine learning community has worked and commented on interpretable prediction rules in multiple places (Bien et al., 2011;Carrizosa et al., 2016;Emad et al., 2015;Freitas, 2014;Huysmans et al., 2011;Rüping, 2006;Vellido et al., 2012;Wang et al., 2015). There are been applications to biomedical domains (Haury et al., 2011;Song et al., 2013;Ustun and Rudin, 2015). In (Ustun and Rudin, 2015), the authors seek to derive a sparse linear integer model (SLIM) where the coefficients are linear. The simplification of complex models is also known as model distillation (Hinton et al., 2015) or model compression (Bucilua et al., 2006). Most current work in model distillation focuses on deep neural networks, which are hard to interpret.

Limitations
The models derived in this paper are subject to the quality of data collected. For example, the covariate shift problem can occur within a hospital over time, as pointed out in Sec. 2.1. This study is also limited to data collected just for the pregnancy visits. There may be more predictive information in the electronic medical records. However, initial inquiry revealed that since pregnant women are relatively young, the medical records are rather sparse.

Conclusion
The methods presented in this paper to derive stable and interpretable prediction rules have shown promises in predicting preterm births. The accuracy achieved is better than those reported in the literature. As the classifiers are derived directly from the hospital database, they can be implemented to augment the operational workflow. The prediction rules can be used in paper-form as a check-list and a fast look-up risk table.