{'fraction_non_alphanumeric': 0.03279403675815109, 'fraction_numerical': 0.017051381749900424, 'mean_word_length': 4.62569355527102, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Handling missing data in large healthcare dataset: a case study of unknown trauma outcomes

Handling of missed data is one of the main tasks in data preprocessing especially in large public service datasets. We have analysed data from the Trauma Audit and Research Network (TARN) database, the largest trauma database in Europe. For the analysis we used 165,559 trauma cases. Among them, there are 19,289 cases (13.19%) with unknown outcome. We have demonstrated that these outcomes are not missed `completely at random' and, hence, it is impossible just to exclude these cases from analysis despite the large amount of available data. We have developed a system of non-stationary Markov models for the handling of missed outcomes and validated these models on the data of 15,437 patients which arrived into TARN hospitals later than 24 hours but within 30 days from injury. We used these Markov models for the analysis of mortality. In particular, we corrected the observed fraction of death. Two na\"ive approaches give 7.20% (available case study) or 6.36% (if we assume that all unknown outcomes are `alive'). The corrected value is 6.78%. Following the seminal paper of Trunkey (1983) the multimodality of mortality curves has become a much discussed idea. For the whole analysed TARN dataset the coefficient of mortality monotonically decreases in time but the stratified analysis of the mortality gives a different result: for lower severities the coefficient of mortality is a non-monotonic function of the time after injury and may have maxima at the second and third weeks. The approach developed here can be applied to various healthcare datasets which experience the problem of lost patients and missed outcomes.


Introduction
Enthusiasm for the use of big data in the improvement of health service is huge but there is a concern that without proper attention to some specific challenges the mountain of big data efforts will bring forth a mouse [1]. Now, there is no technical problem with "big" in healthcare. Electronic health records include hundreds of millions of outpatient visits and tens of millions of hospitalizations, and these numbers grow exponentially. The main problem is in quality of data.
"Big data" very often means "dirty data" and the fraction of data inaccuracies increases with data volume growth.
Human inspection at the big data scale is impossible and there is a desperate need for intelligent tools for accuracy and believability control.
The second big challenge of big data in healthcare is missed information. There may be many reasons for data incompleteness. One of them is in health service "fragmentation". This problem can be solved partially by the national and international unification of the electronic health records (see, for example, Health Level Seven International (HL7) standards [13] or discussion of the template for uniform reporting of trauma data [37]). However, some fragmentation is unavoidable due to the diverse structure of the health service. In particular, the modern tendency for personalization of medicine can lead to highly individualized sets of attributes for different patients or patient groups. There are several universal technologies for the handling of missing data [40,41,36,46,23,22,14]. Nevertheless, the problem of handling missed values in large healthcare datasets is certainly not completely solved. It continues to attract the efforts of many researchers (see, for example, [8]) because the popular universal tools can lead to bias or loss of statistical power [21,51]. For each system, it is desirable to combine various existing approaches for the handling of missing data (or to invent new ones) to minimize the damage to the results of data analysis. For the best possible solution, we have to take into account the peculiarities of each database and to specify the further use of the cleaned data (it is desirable to understand in advance how we will use the preprocessed data).
In our work we analyze missed values in the TARN database [54]. We use the preprocessed data for: • the evaluation of the risk of death, • the identification of the patterns of mortality, • approaching several old problems like the Trunkey hypothesis about the trimodal distribution of trauma mortality [55].
The 'two stage lottery' non-stationary Markov model developed in the sequel can be used for the analysis of missing outcomes in a much wider context than the TARN database and could be applied to the handling of data gaps in healthcare datasets which experience the problem of transferred and lost patients and missing outcomes.
In this paper we analyze the unknown outcomes. The next task will be the analysis of missed data in the most common "input" attributes.

Data set
There are more than 200 hospitals which send information to TARN (TARN hospitals). This network is gradually increasing. Participation in TARN is recommended by the Royal College of Surgeons of England and the Department of Health. More than 93% of hospitals across England and Wales submit their data to TARN. TARN also receives data from Dublin, Waterford (Eire), Copenhagen, and Bern.
We use TARN data collected from 01.01.2008 (start of treatment) to 05.05.2014 (date of discharge). The database contains 192,623 records and more than 200 attributes. Sometimes several records correspond to the same trauma case because the patients may be transferred between TARN hospitals. We join these records. The resulting database includes data of 182,252 different trauma cases with various injuries. 16,693 records correspond to patients, who arrived (transferred from other institutions) to TARN hospitals later than 24 hours after injury. This sample is biased, for example the Fraction Of Dead outcomes (FOD) for this sample is 3.34% and FOD for all data is 6.05%. This difference is very significant for such a big sample. (If all the outcomes in a group of the trauma cases are known then we use the simple definition of FOD in the group: the ratio of the number of registered deaths in this group to the total number of patients there. Such a definition is not always applicable. The detailed and more sophisticated analysis of this notion follows in the next section.) We remove these 16,693 trauma cases from analysis but use them later for validation of the "mortality after transfer" model. Among them, there are 15,437 patients who arrived at a TARN hospital within 30 days after injury. We call this group 'IN30' for short (Fig. 1).
As a result we have 165,559 records for analysis ('Main group'). This main group consists of two subgroups: 146,270 patients from this group approached TARN during the first day of injury and remained in TARN hospitals or discharged to a final destination during the first 30 days after injury. We call this group the 'Available within 30 days after injury' cases (or 'Available W30D' for short). The other 19,289 patients have been transferred within 30 days after injury to a hospital or institution (or unknown destination) who did not return data to the TARN system. We call them 'Transferred OUT OF TARN within 30 days after injury' or just 'OUT30' (Fig. 1).
The patients with the non-final discharge destinations 'Other Acute hospital' and 'Other institution' were transferred from a TARN hospital to a hospital (institution) outside TARN and did not return to the TARN hospitals within 30 days after injury.
The database includes several indicators for evaluation of the severity of the trauma case, in particular, Abbreviated Injury Scale (AIS), Injury Severity Score (ISS) and New Injury Severity Score (NISS). For a detailed description and comparison of the scores we refer readers to reviews [30,32]. The comparative study of predictive ability of different scores has a long history [19,43,7,42]. The scores are used for mortality predictions and are tested on different datasets [52,29,53,4].

Definitions and distributions of outcomes
The widely used definition of the endpoint outcome in trauma research is survival or death within 30 days after injury [9,4,48].
A substantial number of TARN in-hospital deaths following trauma occur after 30 days: there are 957 such cases (or 8% of TARN in-hospital death) among 11,900 cases with 'Mortuary' discharge destination. This proportion is practically the same in the main group (165,559 cases): 894 deaths after 30 days in hospital (or 7.9%) among 11,347 cases with 'Mortuary' discharge destination.
Death later than 30 days after injury may be considered as caused by co-morbidity rather than the direct consequence of the injury [4]. These later deaths are not very interesting from the perspective of an acute trauma care system (as we cannot influence them), but they might be very interesting from the perspective of a geriatric rehabilitation centre or of an injury prevention program for elderly patients. On the other hand, when "end of acute care" is used as an outcome definition then a significant portion of deaths remains unnoticed. For example, in the 3332 trauma cases treated in the Ulleval University Hospital (Oslo, Norway, 2000-2004) 18% of deaths occurred after discharge from the hospital [48].
The question of whether it is possible to neglect trauma caused mortality within 30 days after trauma for the patients with the discharge destination 'Home', 'Rehabilitation' and other 'recovery' outcomes is not trivial [48]. Moreover, here are two questions: • How do we collect all the necessary data after discharge within 30 days after trauma -a technical question?
• How do we classify the death cases after discharge within 30 days after trauma; are they consequences of the trauma or should they be considered as comorbidity with some additional reasons?
The best possible answer to the first question requires the special combination of technical and business process to integrate data from different sources. The recent linkage from TARN to the Office for National Statistics (ONS) gives the possibility to access the information about the dates of death in many cases. It is expected that the further data integration process will recover many gaps in the outcome data. The last question is far beyond the scope of data management and analysis and may be approached from different perspectives. Whether or not the late deaths are important in a model depends on the question being asked. From the data management perspective, we have to give the formal definition of the outcome in the terms of the available database fields. It is impossible to use the standard definition as survival or death within 30 days after injury because these data are absent. We define the outcome 'Alive W30D' for the TARN database being as close to the standard definition as it is possible.
In the TARN database discharge destinations 'Home (own)', 'Home (relative or other carer)', 'Nursing Home', and 'Rehabilitation' are considered as final. If we assume that these trauma cases have the outcome 'Alive W30D' then we loose some cases of death. From the acute care perspective these cases can be considered as irrelevant. Let us accept this definition. There still remain many cases with unknown outcome. For analysis of these cases we introduce the outcome category 'Transferred'. In this category we include the cases which left the TARN registry to a hospital or other institution outside TARN, or to an unknown destination within 30 days. The relations between the discharge destinations and these three outcomes are presented in Table 1.
As we can see from Table 1, 19,289 trauma cases (or 11.35% of all cases) have unknown outcome. The first standard question is: can we delete these data and apply available case analysis? For this purpose we have to consider these outcome data as "Missing Completely at Random" (MCAR) [39,40,36,46]. This is definitely not the case. The group with unknown outcomes is exactly the 'OUT30' group. The probability of belonging to this group depends, for example, on the severity of injury (which can be measured, by the maximal severity, by NISS, by GCS or by another severity score). The χ 2 test of independence shows that transfer depends on the severity with p-value p < 10 −300 (this is the probability that such a strong dependence might appear by chance).
One can consider all these cases as alive because these patients have been alive at the point of discharge from TARN hospitals. If we consider all transferred as alive then the FOD is 6.35%. If we delete all the transferred patients (study only the Available W30D group) then the FOD is 7.2%. If we test this hypothesis on 15,437 patients of the group 'IN30' transferred to TARN hospitals from outside the network within 30 days after injury then we find that the nonzero mortality for them (3.10%).
The data table with known outcomes is necessary for further machine learning and the main goal is outcome prediction and risk evaluation.
We choose to remove the OUT30D group from data table but simultaneously to adjust the weights of the retained cases to compensate for the removal. The information about the OUT30D cases will be used in the construction of the weights. It is necessary to evaluate the mortality of the patients transferred from TARN before removing their records and reweighting of the rest. In the next section we develop, identify and validate Markov models for the analysis of the mortality of transferred patients.
Another method for handling missed outcomes is multiple imputation of the outcomes (about multiple imputations see, for example, [22]). Both methods use similar stochastic models of mortality and transfer. The large number of cases allows us to use the reweighting approach. A significant majority of the evaluated weights are between 0.9 and 1.1 (see Section 6).
1-α-ν Figure 4: The Markov model of mortality and transfer from TARN hospitals to hospitals outside TARN for the limit case of 'retarded transfer', when the lottery of transfer (Fig. 2 b) occurs every day after the lottery of survival (Fig. 2 a). It has the same states as the model with advanced transfer (Fig. 3) but different transition probabilities.

4.
Non-stationary Markov model for the analysis of missing outcomes 4.1. Structure of model We propose a system of Markov models for evaluation of mortality in trauma datasets. In these models each day each patient can participate in two 'lotteries' (Fig. 2). The first lottery (recovery/death), Fig. 2 a, has three outcomes: 'R' (recovery), 'D' (death), and 'H' (remains in a TARN hospital). The second lottery (of transfer), Fig. 2 b, has two outcomes: 'H' (remains in a TARN hospital) and 'L' (transfer from the TARN hospital to a hospital or 'other institution' outside TARN). The probabilities of outcomes depend on the time from the injury t and on the state of the patient after injury s. It is important to stress that s in our models characterizes the state of the patient on the first day after trauma and may include severity, type of injury (blunt/penetrating), localization of traumas, age, gender, airway status, systolic and diastolic blood pressure, etc, but cannot change in time.
The description of state s may vary in the level of detail depending on the available information. We have fitted and tested two models based on the severity of trauma: the maximal severity model and the (binned) NISS model. In Section 5 we demostrate that it is necessary to refine the model and to include the age group in s for low severities. For different purposes the mortality model can include more detail.
The lotteries (Fig. 2) do not commute. We consider two limit cases: 'advanced transfer' (Fig. 3) and 'retarded transfer' (Fig. 4). In models with advanced transfer the lottery of transfer Fig. 2 b) each day precedes the lottery of recovery/death (Fig. 2 a). In models with retarded transfer, conversely, the lottery of recovery/death precedes the lottery of transfer.
These two models are important because many other much more general Markov models are between them in the following exact sense. It is a very strong assumption that every day there are two steps only: the recovery/death lottery and the transfer lottery. It may be more realistic to assume that every day there are many 'fractional steps' of recovery/death and of transfer from TARN and the result of the day is the aggregate result of all of these fractional steps. Assume that the events of recover, death and transfer are sampled for every day after injury t from a number M consecutive random choices with probabilities α i , ν i for recovery/death and µ i for transfer out of TARN (i = 1, . . . , M ), and this chain of choices is Markovian (the choices for a patient do not depend on the previous choices directly but only on the current state, H, R or L). It is non-stationary because the transition probabilities depend on time. They are different for different days after injury.
This sequence of choices is displayed as a sequence of fractional steps: The probability of in-TARN death in the above model of sequential choice, on a given day after trauma is Similarly, the probability for recovery is Finally, the probability of transfer to a hospital outside of TARN is The probabilities α i , ν i for the fractional steps should be consistent with the daily probabilities α, ν: if there is no transfer then the resulting probabilities of recovery or death should be the same: Similarly, for µ i we get the conditions (2) Proposition 1. The probability of in-TARN death in the described model of sequential choice for every day after trauma is between the probabilities for the Markovian model with advanced transfer (Fig. 3) and the Markovian model with retarded transfer (Fig. 4): Proof. According to conditions (1), (2), Notice that for every and the following inequality holds The left inequality in (3) is proven. The right inequality in (3) follows from condition (1) because for every product The proofs of the following propositions are very similar Proposition 2. The probability of in-TARN recovery in the described model of sequential choice for every day after trauma is between the probabilities for the Markovian model with advanced transfer (Fig. 3) and the Markovian model with retarded transfer (Fig. 4): Proposition 3. The probability of transfer outside TARN in the described model of sequential choice for every day after trauma is between the probabilities for the Markovian model with advanced transfer (Fig. 3) and the Markovian model with retarded transfer (Fig. 4):

Transition probabilities and their evaluation
In the above models (Figs. 3 and 4), death and recovery of the transferred patients have the same probabilities as for the patients of TARN hospitals. These probabilities are defined by the state of the patient s and by the time after injury. Of course, in reality there is often a hope that the transfer will improve the situation and the probability of death will decrease for the same state of the patient. Nevertheless, in this paper we will neglect the changes of probabilities after transfer (just because we have no sufficient reason for such a change). Of course, these models could be extended to include the changes of mortality for transferred patients, if necessary.
Another question is the definition of s. Which attributes should be included in the 'state' for the models (Figs. 3, 4)? To motivate this choice, we should take into account two considerations: 1. The models will be used to analyse data with unknown outcomes. Trauma cases with missed outcomes make up 10-12% of the dataset. Therefore, an error of 10% in mortality for data with unknown outcomes will cause an error of ∼1% in mortality for the whole dataset and it is possible to use relatively coarse models (see below). 2. The description of the state s should include attributes whose values are known for a significant majority of cases. This is especially important because for cases with unknown outcomes many of the attributes are often also unknown (a more detailed analysis of data with missed attributes is presented in the next section).
Formally, there are many possibilities for defining s. It could include the initial state after trauma (characteristics of injury and coma status, for example), age, gender, the current state (t days after trauma), fragments of history, etc. For our purposes, we select, identify and compare three coarse models: 1. The coarsest model s = ∅.
2. The maximal severity model, s =the maximal severity score (an integer from 1 to 6). We observe that the cases with maximal severity 1 (or NISS=1-3, which is the same) are very special. First of all, the age distributions in this group for the 'Available W30D' and the 'OUT30' subgroups are very different (Fig. 5). If we do not take into account this difference then we overestimate mortality in this group. The necessary refinement of the model with isolation of elderly patients with low severity of trauma is presented in Section 5. Our approach may be combined with any stochastic model for early outcome prediction (see, for example, [29,47,5]).
For the finite set of s values, evaluation of all the coefficients α(t, s), ν(t, s), and µ(t, s) is a particular case of a standard statistical problem of proportion estimate for each given value of s; we use the Wilson score interval (CI) [56]: wherep is the coefficient estimate, z is the error percentile (z = 1.96 for the 95% confidence interval), and n is the number of degrees of freedom (for a dataset without weights this is just the sample size). For the coarsest model the fraction of patients transferred outside TARN is 11.65%. This is just the fraction of patients transferred (within 30 days after injury) in Table 1. The 95% CI (8) for this fraction is 11.5-11.8%. For the maximal severity (Table 2) and the binned NISS (Table 3) models the fraction of patients transferred outside TARN depends on s (bins) and the CI in each bin is larger than for the total fraction in the coarsest models. Nevertheless, the CIs for different bins in these models do not intersect (the only exclusion is the CI for the smallest bin, maximal severity 6, in the maximal severity model, Table 2). In particular, this means that the probability of transfer outside TARN hospitals depends strongly on the trauma severity.
For each value of s and time after injury t the following quantities are found for the analysed dataset: • H(t, s) -the number of patients in state s registered as alive in a TARN hospital at any time during day t after injury (in this number we include the patients which have stayed at a TARN hospital during day t after injury, the patients who have died on this day in a TARN hospital, have been discharged, or have been transferred outside TARN on this day); • ∆D(t, s) -the number of patients in state s who died in TARN hospitals on day t after injury; • ∆R(t, s) -the number of patients in state s who recovered (discharged to one of the final recovery destinations) in TARN hospitals on day t after injury; • ∆L(t, s) -the number of patients in state s who transferred out of TARN hospitals to other hospitals, institutions or unknown destinations on day t after injury.
Just for control, the following identity should hold: s) because state s in our models does not change in time.
For the model with advanced transfer from TARN hospitals the coefficients are defined following the scheme presented in Fig. 3: For the model with retarded transfer from TARN hospitals the coefficients are defined following the scheme presented in Fig. 4:

Evaluation of FOD
Each model provides us with the corrected FOD. We use the basic assumption that the probability of dying at time t after injury depends on s but is the same inside and outside TARN. For each t and s we define the specific cumulative FOD (scFOD(t, s)) as the fraction of patients  with state s who died during the time interval [1, t]: The cumulative FOD at time t (cFOD(t)) for the whole model (for all s together) is where H 0 = s H(1, s) is the total number of patients in our dataset (in our case study, H 0 = 165, 559). The functions cFOD(t) and scFOD(t, s) for all s, grow monotonically with t.
If we define the final outcome as survival or death within 30 days after injury then the target value is FOD= cFOD (30).
Let us compare two following naïve approaches to the handling of missing outcomes with the Markov models we have created.
• Available case analysis. Just delete all of the 19,289 cases with the outcome 'Transferred OUT OF TARN within 30 days after injury' from the dataset. In the remaining cases all outcomes are known and the FOD is the ratio Dead (W30D) Total in the reduced dataset.
• Consider all transferred patients as alive. In this case, the total number of patients does not change and the FOD is the ratio Dead (W30D) Total , where the number 'Dead (W30D)' is the same but the number 'Total' is calculated for the whole original dataset (Table 1).

Remark 1.
If we apply available case analysis then none of the numbers ∆D(t, s) and ∆R(t, s) change but the numbers H(t, s) of the patients in TARN will decrease for all t and s (or do not change if there is nothing to delete). The corresponding mortality coefficients ν(t, s) they will be larger then the coefficients (9), (10) for all the Markov models considered before. This means that the MCAR (Missing Completely At Random) approach to missed outcomes always overestimates mortality, while the second naïve approach ('Consider all transferred patients as alive') always underestimates mortality.
We have created six Markov models for mortality of transferred patients. They differ by the state variable s (the coarsest model without s, the maximal severity model with six states and the binned NISS model with seven states) and by the order of the 'recovery/death' and 'transfer' lotteries ( Fig. 2). In Table 4 we compare the mortality evaluated by these models, and by the two naïve models. We can see that the difference between all of our Markov models is not significant; we cannot reject the hypothesis that they coincide with any one of them (p-value is between 0.20 and 0.56). Both of the naïve models differ significantly from all of the six Markov models. The difference between the naïve models is also significant. The interval of mortality predicted by the Markov models is (6.77%, 6.91%). The average of the six Markovian predictions is 6.84%. None of the Markov model predictions differ significantly from this average. Both of the naïve predictions are significantly different.

Validation of the models on the excluded trauma cases:
patients transferred to TARN ('IN30') For each type of model the coefficients µ, α and ν are evaluated using the dataset of 165,559 patients entering TARN in the first day of injury (Fig 1, Main Group). Let us test the models with evaluated coefficients we have described here on data we have not used before. These data consist of the 16,693 cases who came to TARN hospitals more than one day after injury, which we deleted from the original set before modelling. This is a special and biased sample, 'IN30' (see Fig. 1). We now apply the models developed and identified in the previous subsections to analyse this sample. We expect that there should be some similarity between the groups of patients transferred from TARN ('OUT30') and the patients transferred to TARN ('IN30') ( Fig. 1). We do not expect quantitative coincidence of the results for the groups 'OUT30' and 'IN30' because there is no precise symmetry between the patients moved to TARN and the patients moved from TARN. The hospitals in TARN are those with a special interest in trauma -in particular the large major trauma centers, so the transfers in (mainly for acute specialist care) will not be the same as those transferred out (mainly for complex rehabilitation, or special geriatric care, etc.). Therefore, the estimated behavior of the mortality of the group transferred from TARN can be qualitatively validated using the observed mortality in the group who moved to TARN.
We consider survival during the first 30 days. Hence we have to use the records which correspond to this period only. There are 15,437 such records among the 16,693 in 'IN30'.
In these estimates of the FOD we explicitly use the empirical fluxes into and from TARN hospitals. For each t, s we have the following quantities: • L in (t, s) -the number of patients in state s which came to TARN on day t after injury; • L out (t, s) -the number of patients in state s from 'IN30' which were transferred from TARN on day t after injury.
with initial condition For each model, the coefficients α(t, s) and ν(t, s) are evaluated using the previously analysed dataset (without IN30) by formulas (9) and (10). The results are presented in Table 5.
We can see that all the models overestimate mortality in 'IN30'. The available case analysis demonstrates the worst performance (the relative error exceeds 100% of empirical mortality). Models with retarded transfer perform better in this test than the models with advanced transfer. The NISS binned model with retarded transfer is the best (the relative error in prediction of FOD is 16% of the empirical data and, at least, the 95% confidence intervals for the result of this model and for the empirical data intersect). There exist further possibilities for improving the models presented but already the relative error of 16% for 'IN30' in the estimation for the total database will give the input in the relative error in the FOD 1% (or absolute error 0.07%). That is much better than the errors of the available case evaluations or of the approach 'all are alive' to the evaluation of mortality of transferred patients.

Model refinement
We use a coarse model based on the severity of trauma for the evaluation of FOD in the group 'OUT30'. The reason for selection of such a coarse model is that a fraction of cases in this 'OUT30' cohort is relatively small with respect to the'Available W30D' cases. As we can see from Table 3, this fraction is relatively small in all cells except small severities (NISS bin 1-3). For refinement of the Markov model for this cell, we compare the age structure of the 'Available W30D' and the 'OUT30' fractions of this severity bin (Fig. 5). We see that the fraction of elderly patients with low severities in TARN hospitals is high, whereas for patients transferred from TARN this fraction is much lower. Mortality in the group of patients 74.5+ is much higher than in the adult group, therefore the model overestimates mortality in the low severity states. To refine the model let us use two cells for low severity: 'NISS 1-3 y' (NISS bin 1-3 and age < 54.5) and 'NISS 1-3 o' (NISS bin 1-3 and age > 54.5). This refined model gives a significantly different FOD for NISS 1-3. In the cell 'NISS 1-3 y' the corrected FOD is 0.54% and in the cell 'NISS 1-3 o' it is 4.08% (almost eight times greater). The corrected overall FOD for NISS 1-3 is 1.42% versus 2.68% in the NISS retarded model without the above refinement.
The effect of the refinement on the FOD for trauma cases is less because the fraction of traumas with NISS severity 1-3 is relatively small (2.0%). For the refined model with retarded transfer the FOD for transferred patients decreases from 3.79% (retarded transfer NISS model) to 3.59% and the total fraction of death is changed from 6.81% to 6.78% (compare to Table 4).

Weighting adjustment of death cases for further analysis
Single imputation of missed values does not reflect the uncertainty in data properly. From the probabilistic point of view, a datapoint with missed values should be considered as a conditional probability distribution of the form P(missed values | known values).
Two approaches utilize this idea the multiple imputation and weighting adjustment.
In the multiple imputation approach several replicas of the database are created, which differ in the imputed values [40,41,23,51]. The distribution of this values should reflect the conditional means and conditional variances of the imputing attributes. It is not completely clear, how many imputations should be generated. Rubin claims that "typically as few as five multiple imputations (or even three in some cases) is adequate under each model for nonresponse" [41]. Nevertheless, more recently, Graham et al produced practical recommendations for selection of number of imputations m and demonstrated that a reasonable choice is m ≥ 20 and for some cases m = 100 is not enough [23]. The multiple imputation algorithms are implemented in the standard statistical software [38]. Sterne et al [51] discussed use and misuse of imputation in epidemiological and clinical research and tried to produce a standard for reporting of handling of missed data in medical research.
The weighting adjustment approach substitutes a datapoint with missed values by a set of additional weights on the complete datapoints [33,26,34]. The simplest version of this approach is the cell weighting adjustment. This follows the assumption that complete datapoints within a cell represent the incomplete datapoints within that cell. An incomplete datapoint within the cell is substituted by the equidistribution on the complete datapoints there. Of course, cell weighting can inflate the variances for large cells. In this section, we use cell weighting adjustments for the handling of missed outcomes. Cells are defined by state s and the outcome.
We will use the database for evaluation of the death risk for trauma patients. The 'Main Group' selected for further analysis includes the 'OUT30' subgroup with 19,289 data cases transferred from TARN hospitals within 30 days after injury (Fig 1). The targeted outcome (alive or dead within 30 days after injury) is unknown for these patients. Data without outcome cannot be used for risk evaluation and should be deleted. Let us call the result of deletion the truncated database. It is demonstrated in the previous sections that the simple removal of the cases with unknown outcome shifts the risk estimates; the proportion of Dead and Alive outcomes in the truncated database differs from reality and the risk is overestimated (the pessimistic evaluation). This bias may be compensated by reweighting of the cases with known outcomes. There are 146,270 such 'Available W30D' cases. In this subsection we estimate weights w(t, s) that should be assigned to the cases of death on day t after injury with state s to hold the probability of death for the truncated database. For the estimation of the proper FOD that should be kept we use the Markov model of mortality based on binned NISS with delayed transfer out of TARN (after selection dead and recovered patient, see Fig. 4). This model demonstrates the best verification results ( Table 5) and is the most plausible from the common sense point of view.
According to the model, the probability of the patient in state s dying on day t after injury is evaluated as where H 0 (s) = H(1, s) is the initial number of patients in state s on the first day after injury. For the truncated data with weights this probability is evaluated as the ratio of the sums with weights: This should be the same number. Therefore, the weighted sum of deaths for the truncated database is: (H(31, s) + R(30, s)).
The last expression in the brackets is just the number of 'Alive within 30 days' outcomes. Immediately we get (30, s)).
The formula for the calculation of the weights of death cases in the truncated database is The weighting procedure changes the number of effective degrees of freedom can affect the statistical power of the dataset but for the TARN dataset this change is rather minor. For example, for the standard problem of the evaluation of the confidence interval in the proportion estimate the number of degrees of freedom n w in the weighted database with weights w i is For our dataset n w = 143, 574.85 and the number of Available W30D records is 146,270 (Fig. 1). The difference of degrees of freedom for the non-weighted and weighted datasets is less than 2%.

FOD and patterns of mortality
The models we have developed alow us to evaluate the FOD for various groups of patients. The rich TARN data give us the chance of studying various special groups and detailed stratifications of the trauma cases: by the severities of various injuries in combined traumas, by the age of patients, and by time (day) after trauma. Each example below is supplemented by a medical commentary.

Example: FOD as function of age
The age distribution of trauma cases and the dependence of FOD on age are shown in Fig. 6. Here we find surprisingly high accuracy of the piecewise linear approximation of FOD for adult and elderly patients with a jump in the slope at age ≈ 62.
The number of cases per year in the dataset drops down at age 65 because for age≥ 65 some traumas are excluded from the database (see Fig. 1). Age   FOD   22  20  18  16  14  12  10  8  6  4  2  0  100  90  80  70  60  50  40  30  20  Medical commentary. The increase in mortality with age is well established. Previous versions of the standard trauma outcome prediction system had two different models with an age cutoff at 55 years. More recent models have age as a weighted continuous variable with an interaction term between gender and age. There has been a dramatic change in the trauma population over the last 10 years, with a rapid increase in the number of older patients with major injury. Understanding the effects of age on trauma care and adapting to a changing population will be a key challenge for trauma systems in the developed world over the next 10 years. (s 1 ≥ s 2 ≥ s 3 ) (see, for example [29,53,52]). The best severity score should give the best evaluation of mortality. This is a basic and rather old idea for defining and comparing trauma indices [44]. Of course, it is possible to use three (or more) severities together as a multidimensional trauma severity index ('severity profile' [43]) but the combination in one index may be beneficial from different points of view. The simplest method of combination is:

Fraction of cases
• Calculate FOD for every combination of severities for combined traumas for a large database; • Either use this FOD instead of the severity score • Or find and use a convenient analytic approximation for this FOD (smoothed FOD).
Of course, such evaluation of probabilities for several input attributes were used by many authors and compared to other approaches [4,5]. In this paper, we use TARN database and evaluate FOD of combined traumas as a function of three input attributes, three biggest severity scores s 1 ≥ s 2 ≥ s 3 (like in NISS). We use the dataset of 146,270 'Available W30D' patients approached TARN during the first day of injury and remained in TARN or were discharged to a final destination within the first 30 days after injury (Fig. 1).
Using our models, we calculate estimates with weights which take into account modeled mortality/survival of the patients transferred from TARN and other patients with unknown outcomes. Results for the maximal severity s 1 = 5 are presented in Table 6. The available case analysis gives qualitatively the same results, hence, the effects we observe are not generated by the reweighting procedure.
The results presented in Table 6 seem to be counterintuitive: FOD for combined injuries with severities s 1 = 5 and 1 ≤ s 2 ≤ 4 are less than FOD for s 2 = s 3 = 0 and the same maximal severity s 1 = 5. Similar non-monotonic behavior is observed for other values of the maximal severities. Elementary estimates demonstrate that the probability p of obtaining these (or larger) deviations to below from the FOD for single injuries (s 1 = 5, s 2 = s 3 = 0) for all cases with 1 ≤ s 2 ≤ 4 simultaneously is less than 10 −10 . The number of cases used for these estimates are given in Table 7. If the second severity coincides with the maximal one, s 2 = s 1 = 5 then the FOD is larger than for single traumas.
All the coefficients are estimated using weighted least squares method. The weight of the severities combination (s 1 , s 2 , s 3 ) is defined as the sum of weights of the corresponding trauma cases.
Medical commentary. The complete outcome dataset derived from this work allows all patients to be included in the analysis of the effect of combined injuries. The counter-intuitive results from this analysis (some combinations of injuries seem to have better outcomes than a single injury of the same severity) provides a fertile area for further work. It may be that the explanation is technical, within the way that the continuum of human tissue destruction from trauma is reduced to a simple 5 point scale. Each point on the scale is actually a band that covers a range of tissue damage. There might also be a true physiological explanation for the lower lethality of combined injuries, as each injury absorbs some of the force of impact. The same concept is used in Formula 1, where the cars are designed to break into pieces, with each piece absorbing some of the impact. In humans there is a well known concept that the face can act as a crumple zone and mitigate effect of force on the brain. The effect of injury combinations shown in Table 6 is a novel finding that requires further analysis.
7.3. Example. Time after trauma, non-monotone and multimodal mortality coefficients In the early 1980s a hypothetical statement was published that the deaths from trauma have a trimodal distribution with the following peaks: immediate, early and late death [3,35]. This concept was clearly articulated in a popular review paper in Scientific American [55]. The motivation for this hypothesis is simple: Trunkey [55] explains that the distribution of death is the sum of three peaks: "The first peak ('Immediate deaths') corresponds to people who die very soon after an injury; the deaths in this category are typically caused by lacerations of the brain, the brain stem, the upper spinal cord, the heart or one of the major blood vessels. The second peak ('Early deaths') corresponds to people who die within the first few hours after an injury; most of these deaths are attributable to major internal hemorrhages or to multiple lesser injuries resulting in severe blood loss. The third peak ('Late deaths') corresponds to people who die days or weeks after an injury; these deaths are usually due to infection or multiple organ failure." Strictly speaking, the sum of three peaks does not have to be a trimodal distribution. Many groups have published refutations of trimodality: they did not find the trimodal distribution of death. In 1995, Sauaia et al reported that the "greater proportion of late deaths due to brain injury and lack of the classic trimodal distribution" [45]. Wyatt et al could not find this trimodal distribution in data from the Lothian and Borders regions of Scotland between 1 February 1992 and 31 January 1994 [57]. They hypothesised that this may be (partly) due to improvements in care.
Recently, more data has become available and many such reports have been published [12,27,6]. The suggestion that the improvement in care has led to the destruction of the second thd third peaks has been advanced a number of times [27]. In 2012, Clark et al performed an analysis of the distribution of survival times after injury using interval censored survival models [10]. They considered the trimodal hypothesis of Trunkey as an artifact and provide arguments that the observed (in some works) second peak is a result of differences in the definition of death.
K. Søreide et al analysed the time distribution from injury to death stratified by cause of death. They demonstrated that the trimodal structure may be, probably, extracted from data but its manifestation is model-dependent (see Fig. 6 in [50]). There were several discussion papers published: "Trimodal temporal distribution of fatal trauma -Fact or fiction?" [2,28].
The trimodal hypothesis was tested on TARN data [31]. It was demonstrated that "the majority of in hospital trauma deaths occur soon after admission without further peaks in mortality". We reproduce the same results, indeed. But TARN database, the largest European trauma database, allows us to make a stratified analysis of mortality and the preliminary results demonstrate the richness of the possible patterns of death.
Let us test the famous Trunkey hypothesis. In Fig. 7 the daily mortality coefficients are presented for low severities (a) (NISS severities 1-8, 27,987 cases in database, 508 death in TARN, 3,983 patients transferred from TARN within 30 days after injury), and for the whole database (b). For the prediction of death in the 'OUT30' group we used the model with retarded transfer.
The non-monotonicity and peaks in the mortality for low severities of injury are illustrated in Fig. 7. Further analysis of these patterns should involve other attributes such as the age of the patient and the type and localization of the injury.  Medical commentary. It has been widely accepted that the Trunkey trimodal distribution was a theoretical concept designed to illustrate the different modes of dying following injury. Previous analysis of trauma data has looked at all patients and has not shown any mortality peaks, however this new analysis shows that there are peaks (patterns) if subgroups are studied. The underlying clinical or patient factors are not immediately obvious, but future analysis giving a better understanding of patterns of death could act as a stimulus to look for the clinical correlates of these patterns -with the potential to find modifiable factors. The pattern of death in various subgroups as shown in Figure 7 is a novel finding that requires further analysis.

Discussion
Handling of data with missed outcomes is one of the first data cleaning tasks. For many healthcare datasets, the problem of lost patients and missed outcomes (in 30 days, in six months or any other period of interest) is important. There are two main approaches for solving this problem: 1. To find the lost patients in other national and international databases; 2. To recover the distribution of the missed outcomes and all their correlations using statistical methods, data mining and stochastic modelling.
Without any doubt the first approach is preferable if it is available: it is better to have complete information when it is possible. Nevertheless, there may be various organizational, economical and informational restrictions. It may be too costly to find the necessary information, or this information may be unavailable or even does not exist in databases. If there are only small number of lost cases (dozens or even hundreds) then they may be sought individually. However if there are thousands of losses then we need either a data integration system with links to appropriate databases like the whole NHS and ONS data stores (with the assumption that the majority of the missed data may be taken from these stores) or a system of models for the handling of missed data, or both because we might not expect all missed data to be found in other databases.
In the TARN dataset, which we analyse in this paper the outcome is unavailable for 19,289 patients. The available case study paradigm cannot be applied to deal with missed outcomes because they are not missed 'completely at random'. Non-stationary Markov models of missed outcomes allow us to correct the fraction of death. Two naïve approaches give 7.20% (available case study) or 6.36% (if we assume that all unknown outcomes are 'alive'). The corrected value is 6.78% (refined model with retarded transfer). The difference between the corrected and naïve models is significant, whereas the difference between different Markov corrections is not significant despite the large dataset.
Non-stationary Markov models for unknown outcomes can utilize any scheme of predictive models with using any set of available attributes. We demonstrate the construction of such models using maximal severity model, binned NISS model and binned NISS supplemented by the age structure at low severities. We use weighting adjustment to compensate for the effect of unknown outcomes. The large TARN dataset allows us to use this method without significant damage to the statistical power.
Analysis of mortality for a combination of injuries gives an unexpected result. If s 1 ≥ s 2 ≥ s 3 are the three maximal severities of injury in a trauma case then the expected mortality (FOD) is not a monotone function of s 3 , s 3 , under given s 1 . For example, for s 1 = 4, 5 expected FOD first decreases when s 2,3 grow from 0 to 1-2 and then increases when s 2 approaches s 1 .
Following the seminal Trunkey paper [55], multimodality of the mortality curves is a widely discussed problem . For the complete TARN dataset the coefficient of mortal-ity monotonically decreases in time but stratified analysis of the mortality gives a different result: for lower severities FOD is a non-monotonic function of the time after injury and may have maxima at the second and third weeks after injury. Perhaps, this effect may be (partially) related to geriatric traumas.
We found that the age distribution of trauma cases is strongly multimodal (Fig. 6). This is important for healthcare planning.
The next step should be the handling of missed values of input attributes in the TARN database Firstly, we should follow the "Guidelines for reporting any analysis potentially affected by missing data" [51], report the number of missing values for each variable of interest, and try to "clarify whether there are important differences between individuals with complete and incomplete data". Already preliminary analysis of the patterns in the distribution of the missed input data in the TARN dataset demonstrates that the gaps in data are highly correlated and need further careful analysis. Secondly, we have to test and compare various methods of handling missing input attributs in the TARN database.
It is not necessary to analyse all attributes in the database for mortality prediction and risk evaluation. It is demonstrated that there may exist an optimal set of input attributes for mortality prediction in emergency medicine and additional variables may even reduce the value of predictors [20]. Therefore, before the analysis of imputation efficiency, it is necessary to select the set of most relevant variables of interest.
The models developed in this case study can be generalized in several directions. Firstly, for trauma datasets, different attributes could be included in the 'state' s for the non-stationary Markov models (Figs. 3, 4). We did not explore all such possibilities but have studied just simple models of the maximal severity and binned NISS. An example of model refinement with inclusion of age in the state variable s is presented in Section 5. Secondly, the 'two stage lottery' non-stationary Markov model could be used as a general solution applicable to any health dataset where 'TRANSFER IN' or 'TRANSFER OUT' is a feature. Transfer between hospitals is common in healthcare, therefore, we expect that models of this type will be useful for all large healthcare data repositories. Analysis of the missed outcomes demonstrated that they cannot be considered as misses 'completely at random'. Therefore, the analysis of available cases is not applicable for the TARN database. Special efforts are needed to handle data with missed outcomes. 3. We have developed a system of non-stationary Markov models for the handling of missed outcomes and validated these models on the data arising from patients who moved to TARN (and excluded from the model fitting). We have analysed mortality in the TARN database using the Markov models which we have developed and also validated. 4. The results of analysis were used for weighting adjustment in the available cases database (reweighting of the death cases). The database with adjusted weights can be used for further data mining tasks and will keep the proper fraction of deaths. 5. The age distribution of trauma cases is essentially multimodal, which is important for healthcare planning. 6. Our analysis of the mortality coefficient in the TARN database demonstrates that (i) for complex traumas the fraction of death is not a monotone function of all severities of injuries and (ii) for lower severities the fraction of death is not a monotonically decreasing function of time after injury and may have intermediate peaks in the second and third weeks after injury. 7. The approach developed here can be applied to various healthcare datasets which have the problem of lost patients, inter-hospitals transfer and missing outcomes.