{'fraction_non_alphanumeric': 0.032952023220078536, 'fraction_numerical': 0.011738091172955437, 'mean_word_length': 4.806443618339529, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
CRFs based de-identification of medical records

De-identification is a shared task of the 2014 i2b2/UTHealth challenge. The purpose of this task is to remove protected health information (PHI) from medical records. In this paper, we propose a novel de-identifier, WI-deId, based on conditional random fields (CRFs). A preprocessing module, which tokenizes the medical records using regular expressions and an off-the-shelf tokenizer, is introduced, and three groups of features are extracted to train the de-identifier model. The experiment shows that our system is effective in the de-identification of medical records, achieving a micro-F1 of 0.9232 at the i2b2 strict entity evaluation level.


Introduction
Clinical documents contain much valuable information that is significantly attractive to researchers. However, a large amount of protected health information (PHI) exists in clinical documents, and may reveal identities or other personal private information of the patients. To avoid exposing private information, PHI must be removed before researchers do research on these clinical documents. Hence, identifying PHI from clinical documents is essential.
De-identification is the process of identifying and removing PHI from medical records. However, manually de-identifying medical records is time consuming and has significant costs. In 2006, de-identification was first introduced into the Informatics for Integrating Biology and the Bedside (i2b2) project [1]. Automatically removing PHI from discharge summaries was challenging. In this task, Wellner et al.'s method [2] obtained the highest score, using an existing toolkit that was based on conditional random fields (CRFs). Methods for de-identification generally fall into three categories: rule-based methods, machine-learning methods and a combination of the two. Meystre et al. [3]  analysis. The authors concluded that the rule-based methods perform better with PHI that is rare in clinical documents, whereas machine-learning methods are good at identifying PHI that is not in the dictionaries. For example, "Johnson and Johnson" is a PHI term of ORGANIZATION and appears less frequently in clinical documents, but rule-based methods can identify it as ORGANIZATION rather than NAME; machine-learning methods can strongly associate the word "Johnson" in "Johnson was admitted to hospital yesterday" with PATIENT, although "Johnson" is not in the dictionary of PATIENT. Rule-based methods need experts to construct rules or patterns manually to identify PHI entities. For example, Neamatullah et al. [4] proposed a pattern-matching de-identification system for nursing progress notes, which were obviously less structured than discharge summaries were, using dictionary look-ups, regular expressions and heuristics. In nursing progress notes, technical terminologies, non-standard abbreviations, ungrammatical statements, misspellings, incorrect punctuation characters, and capitalization errors occur frequently. The evaluation showed that their method experienced a high recall rate. Compared with rulebased methods, machine-learning methods such as conditional random fields, support vector machines (SVM) and so on are chosen by many more researchers for de-identification. Zuccon et al. [5] developed a tool to de-identify free-text health records based on conditional random fields. Linguistic features, lexical features, and features extracted by pattern matching were applied to train the classifier. Uzuner et al. [6] used support vector machines and local context to de-identify medical discharge summaries. They created an effective deidentifier and found that local context could improve the effect of de-identification. Boström and Dalianis [7] utilized an active learning method to train a random forest de-identifier on Swedish health records. Szarvas et al. [8] proposed an iterative learning method to deidentify discharge summaries using decision trees with local features and dictionaries. Deidentification became a track in i2b2 shared tasks again in 2014. In the de-identification task, many of the teams employed methods using machine learning and rules [9].
Compared with the de-identification task in 2006, medical records and PHI categories in the 2014 i2b2 shared tasks have changed significantly. First, the content in the medical records is not tokenized, and many irregular terms exist in which one token may contain two words. For example, the token "Since6/03/04" actually means "Since 6/03/04". Furthermore, many abbreviations exist in medical records, which may decrease the precision of the tokenizer and part of speech (POS) tagger. Moreover, the number of PHI categories in i2b2 2014 deidentification is far greater than in 2006. Consequently, de-identification of medical records in i2b2 2014 has become a new challenge for researchers.
In this paper, a new de-identifier of medical records, WI-deId, is proposed based on an implementation of conditional random fields. There has been a significant focus on preprocessing and feature generation from medical records. In the following sections, the details of the preprocessing procedure in WI-deId will be described, and features used in the classifier will be listed. Moreover, the experiment results and future directions of WI-deId will be discussed.

Methods
In this study, we utilized the medical records from task 1 of 2014 i2b2/UTHealth shared tasks. The dataset consists of 1,304 medical records annotated for de-identification; 790 of these are used for training, and the remaining 514 are used for testing. The PHI entities have been grouped into 7 main categories and 25 sub-categories, and these medical records have been annotated according to annotation guidelines formulated by specialists [10].
The flow diagram of the WI-deId system is shown in Figure 1. The WI-deId system consists of four main modules: 1) the preprocessing module, 2) the feature generation module, 3) the CRFs training module that trains a model based on the features generated from the second module, and 4) the CRFs decoding module that identifies the PHI entities from unseen data using the model trained in the third module. The details of the WI-deId system will be described in the following sections.

Preprocessing of Medical Records
Medical records in the dataset of 2014 i2b2 de-identification task are not tokenized, which makes automatically identifying PHI from the records very difficult. Therefore, focused on tokenization, the preprocessing module was sequentially performed in three steps. First, the sentence boundary detector and tokenizer modules in an existing open source toolkit, OpenNLP 1 , were utilized to split sentences and tokens in the medical records, respectively. Second, pattern-matching techniques were used to recognize some specific tokens such that when some tokens should be split into two parts, they are split based on the corresponding rules. All regular expressions used in the preprocessing module are listed in Table 1, and how to split the matched tokens is shown in the table. Finally, punctuation characters ("",:;#*-=/<>[]{}) were segmented from other characters. For example, "2106-02-12" is split as "2016 -02 -12".

Feature Generation
Before model training, the WI-deId system must extract a large number of features from the medical records for training. The features, which are listed in Table 2, can be categorized into three groups: lexical features, orthographic features, and dictionary features.
Lexical features contain the lowercase of the token and the first and last four characters of it [11]. These features can help the classifier "memorize" the categories of some tokens, which is necessary because some tokens often belong to one of the PHI categories, but some often belong to a non-PHI category. For instance, "April" is always recognized as a PHI term, and "at" is always identified as a non-PHI term. Moreover, the POS tag of a token is always very helpful in named entity recognition. WI-deId found the POS tag using the tool GENIA [12], which was trained on biomedical text, instead of using OpenNLP, whose POS tagger is trained on open domain corpus.
Tokens that are similar in shape may be classified into the same PHI category. We replaced uppercase letters, lowercase letters and digits in a token by "A", "a" and "0", respectively.
Length of a token is a significant feature of tokens in some PHI categories. For example, the tokens in ZIP, AGE, and PHONE are mostly fixed length. Information about capital letters can help us identify PATIENT, DOCTOR and other PHI entities that mostly begin with a capital letter such as "Nick" and "Hayes". Digit-related features can aid in identification of the PHI entities that contain digits such as ZIP, AGE, and DATE. Moreover, PHONE, DATE and some other PHI categories can benefit from punctuation features because there usually is a punctuation character in these PHI entities. Round brackets occur often in the entities of PHONE, such as "(784) 032-8966", and "-" or "/" frequently appears in DATE.
In addition, we extracted PHI entities of CITY, STATE, STREET, COUNTRY, and DATE in the medical records for training and combined them with webpages 2 of city, state, and country to generate the dictionaries. In these dictionaries, all of the elements are tokens, not phrases, and all of the tokens are lowercased. For example, "Los Angeles" is a PHI term of CITY, but "los" and "angeles" are separated elements in the dictionary of CITY. Dictionary features are generated by judging whether the lowercase of the token is in the dictionary. Note that the dictionary features cannot classify one token into its dictionary type directly; this function is different from post-processing. If one token belonging to COUNTRY does not appear in the training data, it will not be classified into COUNTRY in the testing data, although the token is one element of the COUNTRY dictionary. This behavior may weaken the effect of the dictionaries, but it also reduces problems caused when a token belongs to two or more dictionaries simultaneously.
In the WI-deId system, all of the above-mentioned features of the tokens within a ±2 context window of the current token are considered.

The CRFs Classifier
The proposed classifier was based on the conditional random fields algorithm which was widely used in named entity recognition. Given an observed sequence of tokens, x = x 1 x 2 ⋯ x n , a CRF predicts a corresponding sequence of labels, y* = y 1 y 2 ⋯ y n . y* maximizes the conditional probability P(y|x) for all y in the set of possible label sequences [13].
De-identification aims to automatically identify the boundaries of PHI entities and assign the PHI categories to them. The existing open source toolkit CRF++ 3 is utilized to classify the tokens in a sequence into the BIO scheme. "B" indicates a token is the beginning of a PHI entity, whereas "I" shows that a token is inside of a PHI entity, and "O" means that a token does not belong to any category of the PHI entities. In this classifier, the data files for training and testing using CRF++ are obtained in two steps. The first step is the preprocessing of medical records, which is described in Section 2.1. Then, the training and testing data files are created based on the features shown in Section 2.2. A CRFs model considering all of the categories can be learned after training on the training data file, and the tokens in the testing data file can be classified into one of these categories using the model.

Evaluation
De-identification can be evaluated at either the entity level or token level [9]. Entity level evaluation contains two metrics: the "strict" metric measures whether the position and the type of a PHI result exactly matches the gold standard; the "relaxed" metric loosens the position constraint, allowing the end position of a PHI result to be off by two characters. The token level metric's constraint is less than the entity-level metrics. If a PHI entity in the gold standard is a phrase comprising several tokens and the system identifies these tokens as separate entities of this PHI's type, the system output is considered correct. For example, "Newton Hospital" is a single entity in the gold standard, but the system annotated "Newton" and "Hospital" as two separate entities. The token level metric treats this system annotation as correct when type attributes match. There are two sets of PHI categories: the i2b2 PHI categories, and the PHI categories that are defined by the Health Insurance Portability and Accountability Act (HIPAA). The i2b2 PHI categories contain some PHI types that are not included in HIPAA. The performance of the system is evaluated by calculating the precision, recall and F-measure of the metrics, which are composed of the entity level and token level metrics of the two category sets. The formulas of the precision, recall, and F-measure are shown in Eqs. (1)-(3), and they are evaluated at a micro level and at a macro level (micro means all of the PHI tags in the dataset are evaluated together; macro indicates that all of the PHI tags in each medical record are evaluated, then all the evaluations of the medical records in the dataset are averaged). The micro F-measure of i2b2 strict metric is chosen to rank in this task. (1)

System Results
Using the evaluation metrics described above, the best official submission and two unofficial development runs are shown in Table 3. The best official run of the WI-deId system ranked fourth in the de-identification task of i2b2 2014, and the best unofficial development run (unofficial run2) would have ranked second. The details of the best unofficial system are described in Section 2. The following analysis in Sections 3.3 and 3.4 is also based on the best unofficial system.
Compared with the best unofficial system, the best official system has some differences in the preprocessing module and feature generation module. The best official system preprocessed the medical records without using pattern-matching techniques and generated only a part of the features listed in Section 2.2. The features generated in the best official system included current token, POS tag of current token, two previous tokens, POS tag of two previous tokens, two next tokens, POS tag of two next tokens, length of current token, whether the current token contains a digit, whether the current token contains a letter, whether the current token begins with a capital letter and whether the current token contains a punctuation character. Different from unofficial system2, unofficial system1 did not use the following features: the orthographic and dictionary features of the two previous tokens and the orthographic and dictionary features of the next two tokens.
As shown in Table 3, it is clear that the evaluation results of the HIPAA PHI categories are better than the results of the i2b2 PHI categories and that the token level outcomes exceed the corresponding entity level outcomes. Table 4 lists the details of each PHI type in the best unofficial system output at the i2b2 strict entity evaluation level. In the seven main categories, NAME, DATE, AGE, CONTACT, and IDs received high F-measures of above 0.9, but PROFESSION had a quite low F-measure of 0.5929. Compared with the other main categories, the numbers of PHI terms of PROFESSION and CONTACT in the training dataset are obviously less, particularly PROFESSION. The lack of training instances is the fundamental reason for the poor performance in the PROFESSION category, and it also causes the low F-measures of some sub-categories such as ORGANIZATION, COUNTRY, LOCATION-OTHER, FAX, EMAIL and DEVICE. As is shown in Table 4, PROFESSION, HOSPITAL, STATE, COUNTRY, and LOCATION-OTHER achieved F-measures below 0.9; these categories do not belong to the HIPAA PHI categories. Consequently, the overall result of the i2b2 PHI categories is worse than that of the HIPAA PHI categories. In addition to the weak identification results for some PHI types, the WI-deId system also had good performance in several PHI types. For example, the F-measures of USERNAME, ZIP, DATE, and MEDICALRECORD exceeded 0.97.

Comparison
Two groups of experiments were designed to explain the effectiveness of the preprocessing module and the feature sets used in the WI-deId system. In the first group, as a comparison system, the preprocessing module was removed from the WI-deId system. The second group was for analyzing the contribution of the three feature sets to the system performance by adding the feature sets in a greedy way.
The experimental result of the first group is shown in Table 5. Compared with WI-deId, the performance of the system without the preprocessing module dropped significantly. All of the F-measures of the comparison system were lower than WI-deId, particularly AGE, whose F-measure decreased more than 20 percent. This experiment shows the importance of the preprocessing module in the system. Figure 2 shows F-measures of the 7 main categories using three different feature sets (Lex, using lexical features; Lex+Ort, using lexical features and orthographic features; and Lex +Ort+Dic, using lexical features, orthographic features, and dictionary features). The major contribution of orthographic features lies in the performance of LOCATION, CONTACT, and IDs, which increased more than 5, 8, and 4 percent in the F-measure, respectively. Dictionary features focuses the contribution to the system result on LOCATION, which accords with the definition of those dictionaries. The curve shown in Figure 3 describes the improvement of the overall system performance by adding the feature sets.

Error Analysis
The errors in the WI-deId system were analyzed according to the error analysis method in [2]. In this method, errors were divided into four groups: type errors (entity is correct but type is wrong), extent errors (entity in system output has an additional or a missing part), missing errors (entity is in the gold standard but not in the system output) and spurious errors (entity is in the system output but not in the gold standard). Table 6 lists the error distribution of the WI-deId system at the i2b2 strict entity evaluation level according to the four groups of errors. Missing errors constitute the highest proportion, 64.42%, causing the recall of the WI-deId system to be relatively lower than the precision. Furthermore, missing errors will expose the protected health information of the patients; thus, these are more serious than other errors. Table 7 shows the details of errors in groups by type errors, missing errors and spurious errors. The maximum number of type errors was produced by PATIENT and DOCTOR. Forty-four PATIENT entities were identified as DOCTOR, whereas 14 DOCTOR entities were marked as PATIENT. It is most difficult to distinguish between PATIENT and DOCTOR because they not only are similar in spelling but also belong to the same main category, NAME. Missing PHI entities account for 7.4% of the whole gold standard PHI entities (the number of missing PHI entities is 851, and the number of gold standard PHI entities is 11462). Moreover, PROFESSION, ORGANIZATION, LOCATION-OTHER, FAX, EMAIL and DEVICE have a missing percentage (calculated by dividing the missing number by the total number in each row in Table 7) of above 50%. COUNTRY also has a relatively high missing percentage. These are largely caused by the lack of training instances; for example, "Dutch" and "Finland" are PHI terms of COUNTRY in the gold standard but do not occur in training data as PHI. All of these characteristics lead to the low recall rate of the WI-deId system. Compared with missing errors, the proportions of spurious errors generated in the PHI categories are much lower. The highest occurrence rate of spurious errors, up to 9.6%, appears on PROFESSION. Table 8 shows the extent errors that occurred in system output. In this table, "Short" indicates a PHI term produced by a system that is a part of the corresponding PHI term in the gold standard. However, "Long" shows that a PHI term produced by the system exceeds the span of the corresponding PHI term in the gold standard, and "S&L" means that a PHI term produced by the system does not capture the entire text of the corresponding PHI term in the gold standard but spans more text. Almost one-third of the extent errors occurred on DATE, with 42 short errors and 20 long errors. For example, "5 August 2060" is a PHI term of DATE in the gold standard but the WI-deId system identified "August 2060" as DATE. This difference leads to a short error. Similarly, the system tagged "November 1 morning" as DATE, but only "November" is a PHI term in the gold standard. This difference generated a long error. Among the three groups of extent errors, "S&L" errors occurred the least, only 10 times, but "Short" errors accounted for more than half, up to 114 times. Similar to missing errors, "Short" errors will disclose a portion of protected health information, which is harmful to the patients.
Because tokenization is a key processing procedure in the WI-deId system, it also caused some errors in the system output. For example, "Sergio-Steven" is a PHI entity of PATIENT in the gold standard; it was tokenized into "Sergio -Steven" after preprocessing. Ultimately, the system identified "Sergio" as a PHI entity of PATIENT.

Conclusion
This work presents a system that is completely machine-learning-based; it uses neither a rule-based method nor a post-processing module. In this de-identification task, we achieved a micro F-measure of 0.8852 at the i2b2 strict entity evaluation level, which ranked fourth. After this task, the F-measure was promoted to 0.9232, which would have ranked second in the challenge rankings, and the content of this paper is the description of the improved system. The preprocessing of the corpus plays an extremely crucial role in the process of deidentification. The features chosen further enhance the performance of the WI-deId system. We did not use a sentence boundary detector or a tokenizer trained on clinical text; therefore, these domain-restricted tools can be used to update the preprocessing module in our system. Because the text in the medical record has a fixed width, a large proportion of the sentences are cut off, which destroys the integrity of the sentences and decreases the precision of the POS tagger. Future work should attempt to recover the integrity of these sentences.   Overall performance using different feature sets at the i2b2 strict entity evaluation level  Table 2 Features Used in the CRFs Classifier   Table 4 Evaluations of Each PHI Type at the i2b2 Strict Entity Evaluation Level    Table 7 Type Errors, Missing Errors, and Spurious Errors in the WI-deId System at the i2b2 Strict Entity Evaluation Level The first two letters are used to represent all PHI types except for STREET and STATE, for which the first three letters are used. Type errors are bed, and missing errors and spurious errors are in i text. (The sub-categories whose count is zero in the gold standard dataset are not listed in this   The first two letters are used to represent all PHI types except for STREET and STATE, for which the first three letters are used. (The sub-categories whose count is zero in the extent errors of system output are not listed in this table.)