{'fraction_non_alphanumeric': 0.026016615653694796, 'fraction_numerical': 0.011587232181897682, 'mean_word_length': 5.023041474654378, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Inter-Rater Reliability of Unstructured Text Labeling: Artificially vs. Naturally Intelligent Approaches.

Unstructured medical text labeling technologies are expected to be highly demanded since the interest in artificial intelligence and natural language processing arises in the medical domain. Our study aimed to assess the agreement between experts who judged on the fact of pulmonary embolism (PE) in neurosurgical cases retrospectively based on electronic health records and assess the utility of the machine learning approach to automate this process. We observed a moderate agreement between 3 independent raters on PE detection (Light's kappa = 0.568, p = 0). Labeling sentences with the method we proposed earlier might improve the machine learning results (accuracy = 0.97, ROC AUC = 0.98) even in those cases that could not be agreed between 3 independent raters. Medical text labeling techniques might be more efficient when strict rules and semi-automated approaches are implemented. Machine learning might be a good option for unstructured text labeling when the reliability of textual data is properly addressed. This project was supported by the RFBR grant 18-29-22085.


Introduction
Unstructured text labeling is an essential and the most time-consuming step when preparing for its classification with machine learning in any professional area. We expect data labeling technologies that address unstructured medical texts to be highly demanded as the interest in artificial intelligence and natural language processing arises in the medical domain.
Medical records may provide a lot of valuable information within the free text. However, its evaluation might depend on the expert's individual experience and, possibly, a subjective perspective. Reliable information extraction or data labeling requires unambiguous professional interpretation of medical content by experts. Our study aimed to assess the agreement between experts who judged on the fact of pulmonary embolism in neurosurgical cases retrospectively based on electronic health records and assess the utility of the machine learning approach to automate this process.

Methods
The dataset for the study was obtained from electronic health records (EHR) of N.N. Burdenko Neurosurgery Center limited by a period between 2000 and 2017 (90,688 cases treated). We queried the medical cases where the key string patterns in Russian referred to as «pulmonary embolism» (PE) were met in narrative text. As a result, 621 cases were identified in which the description of PE was anticipated.
The dataset consisted of clinical notes, doctors' reports, examinations, postmortem findings, and other clinical records sections typed on a keyboard with a free text in the period following the neurosurgical procedure. All the records were independently screened by 3 similarly skilled neurosurgeons. Expert selection criteria were the training in neurosurgical residency accomplished at N.N. Burdenko Neurosurgery Center and the working experience of 7-11 years. The coders had small and equal prior experience labeling data from approximately 2000 medical cases. However, they were not specially trained for PE detection task. The texts were presented to the experts in a special software designed to focus only on medical texts instead of the traditional EHR user interface. Each expert was asked to label the cases with either "PE detected" or "No PE detected" or "the fact of PE could not be well-verified." The latest cases were excluded from the current analysis.
The inter-rater reliability was assessed using Light's kappa coefficient for 3 coders and Cohen's kappa for 2 raters [1]. The evaluation of agreement level was made in a fully-crossed design [2]. The part of the dataset in which all the estimations were fully agreed upon between the independent coders was used to test a logistic regression model as a labeling automation method. The rest of the dataset which was not equally labeled by all three and independent raters was screened by the first author of this article with the methodology described in our previous publications [3]- [5]. The text was tokenized into sentences and then into words lemmatized with MyStem technology [6]. The unique lemmas were screened to select those likely related to or certainly used in the PE description. A corresponding set of sentences containing the initial words matching all lemmas selected was then reviewed to label it with "PE detected" or "No PE detected" or "the fact of PE could not be well-verified" marks. The latest were excluded from the analysis. It is important that a strict rule was implemented in this process: the "PE detected" label was applied only if the information in the sentence was stated firmly. "No PE detected" was used when no relevance of PE was discussed in a sentence. "the fact of PE could not be well-verified" was set in all cases, in which the uncertainty of the PE statement was contained in the manner of writing (e.g., accompanied by "probably", "can't be excluded", "PE?" etc.). That labeled sentence set (without uncertain cases) was used to train and test the logistic regression model to automate sentence labeling.
Both logistic regression models were trained with LASSO regularization in a loop of 300 resamples [7]. The sensitivity (SENS, also referred to as recall), specificity (SPEC), accuracy (ACC), positive predictive value (PPV, also referred to as precision), negative predicted value (NPV), F-measure (F), cutoff value (CO) and area under ROCcurve (AUC) were computed for every iteration. The bounds of 95% AUC confidence interval were calculated using bootstrapping. The cutoff point was chosen to maximize the sum of sensitivity and specificity for every algorithm. The final metrics were averaged across 300 iterations.
The experts agreed with each other fully and independently in 420 cases. The efficiency of the logistic regression model trained with regularization on that dataset (after 18 uncertain cases exclusion) is shown in Table 1. That model exploited the set of a few texts for one clinical case matched with one label (PE confirmed in 78 cases and firmly not detected in 324).
There was no equal agreement between 3 experts observed in 201 cases. Excluding 70 cases of uncertainty, splitting the text into sentences and labeling them with the method we proposed earlier led us to a dataset of 1565 sentences, with 152 indicating PE and 1413 pointing out its absence. The logistic regression trained and tested on distinct labeled sentences (as opposed to set of texts) showed much better performance in terms of all metrics except for sensitivity (see Table 1).  Figure 1 demonstrates the superimposed ROC curves in the experiments with sets of texts equally coded by experts and with sets of sentences labeled using a special technique by additional independent expert for the cases of disagreement.

Discussion
PE is one of the most dangerous life-threatening complications in neurosurgery yet rare and possibly preventable [8]. PE research in neurosurgery requires a significant number of cases accumulated for a long period of time. Retrospective analysis of PE might be valuable but is obviously limited by the quality and completeness of medical records (non-modifiable factor) and by the qualification of experts extracting information. The results of our work demonstrate a moderate agreement (incomplete in 32,4% of cases) between 3 experts with a similar level of professional knowledge and experience. This can be partially explained by the interpretation complexity of certain cases due to a lack of information. In a number of situations, an expert can draw a conclusion based on indirect signs of PE which is discussible. It is important that no expert had been instructed on methods of information evaluation prior to the study. Our observations stress the human factor as it might contribute to the result of machine learning. We have found that strict rules implemented specifically for a certain labeling task can make the process more efficient. Thus, special pretraining might be considered for the experts. The labeling technique we proposed earlier proved its efficiency with a traditional machine learning approach. In that case the whole sentences were labeled (instead of n-grams) which is one of possible modifications of our algorithm.
The limitations of our study are the relatively small sample sizes, different algorithms applied to different samples, sentence labeling by one expert, one type of model tested in machine learning. The state-of-art approaches (e.g., BERT) might be beneficial in future experiments. We also left the cases of uncertainty in experts' conclusions beside the scope of this article. We believe that these cases deserve a thorough consideration in a special research.

Conclusion
Medical text labeling techniques might be more efficient when strict rules and semiautomated approaches are implemented. Machine learning might be a good option for unstructured text labeling when the reliability of textual data is properly addressed. This project was supported by the RFBR grant 18-29-22085.