{'fraction_non_alphanumeric': 0.021904761904761906, 'fraction_numerical': 0.0038095238095238095, 'mean_word_length': 5.146198830409356, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Effect of Dirty Data on Free Text Discharge Diagnoses used for Automated ICD-9-CM Coding

We discuss data quality issues that emerge when applying text mining classification methods for automated ICD-9-CM coding. In particular our work investigates the extent to which errors in input text data propagate to the classification model. Text classification techniques based on two Bayesian machine learning algorithms (naive Bayes and shrinkage) were applied to a set of free-text outcome diagnoses, which were previously coded using the Spanish Edition of the International Classification of Diseases – Clinical Modification (ICD-9-CM). A measure of predictive accuracy was calculated for each of the text mining algorithms under analysis. Subsequently, the quality of the sample data was incrementally deteriorated by simulating typographical errors in the text. The predictive accuracy was recomputed for each of the dirty samples for comparison purposes. Our results suggest a low impact of errors on the performance of automatic coding by ICD-9-CM.