{'fraction_non_alphanumeric': 0.02282333051563821, 'fraction_numerical': 0.00422654268808115, 'mean_word_length': 4.97979797979798, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
DETERMINING RECORD LINKAGE PARAMETERS USING AN ITERATIVE LOGISTIC REGRESSION APPROACH

1. Abstract. Fellegi and Sunter (1969) developed an algorithm for linking records. Each pair of records is scored on a field by field basis. If the records agree on a given field, a bonus is added to the score. If they disagree, a penalty is subtracted from the score. The records are designated as a match provided the score exceeds a certain threshold. In order to employ this method, one has to determine which fields to compare, bonuses and penalties for each field, an appropriate threshold for designating matches, and the accuracy of a particular linkage, based on its score. Parameters for many previous record linkage applications have been set using an iterative approach involving manual review and re-calibration of the matching fields, bonuses, penalties, and the threshold. The results of such ad hoc procedures can be difficult to replicate, justify, and interpret. This paper describes a more rigorous methodology for setting these parameters based on an iterative logistic regression approach. The method also facilitates the evaluation of the accuracy of the resulting linkages.