{'fraction_non_alphanumeric': 0.021813224267211998, 'fraction_numerical': 0.0, 'mean_word_length': 4.825396825396825, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A better edit distance measure allowing for block swaps

Edit Distance, also known as the Levenshtein distance or evolutionary distance, is a concept from information retrieval, and it describes the number of edit operations that have to be made in order to change one string to another. It's one of the most common measures to expose the dissimilarity between two strings. Ordinarily, Edit Distance is based on a character insert, delete and substitution operations. By using these three operators Edit Distance can help us to solve the problem of computing the similarity between two sequences that arise in many areas. However, standard Edit Distance still seems to miss the true relationship between these two similar strings in some cases because of the sequential order of common sub strings. For example, the Edit Distance between "classbook" and "bookclass" is eight, because of the words "book" and "class" is reversed but intuitively the two strings seem much closer. In order to solve this problem, we propose a method for extended Edit Distance, which permits block swap operation. The main contribution in this paper is the method to compute the cut points over a single string, and then allowing block swaps, which move sub strings from one position to another in a string, in order to make common substrings in the right order. Through our experiment, it is revealed that Block Swap Edit Distance can help us to find a better measure for Edit Distance.