{'fraction_non_alphanumeric': 0.029646522234891677, 'fraction_numerical': 0.0, 'mean_word_length': 5.097222222222222, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A finite state approach to operator classification

We consider the problem of uniquely classifying operators in expressions containing binary, prefix unary, and suffix unary operators e.g. binary + (addition), prefix unary ~ (negation), and suffix unary ! (factorial). Since the same name may be used to represent distinct operators in different classes, the problem is one of refinement. In the general case, it can be recast in a grammatical frameword whose solution requires all parse trees of an ambiguous grammar. We consider a much simpler finite state solution. More specifically, we consider an automaton based algorithm which refines the classifications of each operator to a unique class (when it exists) as a function of the allowed classifications of the neighboring operators. We do not consider the more complicated problem where operand types effect the result.