{'fraction_non_alphanumeric': 0.012486992715920915, 'fraction_numerical': 0.0, 'mean_word_length': 5.287581699346405, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
CI-based NLU methodology with applications in information content control

An attempt is made to propose a natural language understanding methodology based on comprehensive information (CI), the latter of which is an integrated concept of syntactic, semantic and pragmatic information. The major feature of the methodology is the full use of CI for sentence understanding in which the syntactic information is used for analyzing the sentence on whether it is grammatically legal and the semantic information is used for analyzing the sentence on whether it is logically true while the pragmatic information is for analyzing the sentence on whether it is useful with respect to the user's goal. The other important feature of the methodology is the possibility to properly combine both the rule-based approach and the statistics-based approach. The successes of a series research projects completed in our research center clearly shows the methodology promising.