{'fraction_non_alphanumeric': 0.017064846416382253, 'fraction_numerical': 0.0, 'mean_word_length': 5.825581395348837, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
What to classify and how: Experiments in question classification for Japanese

This paper describes experiments in Japanese question classification, comparing methods based on pattern matching and machine learning. Classification is attempted over named entity taxonomies of various sizes and shapes. Results show that the machine learning based method achieves much better accuracy than pattern matching, even with a relatively small amount of training data. Larger taxonomies lead to lower overall accuracy, but, interestingly, result in higher classification accuracy on key classes.