{'fraction_non_alphanumeric': 0.021043771043771045, 'fraction_numerical': 0.0025252525252525255, 'mean_word_length': 5.679775280898877, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A QUANTITATIVE ANALYSIS OF STUDY ABROAD PROCESS EVALUATION

This paper reports the reliability of training data for developing an automatic evaluation method that measures the probability of success in study abroad programs. Although the previous evaluation method demonstrated acceptable accuracy, i.e., moderate correlation between the measured and observed data, the reliability of the training data remains unaddressed. The training data consist of 334 students' answers to ten multiple-choice questions and six open-ended questions about the study abroad experience. The reliability was examined using Cronbach α coefficient for the internal consistency that referred to whether students' answers demonstrate similar results for questions. The examination showed contradictory results between data types: the reliability was empirically satisfactory for the length of open-ended answers, but unsatisfactory for the other types of data such as answers to multiple-choice questions and word-choice in open-ended questions. The results suggest the need for minimizing the influence from students’ subjective judgment in multiple-choice questions and word-choice in open-ended questions.