{'fraction_non_alphanumeric': 0.01810436634717785, 'fraction_numerical': 0.0, 'mean_word_length': 5.666666666666667, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Evaluation of Interest and Coherence in Machine Generated Stories

Evaluation of the narrative text generated by machines has traditionally been a challenge, particularly when attempting to evaluate subjective elements such as interest or believability. Recent improvements in narrative machine text generation have been largely driven by the emergence of transformer-based language models, trained on massive quantities of data. In this study, a corpus of stories is generated using the pre-trained GPT-Neo transformer model, with human-written prompts. The stories generated through this process are subsequently evaluated through both human evaluation and two automated metrics: BERTScore and BERT Next-Sentence-Prediction. The results show variation in human evaluation results in comparison to automated metrics, suggesting further work is required to train automated metrics to identify text that is defined as interesting by humans.