{'fraction_non_alphanumeric': 0.024533138044672283, 'fraction_numerical': 0.004393994873672647, 'mean_word_length': 5.413145539906103, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Generating Coherent Paragraphs

Recent years have seen something of a divorce between models of computer-aided instruction and theories of second language learning. For example, it has come to be recognized that a satisfactory model of language teaching must incorporate the notion of 'communicative compe-tence', that is, not only the production of grammatically correct utterances, but also the appropriate use of utterances according to situations. At the same time, it has been recognized that linguistic competence goes beyond sentence-length utterances to include textual relations such as anaphora, inference, logical connectives and the like. Finally , it has been suggested that decontextualized utterances with little or no reference to reality do not necessarily serve the interests of language learners; rather, they should be exposed to 'au-thentic' texts which contain information about the cultural context of the language being taught. Such limitations affect not only instructional systems, but also subject testing environments aimed at capturing in finer detail the nuances of linguistic performance. One solution to this problem lies in the construction of finely crafted environments in which complex linguistic, discourse and pragmatic relations may be represented. On a basic level, there exists a broad range of authoring systems (Calis, from Duke University, to name but one) which allow the manual entry and tagging of authentic texts to produce comprehension exercises. At a more sophisticated level, one finds the use of programmed 'microworlds' which may be navigated by learners. For example, Hamburger (1994) proposes a language learning environment based on a 'Kit-chen World' in which learners may manipulate objects (turning on a faucet, for example), read descriptions and produce utterances ordering that actions be performed. The development of more 'authentic' examples of the sort will depend crucially on progress in dealing with textual inference and incorporating encyclopedic knowledge. Work of the sort has been done. For example, the BORIS program (Lehnert 1983) is capable of understanding a complex narrative concerning divorce. On a broader scale, the CYC project (Lenat 1990) involves the attempt to capture a broad range of encyclopedic information and related inferences. Despite their promise, all of these approaches have a number of weaknesses. BORIS is capable of understanding relatively complex texts dealing with divorce and shows an impressive ability to extract implicit information from such texts. However , the underpinnings of this information are buried in the complex code of the program, with the result that extensions to other domains are difficult. Similarly, …