{'fraction_non_alphanumeric': 0.020673813169984685, 'fraction_numerical': 0.002297090352220521, 'mean_word_length': 5.050925925925926, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
LEFT TO RIGHT PARSING OF LEXICALIZED TREE‐ADJOINING GRAMMARS 1

This paper presents an algorithm (a parser) for analyzing sentences according to grammatical constraints expressed in the framework of lexicalized tree‐adjoining grammar. For the current grammars of English, the algorithm behaves much better and requires much less time than its worst‐case complexity. The main objective of this work is to design a practical parser whose average‐case complexity is much superior to its worst case. Most of the previous methods always required the worst‐case complexity. The algorithm can be used in two modes. As a recognizer it outputs whether the input sentence is grammatically correct or not. As a parser it outputs a detailed analysis of the grammatically correct sentences. As sentences are read from left to right, information about possible continuations of the sentence is computed. In this sense, the algorithm is called a predictive left to right parser. This feature reduces the average time required to process a given sentence. In the worst case, the parser requires an amount of time proportional to G2n6 for a sentence of n words and for a lexicalized tree‐adjoining grammar of size G. The worst‐case complexity is only reached with pathological (not naturally occurring) grammars and inputs.