{'fraction_non_alphanumeric': 0.015281757402101241, 'fraction_numerical': 0.0, 'mean_word_length': 5.238095238095238, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Linear Unit Grammar

People have a natural propensity to understand language text as a succession of smallish chunks, whether they are reading, writing, speaking or listening. Linguists have found that this propensity can shed light on the nature and structure of language, and there are many studies which attempt to harness the potential of natural chunking.This book explores the role of chunking in the description of discourse, especially spoken discourse. It appears that chunking offers a sound but flexible platform on which can be built a descriptive model which is more open and comprehensive than more familiar approaches to structural description. The model remains linear, in that it avoids hierarchies, and it concentrates on the combinatorial patterns of text. The linear approach turns out to have many advantages, bringing together under one descriptive method a wide variety of different styles of speech and writing. It is complementary to established grammars, but it raises pertinent questions about many of their assumptions.