{'fraction_non_alphanumeric': 0.029055690072639227, 'fraction_numerical': 0.0036319612590799033, 'mean_word_length': 4.783216783216783, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Techniques for extracting statistical data from free-form text using APL

A family of methods has been developed for processing free-form text files using APL. We assume that a file has an arbitrary structure and has regions of statistical and numerical data embedded in it. The data in the file are processed to extract the following information: 1. the embedded statistical and numeric regions, 2. titles, keys (legends), and tic-mark identifiers, associated with each region, and 3. dates and other pertinent information. Once these data are extracted, bar, line, pie charts or a combination of those are plotted. An APL program which is automatically loaded and invoked performs these tasks. The program has been tested on hundreds of files with different structures. Some of these files have multiple statistical regions.