{'fraction_non_alphanumeric': 0.01366867140513942, 'fraction_numerical': 0.0, 'mean_word_length': 5.606498194945848, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Document Structure Analysis for Large Electronic Document Collections

Beside document collections containing a wide variety of different documents, such as the web, collections exist, which are collecting documents of a single or very few different types. For example, the EDGAR database of the SAC is collecting different documents of companies, which have to report regularly on different issues of their business. Similar databases exist in medicine or in industry. Due to the nature of this database, to contain very similar documents, it is often desirable to provide analysis functionalities that are able to automatically gain some insight into the filed information. A major problem of the automatic processing is the lack of structure information in electronic documents. The majority of electronic document formats used for archiving, are based on visual representations for human readers. This makes automatic processing complex, because relevant and irrelevant content cannot be automatically distinguished easily. This thesis addresses this issue and describes and evaluates techniques for logical and functional structure analysis. The presented techniques are based on machine learning. Whereas the analysis of logical structure uses mainly geometric and formatting information, inspects the analysis of functional structures the textual content. The problem to identify and analyze errors in the structure analysis results is solved with visualization. The variable text scaling technique is designed to highlight interesting parts in logical and functional structures. It is also applicable to visualize keyword search results in document viewers. Afterwards several examples using the presented techniques are discussed. The thesis concludes with a summary of the results and discusses open research questions.