{'fraction_non_alphanumeric': 0.011216566005176877, 'fraction_numerical': 0.0, 'mean_word_length': 5.27027027027027, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Optimization of Regular Expression Evaluation within the Manatee Corpus Management System

This paper is concerned with searching large text corpora –
electronic collections of texts. Often these are subject to
queries specified by means of regular expressions. Such queries
go beyond a simple keyword search that can be quickly evaluated
using an inverted index, usually they are rather processed by
third-party regular expression libraries and take significantly
more time to evaluate. In this paper we present an index-based
approach for optimization of regular expression evaluation that
we call n-gram prefetching. It is based on the assumption that
most regular expression queries on text corpora contain at
least some fixed string portions representing clues that can be
used for developing heuristics that would prune the number of
potentially matching strings. The presented work has been
designed and implemented within the Manatee corpus management
system. We show that the proposed approach can significantly
speed up regular expression processing by providing evaluation
on a test set of queries executed on a number of billion-word
text corpora.