{'fraction_non_alphanumeric': 0.014167650531286895, 'fraction_numerical': 0.0059031877213695395, 'mean_word_length': 4.848275862068966, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Compressing Large Amounts of NetFlow Data Using a Pattern Classification Scheme

The storage of large amounts of network data is a challenging problem, in particular if it still needs to be actively consulted as for example in the case of network forensics. Here we propose a method to compress NetFlow data while simultaneously adding domain knowledge. Our method is based on a pattern classification scheme by considering all flows from a single source IP address simultaneously. Each pattern can be described by at most 19 attributes that give a good statistical description of the original NetFlow data, while minimising information loss. We estimate that on average a factor of about 300 in storage space can be gained. The process is explained using a real world dataset from a large, high-speed, network, and a formal rationale is provided.