{'fraction_non_alphanumeric': 0.027417027417027416, 'fraction_numerical': 0.011544011544011544, 'mean_word_length': 5.304545454545455, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Multifunctional Full-Packet Capture and Network Measurement System Supporting Nanosecond Timestamp and Real-Time Analysis

In this article, we design and implement a full-packet capture and network measurement (FPC-NM) system in a 20 Gbit/s network. The system has multiple functions, such as packet receiving, nanosecond timestamp, load balancing, data packet preprocessing, application layer protocol analysis, data packet storage, and log management. We design a timestamp module in field programmable gate array (FPGA) that can add a timestamp to each captured packet, and the retrospective packet analysis can be done with 4 ns accuracy. As far as we know, this is the first time that nanosecond timestamp has been applied to FPC-NM system, which greatly improves the accuracy of security incident retrospective analysis. The experimental results show that the FPC-NM system can achieve 17 Gbit/s throughput and 160 000 connections per second under the premise of zero packet loss. Using LZ4 compression, the performance of our system for real-time compression and storage of traffic can achieve 10 Gbit/s. The system performance is improved by about 40% after software optimization including direct memory access buffer size adjustment and single instruction, multiple data (SIMD) instruction set, which validates the practicability and effectiveness of our optimization method.