{'fraction_non_alphanumeric': 0.023668639053254437, 'fraction_numerical': 0.0022189349112426036, 'mean_word_length': 5.473684210526316, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Hybrid Processing System for Large-Scale Traffic Sensor Data

In recent years, with the further adoption of the Internet of Things and sensor technology, all kinds of intelligent transportation system (ITS) applications based on a wide range of traffic sensor data have had rapid development. Traffic sensor data gathered by large amounts of sensors show some new features, such as massiveness, continuity, streaming, and spatio-temporality. ITS applications utilizing traffic sensor data can be divided into three main types: 1) offline processing of historical data; 2) online processing of streaming data; and 3) hybrid processing of both. Current research tends to solve these problems in separate solutions, such as stream computing and batch processing. In this paper, we propose a hybrid processing approach and present corresponding system implementation for both streaming and historical traffic sensor data, which combines spatio-temporal data partitioning, pipelined parallel processing, and stream computing techniques to support hybrid processing of traffic sensor data in real-time. Three types of real-world applications are explained in detail to show the usability and generality of our approach and system. Our experiments show that the system can achieve better performance than a popular open-source streaming system called Storm.