{'fraction_non_alphanumeric': 0.019472361809045227, 'fraction_numerical': 0.002512562814070352, 'mean_word_length': 5.665271966527197, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Large-Scale Intelligent Transportation System Traffic Detector Data Archiving

Archived traffic data can be used in transportation planning, administration, and research by various entities and agencies. During the past two decades, considerable effort has been dedicated to developing and implementing large-scale traffic data archives. The Wisconsin Traffic Operation and Safety (TOPS) Laboratory at the University of Wisconsin–Madison maintains a statewide traffic detector data archiving and retrieving system, which is developed to enable centralized management of statewide ITS detector and configuration data, optimizing the utilization of massive data on a systematic level, and improving the interactivity and accessibility for integration with other transportation data sources such as lane closure data or incident data. This data archive is currently being enhanced to incorporate higher resolution traffic data by migrating from 5-minute to 1-minute and even 20-second sampling intervals. At the same time, there is a desire to generate aggregated datasets such as hourly, monthly, and annual average values from the raw data. As the traffic data requirements continue to grow, the management of the traffic data archive becomes a complex big data problem. This paper describes a proposed redesign of the TOPS Lab traffic detector archived data management system to improve storage, performance, access, and integration capabilities. Particular detail is given to the data archiving process, including data validation, and support for spatial attributes and GIS data integration.