{'fraction_non_alphanumeric': 0.014022140221402213, 'fraction_numerical': 0.0014760147601476014, 'mean_word_length': 4.973568281938326, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Data Reliability and Data Selection in Real Time High Fidelity Sensor Networks

Due to advances in technology, sensors in resource constrained wireless sensor networks are now capable of continuously monitoring and collecting high fidelity data. However, all the data cannot be trusted since data can be corrupted due to several reasons such as unreliable, faulty wireless sensors or harsh ambient conditions. Further, due to bandwidth constraints that limit the amount of data being transmitted in sensor networks, it is important that only the high priority, accurate data is transmitted. In this paper, we propose a data selection model that makes two significant contributions. First, it provides a way to determine the confidence in or reliability of the data values and second, it determines which subset of data is of the highest quality or of most interest given the state of the network system and its current available bandwidth. Our model is comprised of two phases. In Phase I we determine the reliability of each input data stream using a Bayesian network. In Phase II, we use a 0-1 Knapsack optimization approach to choose the optimal subset of data. An evaluation of our best data selection model reveals that it eliminates erroneous data and accurately determines the subset of data with the highest quality when compared with conventional