{'fraction_non_alphanumeric': 0.020841227737779463, 'fraction_numerical': 0.005683971201212581, 'mean_word_length': 5.0411899313501145, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Fault Isolation : Dynamically Re-arranging Data to Increase Availability

As traditional media is progressively replaced by digital files stored in the cloud, our cultural mindset is shifting from considering digital data to be ephemeral to it being the primary, and often only, store of vital information. We are also seeing the first generations that record their entire lives and expect this data to be digitally stored perpetually, inexpensively, and searchably, creating massive data stores with both primary and archival usage patterns [2], [5]. We define the workload that this sort of data creates as “archival-by-accident.” In an archival-by-accident system, there are dynamic pockets of primary use in a storage system that otherwise exhibits traditional secondary or even tertiary workload characteristics. To handle these primary pockets on large systems over a long period of time, we must create a system with low cost, high reliability, and high availability. This availability requirement is the key addition our work imposes on traditional archival systems. The goal of this project is to exploit patterns in large data to co-locate working sets and thus improve system availability. Improving availability is increasingly important as storage systems blossom in size and expected time of data retention. A 100PB system that has no failures 99.99999% reliability against data loss will still lose ove r 100 GB on average every hundred years and have to rebuild many terabytes more. Depending on the reliability scheme, certain failures can cause very long rebuilds that have a large impact on system availability. Rebuild time is also a function of the size of the components of the system. As these components increase in size, the availability loss caused by rebuilding data increases accordingly. Systems typically place data evenly across disks in order to distribute and thus dilute the impact of failure events. This data placement assumes that all data has identical probability of being accessed. While this assumption suffices for traditional archival systems, archival-by-accident systems have shifting subsets of data in active, primary use. For example, suppose a storage system contains the equal size inputs for ten experiments striped evenly across five disks. If one disk fails, all ten projects are stalled by the rebuild since they each have a small percentage of missing data. If, on the other hand, the data was arranged such that each experiment had the files it actively needed colocated, only two experiments would be stalled during rebuild while the other eight would only suffer reduced bandwidth.