{'fraction_non_alphanumeric': 0.026239067055393587, 'fraction_numerical': 0.004859086491739553, 'mean_word_length': 4.404199475065616, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A File System Utilization Metric for I / O Characterization

A high performance computing (HPC) platform today typically contains a scratch high-performance parallel file system for data storage. Today, such file systems encompass 10-20% of the purchase price of a HPC resource. Looking forward, it is apparent that the rate of increase of hard drive performance will not keep up with the expected gains in processing, and therefore any effort to keep I/O performance at the same relative level will require a larger and larger fraction of the overall budget. Therefore, it will become increasingly important to understand the I/O usage and needs of HPC workloads in great detail in order to ensure resources are adequately provisioned. Although it is relatively straightforward today to measure the total amount of I/O to and from a file system, and the bandwidths achieved, these metrics reveal only part of the overall file system load, because the size of individual I/O requests and their relation to one another can strongly affect how much data a file system can move in a given interval. In this work we introduce a new metric for file system utilization (FSU) that accounts for both the volume of data moved and the number of disk I/O operations required to move that data. We present a description of our model in which we idealize an I/O transaction to disk (on the server) as requiring time t that includes a small, fixed amount of time (the latency, a0) and an amount of time proportional to the size of the I/O: t = a0 + a1 ∗ b. A series of calibration experiments using transactions of various sizes allows us to establish a0 and a1, which we do separately for reads and writes. We collect data on the number and size, N(b), of all I/O transactions, which allows us to calculate the File System Utilization (FSU). The FSU metric provides a view of the I/O workload on the file system. We present early results showing that the Hopper Cray XE6 /scratch file system is about four times as busy as we would estimate from a bandwidth metric alone.