{'fraction_non_alphanumeric': 0.019148936170212766, 'fraction_numerical': 0.0, 'mean_word_length': 5.055793991416309, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Design of Direct Read from Sparse Segments in MPI-IO

As computational power of a supercomputer in-creases, there is an increase in the gap between the computational performance and storage performance. This is because of the slow improvement rate of storage performance. A promising remedy to bridge this gap is the node-local storages seen in recent supercomputers.We had previously proposed sparse segments for scalable write performance. This enabled the awareness of locality and avoided lock contention and performance degradation for single shared file access by converting single shared file (SSF) access to file per process (FPP) access internally within the MPI-IO library. It exhibited linear I/O performance by increasing the number of compute nodes. However, the purpose of this method is to improve write performance. To read a file in a sparse segment format, it is necessary to flush all segments back to a parallel filesystem, which may cause performance degradation.A direct read method for sparse segments is proposed to overcome this setback. We introduce an offset table and provide a scalable lookup functionality against multiple sparse segments. We implement our method in the MPI-IO layer to apply existing MPI-IO applications without any code modification. The experimental results of using IOR on Cygnus supercomputer exhibit a scalable performance for both read and write functions.