{'fraction_non_alphanumeric': 0.017067494181536073, 'fraction_numerical': 0.005430566330488751, 'mean_word_length': 5.3861386138613865, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Evaluation of lossless and lossy algorithms for the compression of scientific datasets in netCDF-4 or HDF5 files

Abstract. The increasing volume of scientific datasets requires the
use of compression to reduce data storage and transmission costs, especially
for the oceanographic or meteorological datasets generated by Earth
observation mission ground segments. These data are mostly produced in
netCDF files. Indeed, the netCDF-4/HDF5 file formats are widely used
throughout the global scientific community because of the useful features
they offer. HDF5 in particular offers a dynamically loaded filter plugin so
that users can write compression/decompression filters, for example, and
process the data before reading or writing them to disk. This study
evaluates lossy and lossless compression/decompression methods through
netCDF-4 and HDF5 tools on analytical and real scientific floating-point
datasets. We also introduce the Digit Rounding algorithm, a new relative
error-bounded data reduction method inspired by the Bit Grooming algorithm.
The Digit Rounding algorithm offers a high compression ratio while keeping a
given number of significant digits in the dataset. It achieves a higher
compression ratio than the Bit Grooming algorithm with slightly lower
compression speed.
