{'fraction_non_alphanumeric': 0.02562862669245648, 'fraction_numerical': 0.0033849129593810446, 'mean_word_length': 4.979768786127168, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
An Efficient Approach for VIIRS RDR to SDR Data Processing

The Visible Infrared Imaging Radiometer Suite (VIIRS) Raw Data Records (or Level-0 data) are processed using the current standard Algorithm Development Library (ADL) to produce Sensor Data Records (SDR; or Level-1B data). The ocean color Environmental Data Records (EDR), one of the most important product sets derived from VIIRS, are processed from the SDR of the visible and near-infrared moderate resolution (M) bands. As the ocean color EDR are highly sensitive to the quality of the SDR, the bands from which the EDR data arise must be accurately calibrated. These bands are calibrated on-orbit using the onboard Solar Diffuser, and the derived calibration coefficients are called F-factors. The F-factors used in the forward operational process may have large uncertainty due to various reasons, and thus, to obtain high-quality ocean color EDR, the SDR needs to be regularly reprocessed with improved F-factors. The SDR reprocessing, however, requires tremendous computational power and storage space, which is about 27 TB for one year of ocean-color-related SDR data. In this letter, we present an efficient and robust method for reduction of the computational demand and storage requirement. The method is developed based on the linear relationship between the SDR radiance/reflectance and the F-factors. With this linear relationship, the new SDR radiance/reflectance can be calculated from the original SDR radiance/reflectance and the ratio of the updated and the original F-factors at approximately 100th or less of the original central processing unit requirement. The produced SDR with this new approach fully agrees with those generated using the ADL package. This new approach can also be implemented to directly update the SDR in the EDR data processing, which eliminates the hassle of a huge data storage requirement as well as that of intensive computational demand. This approach may also be applied to other remote sensors for data reprocessing from raw instrument data to science data.