{'fraction_non_alphanumeric': 0.02592912705272256, 'fraction_numerical': 0.0, 'mean_word_length': 4.676470588235294, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Automatic selection of optimal Savitzky-Golay smoothing.

A method to select the optimal window size of the Savitzky-Golay (SG) algorithm is presented. The approach is based on a comparison of the fitting residuals (i.e., the differences between the input signal and the smoothed signal) with the noise of the instrument. The window size that yields an autocorrelation of the residuals closest to the autocorrelation of the noise of the instrument is considered optimal. The method is applied in two steps. In a first step, the lag-one autocorrelation value of the noise of the instrument is computed through the study of a blank signal. In a second step, the SG algorithm is applied to "smooth" the signal using different window sizes. The method was applied to data from NMR, chromatography, and mass spectrometry and was shown to be robust. It finds the optimal window size for different signal features. This allows the method to be used in an unsupervised way, embedded in a more complex algorithm in which smoothing and/or differentiation of signals is required, provided that the lag-one autocorrelation value of the instrument noise does not change.