{'fraction_non_alphanumeric': 0.02929532858273951, 'fraction_numerical': 0.0, 'mean_word_length': 5.226600985221675, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Multivariate Calibration and Chemometrics for near Infrared Spectroscopy: Which Method?

The four most important regression methods are evaluated on very large data sets: Multiple Linear Regression (MLR), Partial Least Squares (PLS), Artificial Neural Network (ANN) and a new concept called “LOCAL” (PLS with selection of a calibration sample subset of the closest neighbours for each sample to predict). The Standard Errors of Prediction (SEPs) are statistically tested and the results show that the regression methods are almost equal and that the data matrices are more important than the fitting methods themselves. The types of pre-treatments (Multiplicative Scatter Correction, Detrend, Standard Normal Variate, derivative etc.) of the spectra are too numerous to be able to test all the combinations. For each test, the pre-treatment found as the best with the PLS method is fixed for the other ones. The second part of the paper emphasises the importance of the number of samples. If any agricultural commodity, and probably any kind of product measured by an NIR instrument, can be considered as a mixture of several constituents, the databases built by collecting actual samples bringing new information can reach hundreds, if not thousands, of samples.