{'fraction_non_alphanumeric': 0.026511646618658728, 'fraction_numerical': 0.017500100575290663, 'mean_word_length': 4.494694960212201, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Deep learning for Chemometric and non-translational data

We propose a novel method to train deep convolutional neural networks which learn from multiple data sets of varying input sizes through weight sharing. This is an advantage in chemometrics where individual measurements represent exact chemical compounds and thus signals cannot be translated or resized without disturbing their interpretation. Our approach show superior performance compared to transfer learning when a medium sized and a small data set are trained together. While we observe a small improvement compared to individual training when two medium sized data sets are trained together, in particular through a reduction in the variance.


Introduction
Spectral data consist of spectroscopic measurements which contain chemical information about the composition of the sample. Spectral data are in large based on underlying continuous processes, much as is the case for image data, and thus we expect deep learning to work well for applications on spectral data such as e.g. Near Infra Red (NIR) data. Deep learning has thus also successfully been applied to spectroscopic data within a variety of fields. E.g. Risum and Bro (2019) used a deep convolutional neural network (CNN) to detect different types of peaks in gaschromatography. Liu et al. (2017) used a deep CNN to classify different chemical substances based on Raman spectra. Transfer learning for spectroscopic data has been attempted in specialized cases. Liu et al. (2018) applied transfer learning to hyperspectral data of soil. They used data from spectral libraries (the data were acquired under laboratory conditions) to pre-train a model, which was then transferred to field data. Padarian et al. (2019) used transfer learning to convert a global soil clay model to a locally calibrated model. Common for both Liu et al. (2018) and Padarian et al. (2019) is that the original and new data sets hold the same wavelengths at the same positions. This is contrary to e.g. image data, where two images can hold the same scene, and thereby the same label, but still have e.g. different zoom levels, translations or rotations.
Often, chemometricians work with data of few samples and large amounts of input variables, which may be one of the reasons deep learning has not gained a broad use in the field yet. A well known strategy to easily gain more samples is by data augmentation, where different artefacts are added to each sample. Typical techniques for image data apply rotation and translation. For spectroscopy data, Bjerrum et al. (2017) proposed to add different types of scattering to the spectra. This strategy was used to train a deep neural net on the 2002 IDRC Challenge Data (Norris and Ritchie, 2008;Hopkins, 2003). However, data augmentation cannot fully compensate for the lack of original training samples, as it only enables the neural net to compensate for the artefacts one add to the original samples.
Another strategy is to merge multiple data sets. Ma et al. (2015) showed that by merging multiple small QSAR data sets, they were able to learn a much better Deep Neural Net (DNN) with multiple tasks than if the same DNN was trained on the individual data sets. They also found that the gain of merging multiple data sets is larger for many small data sets than e.g. two large data sets.
A third strategy is to use transfer learning, which deep learning has proven useful at due to its focus on learning representations of data (Bengio, 2012). Applications using images, text and speech have particularly benefited from the use of deep learning and transfer learning (LeCun et al., 2015). Well known, pre-trained networks such as AlexNet (Krizhevsky et al., 2012), GoogleNet (Szegedy et al., 2015), etc. are available as off the shelf solutions to quickly and without large amounts of data get started on your own image based deep learning applications.
The lack of consistency in the number of wavelengths used when spectral data sets are constructed makes it difficult to merge these data sets into one. Common practice within image analysis is to resize the images. However, due to the continuous nature of the spectra, this is likely to introduce noise, and thereby reduce the final performance of the model. We therefore propose a new strategy, that makes it possible to merge the information from multiple data sets without resizing them. This is done by assigning a deep neural net to each data set, with the restriction that the weights in the convolutional layers are shared among the nets. We note that this strategy has similarities to transfer learning. However, our strategy differ in one key aspect. In transfer learning, the learned representation is transferred from one data set to another. In our approach, all data sets contribute to learning the optimal representation. This paper is structured as follows: In Section 2 we describe the proposed method for co-training a deep CNN on multiple data sets with different input sizes. In Section 3 we describe the data sets used in this study. Our experimental setup is described in Section 4, the results are presented and discussed in Section 5 and Section 6 concludes the paper.

Method
In this section we describe our proposed method for training deep convolutional neural nets on multiple data sets with varying input sizes through Weight Sharing. Subsequently, we present a regularization cost for achieving sparse and decoupled weights.

Weight Sharing
Consider the 1d convolution between a signal x of length p and a filter of length 2h + 1 given by the parameters θ in Eq. (1) (with a suitable padding strategy at the endpoints). Realizing that the length of the input signal is not the limiting factor, a convolution could also be performed on another signal x * of length p * using the same filter.
Based on this, we propose to create multiple neural nets with the same overall architecture, but with varying numbers of input variables, where the weights of the convolutional layers are shared. In this way, one can learn higher level representations that generalize to multiple data sets regardless of the data sets having different input sizes. The strategy is illustrated in Fig. 1, where samples of different input sizes pass through the same convolutional layers. After the convolutional layers, the net is split into two, with separate fully connected layers.
When training the nets, the k'th net has its own cost function d k (X k , y k ; Θ k ), with (X k , y k ) being the k'th data set and Θ k being the parameters related to the k'th net. One could combine the cost functions into one by the weighted sum d((X 1 , y 1 ), (X 2 , y 2 ), . . .
However, this would require tuning of the weight parameters α i , as a non-optimal choice would result in one cost function dominating the sum. Instead, we propose to use an alternating approach, where we alternate between updating each individual net (and shared weights). Besides avoiding the need to tune α, in our experience, this also helps in avoiding local minima.

Sparse and decoupled weights
In the case of multi-task learning, it can be of interest to enforce independence of the weight vectors for each output if the individual outputs are independent. Let the k'th layer have p (k) units, the (k + 1)'th layer have p (k+1) units and θ (k) = [θ 1,k , θ 2,k , . . . ] be the p (k) × p (k+1) weight matrix associated with with the k'th layer. For each row of θ (k) we only want one non-zero value. This can be achieved by using the regularization cost given in Eq. (2), where λ ≥ 0 is the regularization parameter. In our experience the main issue regarding tuning λ is that it is not set too high, such that the regularization cost dominates the cost function in the beginning of training, which limits the initial learning by forcing the parameters towards zero.

Data
This section describes the data sets used in this paper. They consist of NIR measurements from silage, a mixture of food substances (soya oil, lucerne and barley), pharmaceutical tablets, wheat kernels and diesel fuels, respectively. All data sets are measured at different numbers of wavelengths, but all in the NIR region of the electromagnetic spectrum.

Chimiometrie 2018
This data set was first published as challenge data at the Chimiometrie 2018 conference in Paris and is available at the conference homepage 2 . The data set consists of NIR measurements from 10 different types of silage measured at 680 (unknown) wavelengths. The target was to predict the protein content, as the silage type was not provided for each measurement. A calibration set consisting of 3908 unique spectra and corresponding target values were provided. Furthermore, 429 test spectra were provided without the target values. Out of the 429 test spectra, 57 spectra were also present in the training data. However, the participants were not informed of this.

Benchmarks
At the challenge, the winning calibration was judged by its median absolute deviance (MAD). However, as this is invariant to an additive constant, we will also use the root mean squared error (RMSE) of prediction to evaluate the the models.
In Table 1, the MAD and RMSE of both the winner of the challenge (Winner) and the data providers' own solution (CRA-W) are shown. The winner of the challenge has the best performance and used Gaussian Process Regression and pre-processed the data using a Standard Normal Variate (SNV) transformation on top of a first order Savitzky-Golay derivative.

Chimiometrie 2019
This data set was published as the challenge at the Chimiometrie 2019 conference held in Montpellier and is available at the conference homepage 3 . The data consist of 6915 training spectra and 600 test spectra measured at 550 (unknown) wavelengths. The target was the amount of soy oil (0 − 5.5%), lucerne (0 − 40%) and barley (0 − 52%) in a mixture. The test set was measured using a different instrument, resulting in a shift in the test spectra of 0.5nm, making the challenge harder.

Benchmarks
At the challenge, the objective was to minimize the Weighted RMSE (WRMSE) given in Eq. (3), withȳ soya oil , y lucerne andȳ barley being the average amount of soy oil, lucerne and barley in the training samples.
In Table 2, the WRMSE of both the data providers (UCO 4 ) and the winner of the challenge (Winner) are shown.
The data provider has the best performance and used a combination of Standard Normal Variate (SNV) and 1st order Savitzky Golay filtering as pre-processing of the data. They then used LOCAL (Shenk et al., 1997) as the calibration method.

IDRC 2002
This data set was first introduced at the 2002 International Diffuse Reflectance Conference (IDRC) as a challenge. However, as the homepage is no longer available, we collected the data at Eigenvector's homepage 5 . The data consist of NIR measurements of a total of 655 pharmaceutical tablets each measured on two different instruments at 650 wavelengths with the objective to predict the amount of active ingredient (API). For each tablet the weight of both the tablet and the total amount of API are provided. The data set is divided into a training set consisting of 155 tablets, a validation set of 460 tablets and a test set of 40 samples. Further details of the data are described in Hopkins (2003). We note that in this study, we only use data from instrument 1. Furthermore, we use the 460 validation samples as our test set, and combine the original training set and test set into one training set of 195 samples.

Wheat
This data set was published together with the papers Pedersen et al. (2002); Nielsen et al. (2003) and is available at the KU FOOD Quality and Technology homepage 6 . The data consist of NIR spectra samples of wheat kernels collected from different locations measured at 100 wavelengths. The data set is divided into a training set of 415 samples and a test set of 108 samples.

Benchmarks
This data set has been used in several studies. In Table 3 the RMSE of both the data providers and the current benchmarks for linear and non-linear methods are shown. It is clear that not much is gained from changing into a non-linear method like CNN, we therefore suspect that the signal of interest is linear.

Experimental Setup
In this section we describe the setup of the experiments conducted in this study, the training strategy used and how we will evaluate and compare the final performance. For all studies we consider two architectures of the neural nets, where the difference lies in the filter length of the shared part. Furthermore, we add two fully connected layers separated by a batch normalization layer on top of the shared part. The parameters of the fully connected layers are not shared among the nets. For details on the architecture see Tables 9 and 10 in Appendix A.

Experiment 1: Weight sharing for two medium sized data sets
We train on the Chimiometrie 2018 and 2019 data sets with shared weights among the convolutional layers. The nets are updated 50, 000 times with a batch size of 128 samples from each data set and an initial learning rate of 10 −3 , which is dropped by a factor of 2 when there hasn't been an improvement in the validation error for 10 epochs -this is done until a minimum learning rate of 3 · 10 −5 is reached.
To asses the performance of Weight Sharing, as baseline we perform the same experiments with individual training instead of co-training.
The performance is evaluated using RMSE and MAD on the Chimiometrie 2018 data set and WRMSE and biases of the three targets on the Chimiometrie 2019 data set.

Experiment 2: Weight sharing for a small and a medium sized data set
We train on the Chimiometrie 2019 data set and a smaller data set and share the weights among the convolutional layers. We use the same training strategy as outlined in Section 4.1.
Besides being used in a co-training setting, the proposed method can also be used for transfer learning, even though the pre-trained net doesn't have the same number of input variables as the smaller data set. We do this by picking the two best performing nets from the medium sized data sets in Section 4.1 trained individually. We then transfer the parameters of the convolutional layers and subsequently train the network with the smaller data set. We employ two strategies for updating the parameters. 1. TL WS Stop Gradient: We only update the fully connected layers with the smaller data set. 2. TL WS Full Gradient: We update the entire net using the smaller data set. We update the nets for 200 epochs using a batch size of 128 and an initial learning rate of 10 −3 , which is dropped by a factor of 2 when there hasn't been any improvement for 50 epochs, until a minimum learning rate of 3 · 10 −5 is been reached.
As a baseline, we perform traditional transfer learning, where we either pad the spectra on both sides or interpolate using cubic splines, such that the input size matches that of the pre-trained net. We employ the two training strategies as described above, naming them 1. TL Stop Gradient and 2. TL Full Gradient.
We evaluate the performance of the strategies using RMSE, Standard Error of Prediction (SEP), R 2 and Bias.

Data splits
For each experiment, we perform 40 repetitions in order to assess the statistical properties of the strategies. For each repetition of the experiments we sub-sample our training data into three data sets: training data used to train the models, validation data used during training to decide at which iteration to store the model and a hold out data set used after training to select the architecture. Recall that in all cases, except for the SWRI data set, the test set is fixed and used to evaluate the performance of the opposing strategies for each experiment. The number of samples in each data set is given in Table 4. We note that for the SWRI data set the test data are not overlapping between repetitions. Given a repetition number, we use the same sub-sampled data set for each of the strategies, this produce paired experiments, which will be utilized in the analysis of the results.  Prior to training we scale up the training and validation data by a factor of 10 using the data augmentation strategy described in (Bjerrum et al., 2017) and append it to the original data. However, we do not augment the hold out data used to select between the architectures.

Optimization strategy
For all our experiments we use the Adam optimizer (Kingma and Ba, 2014). During training we keep track of an exponential moving average smoothing of our parameters as shown in Eq. (4) with a decay rate of γ = 0.99. The exponential moving averaged parameters are used to evaluate the validation samples, and we store parameters minimizing the sum of the validation error of each of the neural nets. We use the exponentially smoothed parameters to achieve a more stable estimate of the parameters. We note that the exponentially smoothed model is similar to the Teacher model proposed by Tarvainen and Valpola (2017), where the difference is that we do not penalize the difference in the prediction between the student and teacher models. Finally, we choose among the opposing strategies using a held out data set, as described in Section 4.3. For all the trained neural nets we use the Rectified Linear Unit (ReLU) (Jarrett et al., 2009;Nair and Hinton, 2010) as activation function.
In many spectroscopic applications, it is common to pre-process the spectra. However, as shown by Cui and Fearn (2018), a CNN is able to automatically learn an appropriate pre-processing of the spectra. Therefore, we do not perform any pre-processing.

Cost function
For the IDRC 2002, Wheat, SWRI and Chimiometrie 2018 data sets we use RMSE as the training and validation cost functions. For the Chimiometrie 2019 data set we use the WRMSE added the regularization cost in Eq.

Pairwise comparisons
In Experiment 1 we perform a pairwise comparison of the two training strategies based on MAD, RMSE, WRMSE and Bias. For this we use the Wilcoxon signed-ranks test (Wilcoxon, 1945). Note than when comparing biases, we compare the absolute values, with the smaller the better.

Multiple comparison of strategies
In Experiment 2 we are comparing five strategies simultaneously, for this we use the Friedman Test (Friedman, 1937(Friedman, , 1939 with the improved statistic by Iman and Davenport (1980). This is done on the measures RMSE, SEP, R 2 and Bias.
As a post hoc analysis we use the Nemenyi test (Nemenyi, 1963) to measure if two rankings are significantly different.

Results
This section presents the results from the two experiments. Besides the results presented here, summary statistics of the experiments are given in Tables 11, 12 and 13 in Appendix B.

Experiment 1
Figure 2 illustrates a pairs plot and kernel density estimates of the performance metrics for the baseline (individually trained models) and the Weight Sharing strategy for the 40 repetitions. For RMSE 2018 it is seen that the mode is shifted to the left for the weight sharing strategy, i.e. to smaller values, while the opposite is the case for WRMSE 2019. It is clear that the reason is that the baseline models have a smaller bias when predicting the amount soy oil (Bias 2019 Soy). Furthermore, we see that for the baseline models, MAD 2018 has a plateau like mode while the Weight Share model has a sharp peak. Finally, in 37 cases the Weight Sharing strategy had a WRMSE on the Chimiometrie 2019 data set which was smaller than the benchmark, while this was the case for all 40 cases for the baseline. For the Chimiometrie 2018 data set, none of the strategies were able to beat either the MAD or RMSE benchmarks.
We perform a Wilcoxon signed-rank test on MAD and RMSE for the Chimiometrie 2018 data set and WRMSE and the absolute bias for the three targets for the Chimiometrie 2019 data set. The test values and p-values are shown in Table  5. It is seen that the Weight Share strategy is significantly better in terms of RMSE on the Chimiometrie 2018 data set, while the baseline is better in terms of WRMSE and bias when predicting the amount of soy oil for the Chimiometrie 2019 data set. The reason for this is that there is a large noise component in the 2019 test data set, as the test data is measured using a different instrument, which caused a shift of 0.5nm of the spectra.
Performing an F-test for a change in variance of all the considered performance metrics produces the test statistics and corresponding p-values presented in Table 6. It is seen that in neither case there is a significant change in variance of the performance metrics between the two strategies.

Experiment 2
A pairs plot of the performance metrics for the IDRC 2002 data set is shown in Figure 3. For all metrics the mode for Weight Sharing is better than all four transfer learning strategies. Further, it is clear that the transfer learning strategies produce distributions with long tails of large statistics. Figure 4 illustrates a pairs plot of the performance metrics for the Wheat data set. First it is noted that the both Stop Gradient strategies are clearly separated from the other three strategies. Further, the two Full Gradient strategies are overlapping, with modes sligtly better than that of Weight Share. We also note that for none of the strategies the modes are close to the benchmark. This is expected as we, contrary to the reported benchmarks, are using a subset of the already limited amount of training data to train our models (although in 1 case for TL WS Full Gradient and 4 cases for TL Full Gradient the performance is actually better than the benchmarks). Figure 5 illustrates the performance metrics for the SWRI data set in a pairs plot. It is seen that, for all metrics, the Weight Share and TL WS Full Gradient strategies are overlapping, with the mode of Weight Share being slightly better in all cases. Furthermore, it is seen that the other transfer learning strategies produce distributions with long tails of large statistics.
The test statistics and p-values when testing for a change in variance of the performance metrics is given in Table 7. It is clear that for the IDRC and SWRI data sets, a significant decrease of variance has occurred in most cases. For the Wheat data set, an increase of the variance is observed in most cases -11 out 16 being statistically significant. Here we note that, as seen in Figure 4, the two Stop Gradient strategies are consequently performing worse than the three other strategies.      Table 7: Test statistics and p-values for F test on change in variance of performance metrics. Significant tests are highlighted with bold faced numbers. The test is performed such that a test value larger than 1 corresponds to a variance reduction from the baseline to the Weight Share strategy, and vice verca for a test value less than 1.
The average rankings of the five strategies in terms of the performance metrics RMSE, Sep, R 2 and (absolute) bias are shown in Table 8a. For all metrics the ranking is 1) Weights Share 2) TL Full Gradient 3) TL WS Full Gradient. Based on these, a Friedman rank test is performed with the test statistics and p-values shown in Table 8b. It is clear that all the tests are significant, meaning that there is a significant grouping for all performance metrics. For the post hoc analysis of the rankings we calculate the critical value for pairwise differences as CD = 2.728 · 5 · (5 + 1)/(6 · 40 · 3) =

Conclusion
We have proposed a novel method for training deep convolutional neural networks that learn from multiple data sets containing different numbers of variables using weight sharing. We demonstrated this in two experiments. In the first experiment we combined two medium sized data sets and compared the performance to that of neural nets trained individually on each data set. In the second experiment we combined a medium sized and a small data set, and compared the performance to that of transfer learning from a pre-trained network.
We have showed that when combining two medium sized data sets, this reduces the variance of the produced networks of most of our performance metrics. Furthermore, the proposed strategy produced a significantly smaller prediction error on test samples with the same distribution as the validation set, while for test samples with a different distribution than the validation set (a small shift in wavelengths), the individual trained net performed better.
The proposed method enables training of deep convolutional neural nets though only having few training samples available by co-training with a medium sized data set. Furthermore, it also enables transfer learning without resizing the smaller data set. We showed, that for a small number of training samples, the proposed co-training procedure outperformed both types of transfer learning strategy.