{'fraction_non_alphanumeric': 0.0450035584208984, 'fraction_numerical': 0.015531460627119353, 'mean_word_length': 4.377757766771724, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 4, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Score-oriented loss (SOL) functions

Loss functions engineering and the assessment of forecasting performances are two crucial and intertwined aspects of supervised machine learning. This paper focuses on binary classification to introduce a class of loss functions that are defined on probabilistic confusion matrices and that allow an automatic and a priori maximization of the skill scores. The performances of these loss functions are validated during the training phase of two experimental forecasting problems, thus showing that the probability distribution function associated with the confusion matrices significantly impacts the outcome of the score maximization process.

1. Introduction. Neural networks (NNs) are well-established models in the field of machine learning. Given their flexibility and capability in addressing complex tasks where a large amount of data is at the disposal, deep NNs are the state-of-art of many classification issues, including image recognition [26], speech analysis [5] and medical diagnostics [15]; for a detailed treatment of various NNs architectures, as well as of related deep learning topics, we refer e.g. to [9].
In general, a network depends on a set of weights that are optimized in such a way that an objective function, made of a loss function and a regularization term, is minimized [7]. Loss functions measure the possible discrepancy between the NN prediction and a label encoding the information about the event occurrence. Loss design is a crucial aspect of machine learning theory [28] and, over the years, several functions have been proposed, typically with the aim of accounting for the specific properties of the problem under analysis [14,20,22,36]. Among these choices, Cross Entropy (CE) and its generalizations [8,23,35] possess a solid theoretical background that can be traced back to information theory, they being related to the notions of entropy and Kullback-Leibler divergence, and their minimization corresponds to the maximization of the likelihood under a Bernoulli model (see e.g. [24,Section 2.8]).
While the network is optimized by minimizing a certain loss, the classification results are then evaluated by considering some well-established metrics or skill scores, which are often chosen according to the particular task and derived from the elements of the so-called confusion matrix (CM). Although loss minimization and score maximization are two intertwined concepts, yet a direct maximization of a score during the training of the network is to be avoided, since the score is typically a discontinuous function with respect to the predictions given by the model. Therefore, other strategies have been taken into consideration.
For example, in [13] the authors propose an adaptive loss alignment with respect to the evaluation metric via a reinforcement learning approach. A loss function that approximates the 0-1 loss is proposed in [31]. In [27], a loss that generalizes the Jaccard index is discussed for object detection issues. In order to address the lack of regularity of the score, fuzzy or probabilistic confusion matrices have been examined e.g. in [16], where the minimization of the off-diagonal elements of a probabilistic confusion matrix is considered; in [33], where a confusion entropy metric is constructed; and in [32], where local fuzzy CMs have been employed.
This fuzzy/probabilistic approach was inspirational for the framework introduced in the present paper, which also presents similarities with the main idea investigated in [25] in the context of kernel logistic regression, and which has to be intended as the theoretically-founded generalization of the approach carried out, e.g., in [34]. Indeed, our study addresses the problem of filling the gap between score maximization and loss minimization by constructing a class of Score-Oriented Loss (SOL) functions out of a classical CM. The main idea consists in treating the threshold that influences the entries of the confusion matrix not as a fixed value, but as a random variable. Therefore, by considering the expected value of the matrix, we build losses that are indeed derivable with respect to the weights of the model. Furthermore, we theoretically motivate and then numerically show that the probability distribution chosen a priori for the threshold severely influences the outcome of the a posteriori score maximization. More in details, the optimal threshold is very likely to be placed in the concentration areas of the a priori distribution. As a consequence, using a SOL function related to a certain skill score, we automatically obtain an optimized value without performing any further a posteriori procedure.
The paper is organized as follows. In Section 2, starting from a classical confusion matrix, we construct the theoretical foundation of our SOL functions and we investigate their properties. Section 3 describes two possible choices for the probability distribution associated to the threshold and illustrates their application to two experimental forecasting problems. Finally, our conclusions are offered in Section 4.

Construction and properties of score-oriented losses.
Let Ω ⊂ R m and let X ⊂ Ω be a set of data, m ∈ N ≥1 . Let Y be a finite set of d labels or classes to be learned, where d ∈ N ≥2 and the labels (or classes) are integer encoded or one-hot encoded in practice [11]. Suppose that each element in X is uniquely assigned to a class in Y. Then, the supervised classification problem, where d = 2, consists of constructing a function g on Ω by learning from the labeled data set X , in such a way that g models the data-label relation between elements in Ω and labels in Y. In order to achieve such result, one usually defines a loss function (g(x), y) that measures the possible discrepancy between the prediction g(x) given by the model, x ∈ X , and its true label y ∈ Y. In this view, an NN builds the function g by minimizing the loss on the training set X, as well as focusing on its generalization capability when predicting possible unseen test samples in Ω.
More specifically, the network produces the output where h(x, w) denotes the outcome of the input and hidden layers, being w the vector (matrix) of weight parameters, and σ is the sigmoid activation function defined as During the training procedure, the weights of the network are adjusted via backpropagation in such a way that a certain objective function is minimized, i.e. we consider the following minimization problem where is a loss function and R(w) is a possible regularization term that is controlled by a parameter λ ∈ R >0 . The outcomes of this regression problem can be clustered by means of some thresholding process in order to construct a classifier whose predictive effectiveness is assessed by means of specific skill scores. We now introduce a oneparameter family of CMs from which we will derive a set of probabilistic skill scores. This approach will allow the introduction of a corresponding set of probabilistic loss functions and, accordingly, an a priori optimization of the scores.
2.1. Confusion matrices with probabilistic thresholds. Let us consider a batch of predictions-labels S = S n (w) = {(ŷ(x i , w), y i )} i=1,...,n , where y i ∈ {0, 1} is the true label associated to the element x i ∈ X andŷ(x i , w) ∈ [0, 1] is the prediction for x i given by the model. Furthermore, let n + and n − be the number of elements in S n (w) whose true class y i , i = 1, . . . , n, is the positive ({y = 1}) or the negative ({y = 0}), respectively. Therefore, n + + n − = n.
Let τ ∈ R, τ ∈ (0, 1), and let be the indicator function. The classical confusion matrix is defined as where the elements are In what follows, we let τ be a continuous random variable whose probability density function (pdf) f is supported in [a, b] ⊆ [0, 1], a, b ∈ R. Furthermore, we denote as F the cumulative density function (cdf) We recall that This leads to the following definition.
Definition 2.1. Let S n (w) be a batch of predictions-labels and let CM(τ, S n (w)) be a confusion matrix as defined in (2.2). Moreover, let τ be a continuous random variable on (0, 1) with cdf F . We define the expected confusion matrix as ).

(2.3)
Remark 2.2. We observe that CM F is indeed the expected value of the matrix CM(τ ) with respect to τ , meaning that Moreover, the sum by rows is preserved, that is for any τ ∈ (0, 1) Hence, all considered matrices are contained in the set where A = (A ij ) i,j=1,2 and M + 2 is the set of 2 × 2 positive-valued matrices. Therefore, we can prove the following results by considering well-known concentration inequalities: Proposition 2.3. Let S n := S n (w) be a batch of predictions-labels, n ∈ N, and let τ be a continuous random variable on (0, 1) with cdf F . Then, letting N ∈ {TN, FP}, P ∈ {TP, FN} and ε > 0, we have (2.6) Moreover, denoting as tr(CM) = TN + TP the trace of the confusion matrix, we get Proof. In order to prove (2.6) it is sufficient to consider the McDiarmid's inequality (see e.g. [30,Theorem 4.5,p.88]) and to observe that Then, we can apply the Hoeffding's inequality [12] to TN(τ, S n ) + TP(τ, S n ), thus obtaining (2.7).

Derived scores and loss functions.
A skill score is a function mapping the space of confusion matrices onto R. In the classical setting, the skill score is denoted as s : M 2,2 (N) → R and is constructed on CM(τ, S n (w)) for a fixed value of τ ∈ (0, 1). In the case of confusion matrices with probabilistic threshold, the score explicitly depends on the cumulative distribution function F adopted and is therefore denoted as s F : M 2,2 (R) → R. In this latter case, we can introduce a class of Score-Oriented Loss (SOL) functions as follows.
Definition 2.4. Let S n (w) be a batch of predictions-labels and let s(τ, S n (w)) = s(τ ) be a score calculated upon CM(τ, S n (w)), τ ∈ (0, 1). Indicating with s F (S n (w)) = s F the score s calculated upon CM F (S n (w)), a Score-Oriented Loss (SOL) function related to s is defined as s (F, S n (w))) := −s F (S n (w))).
As an example, taking the accuracy score and then the loss function acc (F, S n (w)) := −acc F (S n (w))).
Remark 2.5. By setting 0/0 := 0 and by virtue of the fundamental theorem of calculus, we observe that s is derivable with respect to the weights vector w, which is a fundamental property for its usage as loss function.
We now prove the following Theorem 2.6. Let S n := S n (w) be a batch of predictions-labels, n ∈ N, and let τ be a continuous random variable on (0, 1) with cdf F . Then, if s is linear with respect to the entries of the confusion matrix, we have Moreover, letting supp(s(τ, S n )) = [a, b], a, b ∈ R, a < b, and ε > 0, we obtain Proof. Due to the linearity of the expected value, the results in Proposition 2.3 imply (2.8). As a consequence, the McDiarmid's inequality can be applied in order to obtain (2.9).
Recalling (2.1), let T (w) := (S(w)) + λR(w) be the target function to minimize during the training of the network, where is a generic loss function. When performing a posteriori maximization of a score s, letting w = argmin w T (w), we compute max τ ∈(0,1) s(τ, S(w )).
In our setting with SOL functions, in view of Definition 2.4 and Remark 2.7, the training minimization problem results in where the equality is achieved under the assumptions of Theorem 2.6. Therefore, the maximization of the expected value of the score is included in the training process. Moreover, we observe that while a direct maximization of the score s(τ , S(w)) for a fixed τ ∈ (0, 1) is not possible due to the lack of derivability, this procedure can be approximated by considering for τ a probability distribution highly concentrated around the mean value τ . In Section 3, we review two different probability distributions that express different concentration properties.
The following result concerns the mean absolute deviation (mad) [6] of a Lipschitz continuous score s from its expected value s F . Theorem 2.8. Let S n be a batch of predictions-labels, n ∈ N, and let τ be a continuous random variable on (0, 1) with cdf F . Assume that s is a Lipschitz continuous function with respect to the entries of CM(τ, S n ) on the set C(S n ). Then, with C(S n ) as defined in (2.5), and mad indicates the mean absolute deviation.
Proof. As in Remark 2.7, we consider s as a function of the vectorized confusion matrix vec(CM(τ )) = (TN(τ ), FP(τ ), FN(τ ), TP(τ )), and thus where K s is the Lipschitz constant defined in (2.10). Then, since We proceed by taking the expected value of both sides of (2.12). Then, in order to conclude the proof, it is sufficient to observe that and to define J F as in (2.11). which yields to K acc = 1/n.
3. Applications to experimental forecasting problems. The implementation of a machine learning approach based on SOL functions requires first the choice of the probability distribution function associated to the threshold τ . In this paper we will utilize two distributions supported in [0, 1].
Specifically, if τ ∼ U(0, 1), i.e. it is a random variable uniformly distributed on (0, 1), its pdf and cdf are f U (x) = 1 and F U (x) = x respectively, x ∈ (0, 1) (see Figure  1, top row). This well-known distribution is characterized by a high variance and does not favor any particular value of the threshold. Therefore, with respect to a classical confusion matrix (2.2), in CM F U the indicator functions of the outputs are replaced by the outputs themselves. Although its probabilistic derivation from the uniform distribution is not trivial, from an empirical viewpoint the confusion matrix CM F U is probably the most straightforward way to extend a classical confusion matrix to having non-integer valued entries [18].
We now test the performances of SOL functions relying on these two distribution functions, when applied to two actual binary classification problems.
A Python implementation of the SOLs is available for the scientific community at https://github.com/cesc14/SOL .

3.1.
Optimized a posteriori thresholds and a priori distributions. We first considered the Adult training dataset [17], where the task is to predict whether a person can make more than 50,000 dollars per year based on census data. First, we performed the following pre-processing steps: 1. We removed the samples containing missing values. 2. We dropped the feature education, which is highly correlated to the feature education-num. Indeed, both features concern the education level of the people in the dataset. 3. The possible values in the native-country column, which reports the country of origin of the people in the data set, were restricted to 1 in place of United-States and 0 for all other countries. 4. All categorical features were one-hot encoded. Finally, all features were standardized. The resulting elements were then described by 48 numerical features. The data set consisted of 30162 samples, where 7508 people made more than 50,000 dollars per year, while 22654 did not. As for the network architecture, we used a fully-connected feed-forward NN with an input layer (50 ReLu neurons), two hidden layers (20 and 5 ReLu neurons) and a sigmoid output unit. During the training phase, 1/3 of training set was used as a validation set. We trained the network on 500 epochs and we imposed an early stopping condition when no improvement in the validation loss was obtained after 30 epochs.
The following experiments were devoted to show how the choice of the distribution of the threshold τ affected the result of the a posteriori score maximization performed by varying the threshold in (0, 1). In doing so, we considered the SOL related to the f 1 -score [21], which is the harmonic mean of precision and recall, i.e.
We observe that this score does not satisfy the linearity assumption in Theorem 2.6 (see also Remark 2.7). We tested the uniform distribution U as well as the raised cosine distribution C(µ, δ) with different values of the parameters µ, δ. More precisely, for each combination of (µ, δ) ∈ {0.3, 0.5, 0.7}×{0.1, 0.2, 0.3} we randomly selected 4/5 elements in the data set for training the NN and we performed the a posteriori score maximization on the training set. We repeated this procedure 100 times.
In Table 1, we present the obtained results. We report that in some cases the training optimization process was unsuccessful, that is the validation loss got stuck from the beginning of the training, and no actual learning took place. Therefore, we did not include such cases in the computation of the other entries of the table.
Let η ∈ N, η ≤ 100, be the number of successful training processes. Figure 2 compares the probability distribution chosen a priori for the threshold as in the left panel of Figure 1 (dashed blue line) to the empirical probability distribution f τ (red histograms) derived from the optimal thresholds τ 1 , . . . , τ η over the η successful runs. We modelled this distribution via normalized histograms. Moreover, we also displayed the average optimal threshold (red cross) resulting from the η runs. The horizontal solid red line indicates the standard deviation related to the optimal thresholds. These experiments show that the optimal threshold is indeed heavily influenced by the chosen distribution, which effectively drives the optimization process to prefer a certain interval of threshold values.  Table 1: Outcomes of the application of the SOL functions in the case of a neural network trained on the Adult database. The chosen SOL relies on the f 1 -score and the threshold τ is drawn according to the uniform and raised cosine distributions.
3.2. SOLs versus CE: addressing strongly imbalanced classes. We then considered the problem of predicting high levels of pollution by using the Beijing air quality data set from the University of California at Irvine (UCI), available at https : //archive.ics.uci.edu/ml/datasets/Beijing + P M 2.5 + Data. The data set  contains PM2.5 pollution data and meteorological data including dew point, temperature, pressure, wind direction, wind speed and the cumulative number of hours of snow and rain. Data are collected every hour for five years, sourced from the data interface released by the US Embassy in Beijing [19].
We first fixed a level of PM2.5 concentration equal to 400 and we labelled the feature vector containing the meteorological data and the PM2.5 concentration at time T with class 1 if the PM2.5 concentration at time T + 1 is over the fixed level. The data set ranges from 01/01/2010 to 12/31/2014: we considered 13104 samples in training (about one year and a half) and 4320 samples in testing (about half a year) by taking the testing period right after the training period. In order to guarantee statistical robustness we repeated 100 times the construction of training and test sets, by shifting the training period and test period of 5 days each time. The data set is strongly imbalanced since the mean rate of positive samples in the training set is about 1.41% and in the testing set is about 1.19%. During the training phase, 1/3 of the training set was used as validation set.
We used a fully-connected feed-forward NN with an input layer (15 ReLu neurons), a hidden layer (8 ReLu neurons) and a sigmoid output unit. We trained on 1000 epochs and we imposed an early stopping condition when no improvement in the validation loss was obtained after 50 epochs. As for loss functions, we considered the SOL functions related to the True Skill Statistic (TSS) [1] and Critical Success Index (CSI) [29], which are very common skill scores adopted in meteorology applications and particularly appropriate in the case of strongly imbalanced data set. They are defined as We point out that the TSS score satisfies the linearity assumption in Theorem 2.6, while the CSI does not. Considering both the uniform and raised Cosine distributions C(µ, δ) (in the case µ = 0.5 and δ = 0.1, 0.3), we compared the proposed SOLs against the CE loss. For each run we performed the a posteriori score maximization on the training set in order to compute the optimum threshold. In Table 2 we reported the performances on both training and test sets by evaluating the skill scores with both the usual threshold 0.5 and the optimum threshold τ selected by the a posteriori maximization. As in the previous application, we did not include cases in which the training optimization process was unsuccessful, and we reported the number of success on 100 runs. Table 2 shows that using the SOL related to a given skill score leads to better results with respect to the ones obtained by using the CE loss, in the case of both the training and the test sets. In particular, the skill scores are largely satisfactory by setting τ = 0.5 without the need to perform the a posteriori score maximization. In the case where the raised cosine distribution is implemented, this is due to the capability of the SOL function of driving the optimization process to promote thresholds in a neighborhood of the parameter µ (equal to 0.5 in these experiments). In the case of the uniform distribution, no particular values of the threshold are promoted in the optimization process and this leads to results with high skill scores independently from the choice of the threshold.
Finally, we notice that the mean number of epochs to reach the convergence condition is lower for SOL functions with respect to the CE loss: in particular such a number is much lower for the SOL function in which the raised cosine distribution is chosen with δ = 0.1.   ) scores and the threshold τ is drawn according to the uniform and raised cosine distributions. In columns "TSS" and "CSI" the scores are computed with respect to the threshold τ = 0.5.

Conclusions.
In this paper, we discussed a new approach for the automatic maximization of a chosen score in classification tasks. After providing a detailed theoretical construction, we showed the effectiveness as well as the consistency of our framework. In particular, we described how this approach naturally allows the introduction of loss functions whose explicit form depends on the form of the skill scores defined on probabilistic confusion matrices. The performances of these loss functions during the training phase of supervised networks are tested in the case of two experimental forecasting problems.
We point out that the SOL functions that we have here introduced in the binary classification setting can be extended to the multi-class case, i.e. for d > 2, in a rather straightforward fashion. In fact, the idea is to use a standard one-versus-rest approach in which d different 2 × 2 matrices CM 1 (τ 1 , S(w)), . . . , CM d (τ d , S(w)) are derived in the case of τ 1 , . . . , τ d different random variables. In this way, the investigation of the multi-class case reduces to one of the binary setting.
As far as future work is concerned, we are currently following two paths. From a methodological viewpoint, we think that the SOL functions approach should be extended to the case of value-weighted skill scores that allow the assessment of prediction errors on the basis of their impact on the decision making process [10].
From the viewpoint of applications, our main interest is to exploit this approach in the framework of space weather [4] and, in particular, for the prediction of extreme solar flaring storms, whereby the training sets of solar magnetograms used for training the networks are strongly imbalanced against highly energetic events [3].

5.
Acknowledgements. SG is financially supported by a regional grant of the 'Fondo Sociale Europeo', Regione Liguria. FM is supported by the financial contribution from the agreement ASI-INAF n.2018-16-HH.0. MP and CC acknowledges this same agreement. This research has been partly accomplished within the Rete ITaliana di Approssimazione (RITA) and with the support of GNCS-INδAM.