{'fraction_non_alphanumeric': 0.04548786873799965, 'fraction_numerical': 0.012916739396055157, 'mean_word_length': 4.177299837339599, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 17, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
The Fisher-Rao Loss for Learning under Label Noise

Choosing a suitable loss function is essential when learning by empirical risk minimisation. In many practical cases, the datasets used for training a classifier may contain incorrect labels, which prompts the interest for using loss functions that are inherently robust to label noise. In this paper, we study the Fisher-Rao loss function, which emerges from the Fisher-Rao distance in the statistical manifold of discrete distributions. We derive an upper bound for the performance degradation in the presence of label noise, and analyse the learning speed of this loss. Comparing with other commonly used losses, we argue that the Fisher-Rao loss provides a natural trade-off between robustness and training dynamics. Numerical experiments with synthetic and MNIST datasets illustrate this performance.


Introduction
Supervised classification is an important problem in machine learning. Training a classifier (e.g., a deep neural network) can be done by empirical risk minimisation: a numerical optimisation algorithm is applied to find the model parameters that minimise the mean value of a loss function on the training dataset. Choosing a suitable loss function is essential, since different choices can affect the performance of the resulting classifier, as well as the training dynamics.
The output of a neural network trained for classification is often interpreted as giving a conditional probability distribution p(y|x x x) of the class y given the input x x x, which prompts the use of cross entropy as a loss function [1,2,3]. Although originally used for regression problems, the mean squared error is also used as loss function, and several works have compared these two losses [4,5,6,7,8]. Moreover, the design of new loss functions has been a topic of interest, and those are often tailored for specific problems or contexts, with many different inspirations, such as the correntropy similarity measure [9], the Wasserstein distance [10,11], and persistent homology [12].
A case of practical interest is when training datasets are corrupted with label noise, i.e., some of the class labels may be incorrect. This is a well-studied problem in machine learning: one of its sources is crowdsourcing labelling, and it can impact the performance of the generated model [13,14]. Many of the proposed solutions to mitigate this issue involve modifying the learning algorithms and have no theoretical guarantees of robustness. On the other hand, changing only the loss functions can be easily implemented and does not require further modifications to the structure of the classifiers. In addition, a sufficient condition for a loss function to be robust to label noise, i.e., for a classifier trained with that loss to achieve the same performance as if it was trained with clean data, was provided in [15,16]. These works also show that commonly used cross entropy and mean squared error losses do not satisfy this condition, while the mean absolute value loss does, but results in slow training. Other robust losses for label noise have been studied in [17,18].
In this work, we study a loss function based on an information-geometric framework [19,20]. It is well known that the Fisher metric is the natural Riemannian metric for statistical manifolds, since it is essentially the only one that is invariant under sufficient statistics [21]. In the case of discrete distributions, the geometry of the corresponding statistical manifold coincides with a spherical geometry; therefore, the Fisher-Rao distance between two points, as the geodesic distance induced by the Fisher metric, can be easily computed as the arc-length of an arc of great circle. The proposed loss function for training classifiers is proportional to the squared Fisher-Rao distance in the manifold of discrete distributions, and has a twofold motivation: first, by the common interpretation that a classifier tries to minimise the error between predicted and ground truth class distributions, and, second, by the fact that the Fisher-Rao distance is a natural distance from the information-geometric viewpoint.
The Fisher-Rao distance has been recently used in unsupervised learning for shape clustering [22], clustering financial returns [23] and image segmentation [24]. It has also found application as a regulariser term for adversarial learning [25], in detection of out-of-distribution samples [26], and to unravel the geometry in the latent space of generative models [27]. Here, we use the Fisher-Rao distance to construct a loss functions on its own, in a standard classification framework, and we compare it with other losses, particularly in the case of label noise. Specifically, we provide an upper-bound for its robustness against label noise, and analyse the learning speed using this loss. We argue that it provides a natural trade-off between these two aspects, and illustrate our theoretical analysis with numerical experiments.
Another well-known application of information geometry in deep learning consists in studying the geometry of neuromanifolds, i.e., the statistical manifolds formed by the joint distribution of the network inputs and outputs, parametrised by the network weights and biases [3]. This gives rise to the celebrated natural gradient method [19], which modifies the gradient descent method by replacing the usual gradient with the gradient computed on this neuromanifold, using the Fisher information metric. We take a different approach in that we focus on the geometry of the outputs of the network, which corresponds to the manifold of discrete distributions, in order to also modify the learning algorithm, but by choosing a different loss function.
The remainder of this paper is organised as follows. Section 2 reviews the geometry of the statistical manifold of discrete distributions. In Section 3, the learning problem is described, and the proposed loss functions is analysed in terms of robustness to label noise and learning speed. Numerical results are presented and discussed in Section 4. Section 5 concludes the paper.

Information Geometry of Discrete Distributions
Let (Ω, F, µ) be a σ-finite measure space and M = p θ = p(x; θ) | θ = (θ 1 , . . . , θ n ) ∈ Θ ⊆ R n a parametric family of probability density functions p θ : Ω → R + with respect to the measure µ. If M is smoothly parametrised by θ ∈ Θ ⊂ R n and satisfies certain regularity conditions [20], a Riemannian metric can be introduced on M , called the Fisher metric, given in matrix form by the denotes the expectation of f (x) with respect to p θ . Furthermore, considering the Kullback-Leibler divergence between two probability densities p and q, the Fisher matrix can be also viewed as the Hessian of the Kullback-Leibler divergence, in the sense that With this structure, which is independent of the parametrisation θ → p θ , M is called a statistical manifold. This is a natural choice for the geometry of probability distributions, as the Fisher metric is essentially the unique (up to a constant) Riemannian metric that is invariant under sufficient statistics on these manifolds [21].
Given a curve θ : [0, 1] → Θ in the parameter space, the length of the corresponding curve γ(t) = p θ(t) in the manifold M is computed, in the Fisher geometry, as The Fisher-Rao distance between two distributions p θ 1 and p θ 2 in M is defined as the infimum of the length of all piecewise smooth paths linking them, i.e., Moreover, when M is a complete and connected manifold, this distance is realised by the length of a geodesic, called a minimising geodesic. Unfortunately, closed-form expressions for the Fisher-Rao distance are only known in a few statistical manifolds, e.g., [28,29,30].
In this paper, we are interested in the statistical manifold structure of discrete probability distri-butions. Given a sample space Ω = {1, . . . , K} and the set of functions δ i : Ω → {0, 1}, defined by the Kronecker delta δ i (j) := δ ij , we hereafter consider the statistical manifold of discrete distributions which is in correspondence with the probability simplex Furthermore, given the set of parameters there is a standard parametrisation ϕ : It is straightforward to show [29] that, in this parametrisation, the Fisher information matrix has the form In this case, there is a simpler way to obtain the geodesics than solving the geodesics differential equations. Considering the reparametrisation z i := 2 √ p i , as in [31,29], we have K i=1 z 2 i = 4, so that the simplex ∆ K−1 is mapped to the part of the radius-two sphere with positive coordinates S K−1 2,+ := z z z = (z 1 , . . . , z K ) ∈ R K + | z z z 2 = 2 . The expression for the Fisher metric in this new parametrisation is Note that this is the formula of the standard spherical metric on S K−1 2,+ , i.e., the Euclidean metric in R K restricted to S K−1 2,+ . As geodesics on the sphere are given by arcs of great circles, the length of a geodesic connecting z z z p and z z z q in S K−1 2,+ (reparametrisation of the distributions p = K i=1 p i δ i and q = K i=1 q i δ i , respectively) is double the angle α between these two vectors: Therefore, we obtain that the Fisher-Rao distance between the discrete distributions p The reparametrisation to S K−1 2 prompts an immediate approximation for the Fisher-Rao dis-tance, given by the chordal distance, i.e., the Euclidean distance in R K , between z z z p and z z z q : This happens to be double the value of the the Hellinger distance, a well-known distance between probability distributions [32, § 2.4]: which is related to the Fisher-Rao distance by Finally, note that, using the Taylor series expansion of the arc-sine function, we get the asymptotic approximation 3 Learning for Classification

Loss Functions
We consider a supervised classification setting: x x x ∈ X ⊆ R m denotes the feature vector, which belongs to exactly one class y ∈ Y := {1, . . . , K}, and we assume the data to follow a joint distribution (x x x, y) ∼ D. Our goal is to learn a classifier (e.g., a neural network) f : X → R K that assigns to each input x x x a vector of scores s s s = (s 1 , . . . , s K ) := f (x x x), which can be used to induce a hard decisionŷ = arg max 1≤i≤K s i . By applying the softmax function σ : R K → ∆ K−1 to the scores s s s, one obtains a conditional probability distribution P (y|x x x) on Y, represented by the vector Learning the model can be done by empirical risk minimisation, which is the population version of the problem min f E D [L(y, f (x x x))] under a suitably chosen loss function L : Y × R K → R + . In the following, we list some possible choices. Two popular criteria are mean squared error and the cross entropy loss functions. Let e e e (y) ∈ R K denote the one-hot vector that has value 1 in the y-th coordinate, and value 0 in the other coordinates.
The mean squared error (MSE) loss is defined as i.e., it is the squared Euclidean distance in the ambient space R K restricted to ∆ K−1 .
The mean absolute error (MAE) loss 1 is defined as which is the taxicab distance in R K restricted to ∆ K−1 .
The cross entropy (CE) loss is given by where i.e., in this case, the cross entropy is equal to the Kullback-Leibler divergence between the true conditional distribution e e e (y) , which is in fact deterministic, and the predicted one An interesting generalisation of the cross entropy loss can be obtained by replacing the logarithm in (8) with the Tsallis q-logarithm 2 [33]: defined for x > 0 and q ∈ R. In particular, for q ∈ [0, 1), this is the loss function proposed in [18], where it was defined in terms of the negative Box-Cox transformation 3 , and given by to which we refer as q-cross entropy loss (q-CE). Interestingly, as remarked in [18], for q = 0, it coincides with the MAE loss, and, for q = 1, with the CE loss.
The main contribution of this work is the study of the Fisher-Rao loss function, inspired by an information-geometric framework, cf. Section 2. This will be defined, up to multiplicative constant, as the square of the Fisher-Rao distance in the manifold of discrete distributions. Taking the square is reasonable, as other loss functions, such as the MSE and q-CE losses, behave, at least locally, as squared distances. Using (2), we can compute d 2 FR e e e (y) , p p p = 4 arccos π y (p p p) Therefore, disregarding the multiplicative constant-which is fine, since multiplying the loss is equivalent to multiplying the learning rate-, we define the Fisher-Rao loss as We advocate that, conceptually, the Fisher-Rao loss is a natural choice of loss function, since it emerges from the Fisher metric in the statistical manifold of discrete distributions, cf. Section 2. Furthermore, using the Taylor series expansion, it can be verified that the q-CE loss (which includes the CE and MAE losses) has an asymptotic approximation in terms of the Fisher-Rao distance, for small values of the distance to e e e (y) : Finally, in view of the arc-chord approximation (5), we can also consider a loss function given by the square of the Hellinger distance, as mentioned in [3, § 3.10]. Using (3), we thus define the Hellinger loss as Note that this is also a particular case of the q-cross entropy loss, for q = 1/2, and that it also provides an approximation for the Fisher-Rao loss.
1. We have the asymptotic approximation: In particular, respectively for q = 0, q = 1 and q = 1/2, we have: 2. Moreover, the following inequality chain holds: Proof. In the following, we omit the argument of the losses, for notational simplicity. We shall first prove item 1 for q ∈ [0, 1). Isolating p y in (10) and substituting it in (11), we obtain The first-order Taylor series expansion of g 1 (x) := arccos 1 − (1 − q)x , yielding the desired result. Now, we prove it for q = 1. Again, isolating p y in (10) and substituting it in (11), we obtain Analogously, the first-order Taylor series of g 2 (x) : , completing the proof of the first item.
For the first inequality of item 2, note that The second inequality is immediate from (4), i.e., the fact that the chordal distance is less than the geodesic distance on the sphere. For the third inequality, note that the first-order Taylor expansion of g 2 (x) around zero can be written as g 2 (x) = x + R 1 , with These results state that the q-CE loss-and, in particular, MAE, CE and Hellinger loss functionscan be seen as first-order asymptotic approximations for the Fisher-Rao loss. However, while the approximations are valid for small values of the losses, the functions may behave largely differently for higher values.

Robustness to Label Noise
Let us now consider the problem of learning in the presence of label noise, cf. [15,16]. In this case, the classifier does not have access to a set of clean samples ( , whereỹ i denotes the noisy labels. We suppose the noisy data follow a joint distribution (x x x,ỹ) ∼ D η .
Here, we focus on the case of uniform or symmetric label noise, i.e., when the noisy labelỹ i does not depend on the feature vector x x x i , nor on the true label y i . In this case, the label noise is modelled by for some constant η ∈ [0, 1], called noise rate. We denote R L (f ) := E D L(y, f (x x x)) and R η L (f ) := E Dη L(ỹ, f (x x x)) the risks with respect to clean and noisy data, respectively. Moreover, let f * andf be global minimisers of R L (f ) and R η L (f ), respectively. The risk minimisation under loss function L is said to be noise tolerant [34,15] iff has the same probability of misclassification as that of f * . In other words, the performance of the classifier trained with the noisy dataset is as good as it would be had it been trained with a clean dataset.
An important contribution from [16, Theorem 1] was to provide a sufficient condition for a loss to be robust under label noise: a loss function L is tolerant under uniform label noise with η < for all x x x ∈ X and all classifiers f . It is shown that the MAE loss satisfies this condition, while the MSE and CE losses do not. However, even for loss functions not satisfying it, if the quantity K i=1 L(i, f (x x x)) is bounded, it is still possible to obtain performance guarantees for robustness to label noise. This has been done for the q-cross entropy loss in [18,Theorem 1]. Inspired by that, we derive a similar result for the Fisher-Rao loss function.
Proof. We shall use Lagrange multipliers to find the critical points of F (p p p) and ∂G ∂p i (p p p) = 1, therefore critical points in the interior of the simplex ∆ K−1 are of the form ∂F is injective, this happens if, and only if, p 1 = p 2 = · · · = p K = 1 K , which leads to F (p p p) = On the other hand, if there are critical points on the boundary of ∆ K−1 , they will have a zero in some position, e.g., p p p = (0, p 2 , . . . , p K ). Repeating the same argument, the critical point will have the form p 1 = 0, p 2 = · · · = p K = 1 K−1 , so that F (p p p) = π 2 4 + (K − 1) arccos 1 By induction, any critical value of F has the form F (p p p) = (K −j) π 2 4 +j arccos 1 √ j 2 , for 1 ≤ j ≤ K. By comparing the values of F in all these points, we find maximum and minimum, respectively, for j = 1 and j = K, which yields the desired inequalities. and where and Proof. The inequalities involving 0 in (15) and (16) are straightforward consequences of the definitions of f * andf . We shall prove the inequalities involving A FR and B FR . For (15), as in the proof of [18, Theorem 1], we have that Using Lemma 2, we obtain and Subtracting these two inequalities, we get (16), we use the same technique, but isolating R L FR (f ) in (19) and (20), with f * and witĥ f swapping places, thus obtaining When the two inequalities are subtracted, we get This result provides an upper bound for the performance degradation of a classifier trained with the Fisher-Rao loss under uniform label noise, in terms of the noise rate η and the number of classes K. This shows that this type of noise can only have a limited impact in the performance of the trained classifier. Table 1 compares the analogous bounds for other loss functions. The results for the MSE loss  are derived in Appendix A. For the MAE loss, the bounds are zero, since it is proved to be robust to uniform label noise [16]. For the CE loss, no such bounds can be obtained, as this loss in unbounded, so its entries are marked as +∞ and −∞ on the table. The table also contains the result for the q-CE losses, which was taken from [18], and in particular for the Hellinger loss (q = 1/2).
To further investigate the behaviour of the bounds that depend on the noise rate η and the number of classes K, we write the noise rate as η = α 1 − 1 K , parametrised by α ∈ [0, 1). Varying this value, the noise rate goes from no label noise to the maximum allowed for a given number of classes K. Figure 1 illustrates how the bounds A(K, η) and B(K, η) vary with the value α for different loss functions and K = 10. For the q-CE loss, we choose the value q = 0.7, which has been used in the experiments in [18]. We see that, as α → 1, the values of A(K, η) tend each to a constant, while those of B(K, η) go to −∞.
To study the dependence on the number of classes K, we plot the bounds versus K, for η = 0.8 1 − 1 K , in Figure 2. We see that, for Fisher-Rao, Hellinger and q-cross entropy losses, after a peak, the bounds get tighter as the number of classes increases. Indeed, for the Fisher-Rao bounds, we have This means that the effect of the label noise gets 'diluted' as the number of classes grows, for any α ∈ [0, 1). Even though this effect may sound intuitive, this is not what happens with the bounds for the MSE loss, which tend to a constant value as K increases.

Learning Speed
Learning in neural networks involves minimising the empirical riskR L = 1 N N i=1 L(y i , f (x x x i )) by means of a numerical algorithm, such as gradient descent. The gradient vector ∇ ωRL = 1 N N i=1 ∇ ω L(y i , f (x x x i )) with respect to the network parameters ω gives the direction and magnitude of each updating step towards the local minimum ofR L . To investigate the learning speed obtained with each loss, we shall study the magnitude of this gradient vector.
An interesting feature of MAE, CE, q-CE, Fisher-Rao and Hellinger losses is that they can all be written as to only depend on the probability p y = σ(s y ) that the model assigns to the correct class. The MSE loss, on the other hand, depends on all coordinates of the vector p p p = (σ • f )(x x x), meaning that it tries to simultaneously make the predicted probability p y of the correct class close to 1 and to minimise the probability p i of all other classes i = y.
Specifically, the MAE, CE, q-CE Fisher-Rao and Hellinger losses can be written as functions of the form for some monotonically decreasing differentiable function h : [0, 1] → R with h(1) = 0. In this case, we have Since the term ∇ ω π y (σ • f )(x x x) does not depend on the loss function, it is enough to study the absolute value of the multiplicative factor |h (p y )| to compare the magnitude of the gradient for different losses. The particular functions h and their derivatives are given in Table 2 and plotted in  Table 2.  Figure 3. Note that this plot recovers the inequality chain obtained in Proposition 1, item 2. As it has been previously noted [17,18], the fact that the gradient of the MAE loss is constant is the main reason why this loss results in slow training, in spite of its robustness to label noise. On the other extreme, with the CE loss, the magnitude of |h (p y )| increases as p y decreases, meaning that the farther away from the local minimum (i.e., prediction is more wrong), the larger the steps in the gradient descent method will be. Also, the form of its derivative implies that more emphasis is put on samples whose predicted class is different from the assigned label (noisy or clean), which explains both the fast convergence speed and the weak tolerance to label noise.
The magnitude of |h (p y )| for the q-CE, Fisher-Rao and Hellinger loss functions similarly depend on the value of p y , indicating they have better training dynamics than the MAE loss. Interestingly, the order of losses with higher magnitude of |h (p y )|, suggesting faster convergence, is the opposite of that observed for robustness (CE, Fisher-Rao, Hellinger, MAE) in Figure 2. Thus, the Fisher-Rao and the Hellinger loss can be seen as providing a trade-off between learning dynamics and robustness against label noise. Particularly, these results indicate that the Fisher-Rao loss can provide a modest improvement in learning speed, at the cost of a modest reduction of robustness, when compared to the Hellinger loss.

Experimental Results
We conduct numerical experiments to study the performance of different loss functions, on a synthetic and a real dataset. The experiments use simple neural networks and are intended to illustrate the theoretical results derived in the previous sections and showcase the trade-off provided by the Fisher-Rao loss, rather than to generate state-of-the-art results. We compare the Fisher-Rao loss function with CE and Hellinger loss, which approach the former, cf. Prosposition 1, and also with the MSE loss, which is commonly used.
We assess the performances both on the original (noiseless) datasets and in the presence of uniform label noise with noise rate η. Only the training datasets are corrutpted with noise, while the testing datasets are clean. In all experiments the activation function is ReLU, and stochastic gradient descent is used for training the model. The learning rate for each loss was hand-tuned by grid search to provide the best performance.

Synthetic Dataset
First we consider a synthetic dataset formed by 8,000 training examples and 2,000 testing examples of 100-dimensional vectors divided into 10 classes. The data are generated by Gaussian distributions centred on the vertices of hyper-cube, using the method make_classification from scikit-learn [35]. We set up a multilayer perceptron (MLP) network with three hidden layers with 80, 40 and 20 neurons, respectively. Batch size is 20 and the model is trained for 20 epochs. The experiments were repeated five times, and we report the averages and standard deviations.
We run tests for noiseless data and with uniform label noise with η = 0.3 and η = 0.5. The evolution of the accuracy on the training (dashed lines) and testing (continuous lines) datasets are shown in Figure 4, where the mean results are represented by opaque lines and the shaded regions indicate the standard deviation. The final test accuracies are summarised in Table 3, where the two best accuracies are boldfaced. We note that, in the absence of noise, the final accuracy achieved by all losses is similar, with the cross entropy providing the fastest training; in the presence of uniform label noise, however, the Fisher-Rao and Hellinger losses consistently provide the highest test accuracies, as expected from the analysis in Section 3. Interestingly, they also provide faster learning than the CE loss-we conjecture this may be due to the fact that, when labels are noisy, the learning process becomes more difficult for the CE loss than for Fisher-Rao and Hellinger losses. These results emphasise that the latter two can be suitable for learning under label noise.

MNIST
The MNIST dataset [36] contains 60,000 training examples and 10,000 testing examples of 28×28 grey images of handwritten digits from 0 to 9. For this experiment, an MLP network is set up, consisting of two hidden layers with 300 and 100 neurons, respectively. We use batch size of 64 and train the model for 40 epochs. The experiments were repeated three times. Results for noiseless data and label noise with η = 0.3 and η = 0.5 are similarly reported in Figure 5 and Table 4, where the two best accuracies are boldfaced. In these experiments, Fisher-Rao achieves competitive performances with noiseless data. Again, in the presence of label noise, both Fisher-Rao and Hellinger show a slight advantage over MSE and CE losses in terms of learning speed and final accuracy, illustrating the benefits discussed in the previous section. Note that in Figures 5b and 5c, the training accuracies are not visible, as they are less than 90%.

Conclusion and Perspectives
In this work we have studied the Fisher-Rao loss function for learning in the context of supervised classification, especially in the presence of label noise. This loss comes from the Fisher metric on the statistical manifold of discrete distributions, and, for small values, is approached both by the Hellinger and the cross entropy loss functions. Theoretical analysis of the robutstness and learning  speed of this loss have been presented and compared with other commonly used losses. We argue that the Fisher-Rao loss provides a natural trade-off between these two features. In numerical experiments, we have observed that, for noiseless data, the performance of the Fisher-Rao loss is on par with other commonly used losses, while, in the presence of label noise, we observe the expected robustness.
While we have focused on the classification problem, one could ask whether it is possible to use similar Fisher-Rao loss functions to regression problems as well. The first requirement to do so is that the model (e.g., neural network) estimates a (parametric) conditional distribution p(y|x x x) of the output y ∈ R, given the input x x x ∈ R n , instead of a simple predicted valueŷ. For instance, this is done for normal distributions in [37], and for mixture models in [1, § 5.6]. The second requirement is a closed-form expression for the Fisher-Rao distance in the corresponding statistical manifold. Finally, it is required to define a target distribution from the training examples, to be compared with the current predicted distribution, in such a way that the distance between them is well defined. The consequences of such an approach remain yet to be studied and constitute a possible future direction of research.
Proof. From (6), we have L MSE (y, f (x x x)) = 1 + p p p 2 2 − 2p y , so that 1 + p p p 2 2 − 2p i = K 1 + p p p 2 2 − 2 Furthermore, F (p p p) := p p p 2 2 = K i=1 p 2 i is a convex function in p p p = (p 1 , . . . , p K ), and, with p p p restricted to the simplex ∆ K−1 , we have 1 K ≤ p p p 2 2 ≤ 1. Thus Proposition 5. For the mean squared error loss function, under uniform label noise with η < K−1 K , and Proof. The proof is analogous to that of Proposition 3, so we present a concise version. We have Using Lemma 4, we obtain Finally, using that , and that η < K−1 K yields the desired results.