{'fraction_non_alphanumeric': 0.04176017172407064, 'fraction_numerical': 0.013172016782125085, 'mean_word_length': 4.525606469002695, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Contour Plots of Objective Functions for Feed-Forward Neural Networks

Error surfaces provide us with very important information for training of feed-forward neural networks (FNNs). In this paper, we draw the contour plots of various error or objective functions for training of FNNs. Firstly, when applying FNNs to classifications, the weakness of mean-squared error is explained with the viewpoint of error contour plot. And the classification figure of merit, mean log-square error, cross-entropy error, and n-th order extension of cross-entropy error objective functions are considered for the contour plots. Also, the recently proposed target node method is explained with the viewpoint of contour plot. Based on the contour plots, we can explain characteristics of various error or objective functions when training of FNNs proceeds.


INTRODUCTION
Feed-forward neural networks (FNNs) can approximate any function with enough number of hidden nodes and this characteristic supports applications of FNNs to many fields [1]. Although there have been theoretical proofs for the capability of FNNs [1]- [4], how to train FNNs is still a challenging problem and various objective functions have been proposed to improve the training of FNNs [5]- [11]. These functions have peculiar properties and they can be compared various ways. The easiest way is to train FNNs using the objective functions for some application problems and to compare their performances. However, this performance comparison with simulation results is heuristic and problem-dependent.
Statistical comparisons are more concrete and there have been many researches of statistically analyzing error functions [12]- [15]. Firstly, there were analytic results that FNNs are Bayesian optimal classifiers if FNNs are trained with infinite number of training samples [12], [13]. Also, the optimal outputs of FNNs have been derived as a function of a posteriori probability that an input sample belongs to a certain class [12]- [14]. Furthermore, the relationship between two optimal outputs has been derived for two-class imbalanced data problems [15]. All these analytic results are functions of one argument and can be plotted as curves [13]- [15]. performances [16]. If FNNs are in a flat region of the error objective functions with some high values, error decreases very slowly during training and it will take a very long time to escape that region. This period dominates learning time [17]. Thus, the contour of objective functions is very important for training of FNNs. In this paper, we draw contour plots of objective functions which are proposed to improve performances of FNNs. In section 2, we briefly introduce the various objective functions. For two dimensional plots of the contour, we assume that FNNs are with two output nodes. The drawings of contours for the objective functions are in section 3 with discussions. And section 4 concludes this paper.

OBJECTIVE FUNCTIONS
Consider a FNN with M output nodes whose activation functions are the bi-modal sigmoid. The traditional meansquared error (MSE) objective function is defined by [5] ( ).
Here, k y denotes the k-th output node of FNN and k t is its desired value. In classification problems, 1 ± = k t where 1 is the desired value for a correct class node and -1 is for the other nodes. That is, if input samples are in the class c, Parameters of FNNs are updated to decrease the error function during training [5]. For performance improvement of MSE, additive noise in the desired value is adopted as [6] ( ), where k n is the zero-mean white noise with variance 2 σ .
The classification figure of merit (CFM) objective function has three essential features that distinguish it from the traditional MSE [7]. Firstly, it has not the desired value but the index of output node representing the correct classification outcome. Secondly, CFM yields decreasing marginal "rewards" for increasingly ideal output pattern. Thirdly, CFM yields decreasing marginal "penalties" for increasingly bad misclassifications. This is to discourage the FNN from attempting to learn outliers heavily.
The resulting CFM objective function compares the output node value that should be at high state with values of all output nodes that should be at low state. It then goes through a sigmoidal function to each of these differences. Using CFM, learning focuses most heavily on the reduction of misclassifications rather than reducing the difference between the output node value and its target output. Thus, CFM is defined by [7] ( where c y denotes the correct class node and Let's consider a training of FNN based on MSE. When outliers are presented to the FNN, the difference between the output node value and its desired value is very large and this causes a heavy updating of parameters of FNN. Thus, outliers disturb learning for whole training samples and degrade the performance of FNNs. In order to suppress the huge updating of parameters by outliers, the mean log-square (MLS) error objective function is defined by [8] ( ) In pattern recognition applications, the desired value of FNN is one of the two extreme values of sigmoidal activation function. If any output node value is near the wrong extreme value, we say the node is "incorrectly saturated"[10] [17]. When an output node is incorrectly saturated, the gradient of MSE is small and this causes slow learning convergence [14]. In order to resolve this problem, the cross-entropy (CE) error objective function is defined by [9] ( ) ( However, CE suffers from overspecialization for training samples since the gradient of CE for correctly saturated node is too strong. For improvement of CE, the n-th order extension of CE (nCE) error objective function is defined as [10] ( ) where n is a natural number.
Also, nCE is modified to attack imbalanced data problems, which is one of recent challenging problems [11]. Assume that the FNN has two output nodes in order to handle two-class imbalanced data problems. Class 1 denotes the minority class whose desired values are where n and m (n<m) are natural numbers.

CONTOUR PLOTS
In this section, we draw contours of the various objective functions introduced in section 2. For contour plots in the twodimensional space, we assume that FNNs have two output nodes (M=2) whose desired values are This means that an input sample belongs to the class 1. In this case, using the max rule for classification, 1 y should be greater than 2 y for correct classification. That is, the region 2 1 y y > corresponds to a correct classification referred to as a "hit" [7]. On the contrary, the region 2 1 y y < corresponds to an incorrect classification referred to as a "miss". This is shown in Fig. 1. Here, the straight line with slope 1 is the border line.     (10) and its contour plot is in Fig. 3 (-1,1), the gradient has a very gentle slope and it will take very long time to move from point (-1,1) to the region near the border line. This slow convergence of learning is the weakness of CFM.
Applying the same assumption of Eq. (8) to Eq. (4), MLS is given by which is derived from Eq. (5) using the assumption Eq. (8). Fig.  5 shows the contour of Eq. (12) that has steep slope in the "miss" region especially near the point (-1,1). Thus, FNN can escape the "miss" region rapidly during training based on CE E . However, the gradient near the point (1,-1) has the effect of overspecialization for training samples [10].
The contour of nCE with n=2 is derived by substituting Eq. Using we can get Here, 1 1 y u + = and 2 1 y v − = .
(16) Fig. 6 shows the contour of Eq. (15), which has steeper slope in the "miss" region. This has the acceleration effect of learning in the "miss" region. Also, the gentle slope in the "hit" region can prevent the overspecialization of learning. This gentle slope characteristic in the "hit" region is in the contour of CFM, too. Here, u and v are defined by Eq. (16) and the contour of nCE with n=4 is in Fig. 7. By comparing Fig. 7 with Fig. 6, we can find that increasing of n in nCE has effects of steeper gradient in the "miss" region and gentler gradient in the "hit" region. Thus, nCE accelerates the training of FNN by the steep gradient in the "miss" region and prevents the overspecialization to training samples by the gentle gradient in the "hit" region.  By using Eqs. (14) and (18), (21) Here, u and v are the same with Eq. (16). The contour of Eq. (21) is plotted in Fig. 8. The gradient along to the horizontal direction from (-1,-1) to (1,-1) is gentler than that along to the vertical direction from (1,1) to (-1,1). This means that the error component of 1 y (the first term in Eq. (20)) is greater than the error component of 2 y (the second term in Eq. (20)). Therefore, the target node method strengthens updating amount of parameters related to the minority class target node 1 y . This can prevent the boundary distortion from the majority class to the minority class [11].
Experimental comparisons of the objective functions for multi-class problems can be found in [7], [9], and [10]. Also, the comparisons for imbalanced two-class problems are in [11]. These experimental results can be complementary with the contour analysis for elucidating the characteristics of objective functions.

CONCLUSIONS
In this paper, we investigate the contours of various objective functions which were proposed to improve performances of FNNs. Decreasing of MSE objective function cannot guarantee the better classification performance. CFM shows steeper gradient near the border line between 'hit' and 'miss' regions. Although updating parameters to the direction of increasing CFM coincides the increasing of classification ratio, CFM shows slow convergence of learning due to the very little gradient in the 'miss' region. MLS shows straight contour lines in the "miss" region. CE shows stepper gradient in the "miss" region than MSE and CFM. In nCE, increasing n makes steeper gradient in the "miss" region for acceleration of learning and gentler gradient in the "hit" region for preventing overspecialization to training samples. In the target node method, learning related to the minority class is strengthened for preventing boundary distortion due to imbalanced data.