{'fraction_non_alphanumeric': 0.02515496612368459, 'fraction_numerical': 0.005766181346403345, 'mean_word_length': 4.771422628951747, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Comprehensive Survey of Regression Based Loss Functions for Time Series Forecasting

Time Series Forecasting has been an active area of research due to its many applications ranging from network usage prediction, resource allocation, anomaly detection, and predictive maintenance. Numerous publications published in the last five years have proposed diverse sets of objective loss functions to address cases such as biased data, long-term forecasting, multicollinear features, etc. In this paper, we have summarized 14 well-known regression loss functions commonly used for time series forecasting and listed out the circumstances where their application can aid in faster and better model convergence. We have also demonstrated how certain categories of loss functions perform well across all data sets and can be considered as a baseline objective function in circumstances where the distribution of the data is unknown. Our code is available at GitHub: https://github.com/aryan-jadon/Regression-Loss-Functions-in-Time-Series-Forecasting-Tensorflow.


I. INTRODUCTION
A Time Series Forecasting approach involves making predictions using historical time-stamped data. It entails analyzing historical data, constructing models, and applying learned patterns and findings to influence future strategic decisions. In past, time series forecasting was dominated by linear methods because of their model interpretability, explainability, and efficiency for a wide range of simpler forecasting problems. However, In many of the latest applications, time series forecasting uses technologies like Machine Learning, Deep Learning, Gaussian Processes, and Artificial Neural Networks. The choice of the loss function also known as objective function is extremely important while working on large-scale time series forecasting problems as they instigate the learning process of the algorithm. Therefore, since 2012, researchers have experimented with various specific loss functions to improve results for their datasets. In this paper, we have summarized such widely used 14 loss functions that have been proven to provide state-of-the-art results in different domains.
We have also explored the conditions to determine which loss function might be helpful in a scenario and compared the performance of all loss functions on four different datasets using Temporal Fusion Transformer [1] for forecasting. Datasets used in experiments are Electricity Load Diagrams 2011-2014 Dataset [2], which has electricity consumption data points of 370 points/clients, PEMS-SF Dataset [3], which has 15 months worth of daily data that describes the occupancy rate of different car lanes of the San Francisco bay area freeways across time, Corporación Favorita Dataset [4] which has Grocery Sales Data and Volatility Dataset [5] which has daily non-parametric measures of how volatility financial assets or indexes were in the past. The code implementation of experiments is available at https://github.com/aryan-jadon/Regression-Loss-Functionsin-Time-Series-Forecasting-Tensorflow In general, Time series forecasting problems can be converted and worked into a regression-type predictive modeling problem. However, the above mentioned components of a time series data make the forecasting problem more complex for simple linear methods. Therefore, an ideal model and a good objective function should be able to learn trend and seasonality of data, while avoiding random noise. With the ease of computation power, the model choice can be varied as much the complexity of data requires, but there is still some uncertainty about the choice of loss functions. In the next section, we have detailed about 14 widely used loss function for forecasting problem. We aim to help industry professionals, as well as researchers by guiding them to correct objective function/loss function which can reduce the set of experiments, converge the model faster, and overall help researchers lead their research in better direction.
The rest of the paper is outlined as follows, Section II provides explanation about time series data, and Section III describes the regression based loss functions in detail. Section IV reports the experiments and results. Section V discusses the limitations of this approach and concludes the paper.

II. TIME SERIES DATA
Regression modeling is a form of predictive modeling technique used to estimate the relationship between two or more variables. It is a supervised learning technique that can be defined as statistical techniques used to model the relationship between the dependent real numbered variable y and the independent variables x i . Time series data varies a bit different from general regression based data, as there is a temporal information added to the features making the objective a bit more complex. A Time Series data have the following components -1) Level -Every time series has a baseline. These baselines are called levels; to these levels we add different components to form a complete time series. 2) Cyclicity -Time series data also has a pattern called cyclicity which repeats aperiodically, meaning it does not occur at the same fixed interval. 3) Trend -It indicates whether a time series is increasing or decreasing over a period of time. That is, it has an upward (increasing) or downward (decreasing) trend. 4) Seasonality -Patterns that repeat over a period of time are called seasonality. 5) Noise -After extracting level, cyclicity, trend, and seasonality, what is left is noise. Noise is an entirely random variation in data. The fundamental goal of every classical machine learning model is to improve the model's selected metrics and reduce the loss associated with it. A vital component of a machine learning or deep learning model used for time series forecasting is the loss function, a model's performance is measured against it, and the parameters it chooses to learn will be based on the minimization of function. Absolute Error, also known as L1 loss, is the absolute difference between a predicted value and the actual value.

III. REGRESSION LOSS FUNCTIONS
The aggregation of all loss values is called the cost function, where the cost function for Absolute Error is known as Mean Absolute Error.
Here, N is the number of data samples, y i is the true value, andŷ i is the predicted value.
Mean Absolute Error is a simple yet robust loss function used in regression models. Because of outliers, variables in regression problems may not be strictly Gaussian [6]. Refer to table I to understand in detail the advantages and disadvantages of using MAE loss.

B. Mean Squared Error (MSE)
Squared Error Loss, also known as L2 loss, is the squared difference between the prediction and actual values.
The cost function for Squared Error is called the Mean of Squared Error. When it comes to regression loss functions, practically every data scientist prefers Mean Squared Error.
Here, N is the number of data samples, y i is the true value, andŷ i is the predicted value.
MSE is also referred to as a quadratic loss because the penalty is squared rather than directly proportional to the error. The outliers are given more weight when the error is squared, creating a smooth gradient for minor errors. Optimization algorithms benefit from this penalization for huge errors as it helps obtain the optimum values for parameters. Given that the mistakes are squared, MSE can never be negative, and the error's value can be anywhere between 0 and infinity. With increasing mistakes, MSE grows exponentially, and the MSE value of a good model will be close to zero [7]. Refer to Table  II to learn in detail about the advantages and disadvantages of using MSE loss.  Advantages Disadvantages MSE aids in the efficient convergence to minima for tiny mistakes as the gradient gradually decreases.
Squaring the values accelerates the rate of training, but a higher loss value may result in a substantial leap during back propagation, which is undesirable. MSE values are expressed in quadratic equations, aids to penalizing model in case of outliers.
MSE is especially sensitive to outliers, which means that significant outliers in data may influence our model performance.

C. Mean Bias Error (MBE)
A measuring procedure propensity to overestimate or underestimate the value of a parameter is known as bias or Mean Bias Error. The only possible direction for bias is either a positive or negative one. A positive bias indicates that the data error is overstated, whereas a negative bias indicates that the error is underestimated.
The discrepancy between the actual and anticipated values is measured as the mean bias error (MBE). The average bias in the prediction is captured and quantified by MBE. Except for not taking into account absolute values, it is virtually identical to MAE. MBE should be treated cautiously because positive and negative faults may cancel one another out [8].
Here, N is the number of data samples, y i is the true value, andŷ i is the predicted value.
Refer to Table III to learn in detail about the advantages and disadvantages of using MBE loss.  Advantages Disadvantages If you wish to identify and correct model bias, you should use MBE to determine the direction of the model (i.e., whether it is biased positively or negatively).
MBE tends to err in one direction continuously, while attempting to anticipate traffic patterns. Given that the errors tend to cancel each other out, it is not a suitable loss function for numbers ranging from (−∞, ∞) .

D. Relative Absolute Error (RAE)
The calculation of relative absolute error involves taking the total absolute error and dividing it by the absolute difference between the mean and the actual value.
RAE is expressed as, Here, N is the number of data samples, y i is the true value, y i is the predicted value, andȳ is the mean of N actual values.
RAE is a ratio-based metric that assesses the efficacy of a predictive model. RAE has a possible value between 0 and 1. Values near zero, with zero being the best value, are characteristics of a good model. This error demonstrates how the target function's mean deviation from its mean and the mean residual relate to one another [9].
Refer to Table IV to learn in detail about the advantages and disadvantages of using RAE loss.

Advantages
Disadvantages RAE can compare models where errors are measured in different units.
If the reference forecast is equal to the ground truth, RAE can become undefinable, which is one of its key drawbacks.

E. Relative Squared Error (RSE)
The relative squared error (RSE) measures how inaccurate the results would have been without a simple predictor. This straightforward predictor merely represents the average of the actual values. As a result, the relative squared error divides the total squared error by the total squared error of the simple predictor to normalize it. It is possible to compare it among models whose errors are calculated in various units [10].
The equation determines the relative squared error- Here, N is the number of data samples, y i is the true value, y i is the predicted value, andȳ is the mean of N actual values.
Refer to Table V to learn in detail about the advantages and disadvantages of using RSE loss.

F. Mean Absolute Percentage Error (MAPE)
The mean absolute percentage error (MAPE), also known as the mean absolute percentage deviation (MAPD), is a metric used to assess the accuracy of a forecast system. It calculates the average absolute percent error as a percentage for each time period by subtracting actual values from actual values divided by actual values. Absolute errors, regardless of sign, are used  to calculate percentage errors, and positive and negative errors are prevented from canceling one other out.
Because the variable's units are scaled to percent units, the mean absolute percentage error (MAPE) is widely used for forecasting errors. When there are no outliers in the data, it works well and is often used in regression analysis and model evaluation [11].
Here, N is the number of data samples, y i is the true value, andŷ i is the predicted value. Refer to Table VI to learn in detail about the advantages and disadvantages of using MAPE loss.

G. Root Mean Squared Error (RMSE)
The square root of MSE is used to calculate RMSE. The Root Mean Square Deviation is another name for RMSE. It accounts for variations from the actual value and measures the average magnitude of the errors. RMSE can be applied to a variety of features because it aids in determining whether a feature enhances model prediction or not.
Here, N is the number of data samples, y i is the true value, andŷ i is the predicted value.
RMSE is most useful when huge errors are incredibly undesirable. According to competition regulations, submissions  Advantages Disadvantages MAPE Loss is computed by standardizing all errors on a single scale of hundred.
As denominator of the MAPE equation is the predicted output, which can be zero leading to undefined value. As the error estimates are expressed in percentages, MAPE is independent of the scale of the variables. The issue of positive numbers canceling out negative ones is avoided since MAPE utilizes absolute percentage mistakes.
Positive errors are penalized less by MAPE than negative ones. Therefore, it is biased when we compare the precision of prediction algorithms since it defaults to selecting one whose results are too low.
are judged based on the logarithm of the anticipated value and the logarithm of the observed sales price. Using logs means that errors in estimating expensive and inexpensive dwellings will impact the outcome.
The RMSE, which offers the average model prediction error in units of the variable of interest, indicates the model's absolute fit to the data. The model's performance and predictions improve with decreasing RMSE, and a greater RMSE denotes a significant departure from the ground truth in the residual [12].
Refer to Table VII to learn in detail about the advantages and disadvantages of using RMSE loss.

H. Mean Squared Logarithmic Error (MSLE)
The mean squared logarithmic error (MSLE) measures the difference between actual and expected values.
Adding the logarithm has reduced MSLE concern for the percentual difference between the actual and forecasted values and the relative difference between the two. Accordingly, MSLE will roughly treat tiny discrepancies between small actual and anticipated values and huge disparities between large true and forecasted values [13].

Advantages
Disadvantages RMSE works as a training heuristic for models. Many optimization methods choose it because it is easily differentiable and computationally straightforward.
As RMSE is still a linear scoring function, the gradient is abrupt around minima.
Even with larger values, there are fewer extreme losses, and the square root causes RMSE to penalize errors less than MSE.
The scale of data determines the RMSE, as the errors' magnitude grows, so does sensitivity to outliers. In order to converge the model the sensitivity must be reduced, leading to extra overhead to use RMSE.
The loss is the mean over the observed data of the squared differences between the true and predicted values after log transformation, or you can write it as a formula.
Here, N is the number of data samples, y i is the true value, andŷ i is the predicted value.
This loss can be interpreted as a measure of the ratio between the true and predicted values since: Log(0) is not defined, but both y and can be defined mathematically. Therefore one is appended to both y and for mathematical convenience.
Refer to Table VIII to learn in detail about the advantages and disadvantages of using MSLE loss.  Advantages Disadvantages Treats small differences between small actual and predicted values the same as big differences between large actual and predicted values.

I. Root Mean Squared Logarithmic Error (RMSLE)
The root mean squared logarithmic error is determined by applying log to the actual and predicted numbers and then subtracting them. RMSLE is resistant to outliers when both minor and large errors are considered.
Here, N is the number of data samples, y i is the true value, andŷ i is the predicted value.
If the predicted value is less than the actual value, it penalizes the model more; if it is greater than the actual value, it penalizes the model less. High errors brought on by the log are not punished. As a result, the model has a far higher penalty for underestimating than for overestimating. This is useful in circumstances where we do not mind overestimation but underestimating is unacceptable [14].
Refer to Table IX to learn in detail about the advantages and disadvantages of using RMSLE loss.

J. Normalized Root Mean Squared Error (NRMSE)
The Normalized Root Mean Square Error (NRMSE) the RMSE facilitates the comparison between models with different scales. The variable has the observed range's normalized RMSE (NRMSE), which connects the RMSE to the observed range. As a result, it is possible to interpret the NRMSE as a

Advantages
Disadvantages RMSLE is applicable at several scales and is not scaledependent. It is unaffected by significant outliers. Only the relative error between the actual value and the anticipated value is taken into account.
Due to RMSLE's biased penalty, underestimating is penalized more severely than overestimation.
portion of the whole range that the model resolves typically [15].
Here, N is the number of data samples, y i is the true value and,ŷ i is the predicted value,ō is the average of the observation value.
Refer to Table X to learn in detail about the advantages and disadvantages of using NRMSE loss.

Advantages
Disadvantages NRMSE overcomes the scale dependency and eases comparison between models of different scales or datasets.
NRMSE loses the units associated with the response variable.

K. Relative Root Mean Squared Error (RRMSE)
RRMSE is an RMSE variant without dimensions. Relative Root Mean Square Error (RRMSE) is a root mean square error measure that has been scaled against the actual value and then normalized by the root mean square value. While the original measurements' scale constraints RMSE, RRMSE can be used to compare various measurement approaches. An enhanced RRMSE happens when your predictions turn out to be wrong, and the error is expressed by RRMSE either relatively or as a percentage [16].
RRMSE Model accuracy is excellent when the model score is below 10%, good when the model score is between 10% to 20%, fair when the model score is between 20% to 30%, and poor when the model score is greater than 30% RRMSE can be expressed as Here, N is the number of data samples, y i is the true value, andŷ i is the predicted value.
Refer to Table XI to learn in detail about the advantages and disadvantages of using RRMSE loss.

L. Huber Loss
Huber loss is an ideal combination of quadratic and linear scoring algorithms. There is also the hyperparameter delta (δ). Loss is linear for values more than delta; for values less than delta, the loss is quadratic. This parameter distinguishes the Huber loss and is changeable based on the data.
Here, N is the number of data samples, y i is the true value, y i is the predicted value, and (δ) is the hyperparameter delta.
This equation effectively states that for loss values less than the delta, the MSE should be used; for loss values greater than the delta, the MAE should be used. This successfully combines the greatest features of both loss functions.
Using the MAE for more significant loss numbers reduces the weight placed on outliers while still producing a wellrounded model. Simultaneously, the MSE is employed for the lesser loss values to keep a quadratic function around the center [17].
Refer to Table XII to learn in detail about the advantages and disadvantages of using Huber loss.

M. LogCosh Loss
Log cosh calculates the logarithm of the hyperbolic cosine of the error. More smoothly than quadratic loss, this function. It functions similarly to MSE but is unaffected by significant prediction errors. Given that it uses linear and quadratic scoring techniques, it is extremely close to Huber loss.

Advantages
Disadvantages Linearity above the delta guarantees that outliers are given appropriate weightage (Not as extreme as in MSE). Addition of hyper parameter delta (δ) allows flexibility to adapt to any distribution Huber loss is computationally expensive due to the additional conditionals and comparisons, especially if your dataset is huge.
The curved form below the delta guarantees that the steps are the correct length during backpropagation.
To achieve the best outcomes must be optimized, which raises training requirements.
Here, N is the number of data samples, y i is the true value, andŷ i is the predicted value.
It has the distinction of being dual differentiable. Some optimization approaches, such as XGBoost, favor such functions over Huber functions, which can only be differentiated once. Log-Cosh computes the log of the hyperbolic cosine of the mistake [18].
Refer to Table XIII to learn in detail about the advantages and disadvantages of using LogCosh loss.

N. Quantile Loss
The Quantile regression loss function is used to forecast quantiles. The quantile is the value that indicates how many values in the group fall below or above a particular threshold. It calculates the conditional median or quantile of the response (dependent) variables across values of the predictor (independent) variables. Except for the 50th percentile, where it is MAE, the loss function is an extension of MAE.
It makes no assumptions about the response's parametric distribution and offers prediction intervals even for residuals with non-constant variance. Advantages Disadvantages As Log-cosh calculates the log of hyperbolic cosine of the error. Therefore, it has a considerable advantage over Huber loss for its' property of continuity and differentiability.
It is less adaptable than Huber since it operates on a fixed scale (no δ).
There are fewer computations required compared with Huber.
The derivation is more challenging than Huber loss and necessitates more research.
Here, γ represents the required quantile, y i is the true value, andŷ i is the predicted value. The quantile values are chosen based on how we wish to balance the positive and the negative errors.
The value of the loss function above ranges from 0 to 1. The first portion of the formula will predominate in cases of underestimation, whereas the second component will predominate in cases of overestimation. Different penalties for over-and under-prediction are given depending on the selected quantile value.
Underestimation and overestimation are penalized by the same factor when = 0.5, and the median is then determined. Overestimation is punished more severely than underestimating when the value is greater. For instance, the model penalizes overestimation when = 0.75, costing three times as much as underestimating. Gradient descent-based optimization techniques learn from the quantiles rather than the mean [19].
Refer to Table XV to learn in detail about the advantages and disadvantages of using Quantile loss.  Huber Loss curves around the minima, which decreases the gradient and is more robust to outliers while optimally penalizing the incorrect values.

LogCosh Loss
Logcosh works similarly to mean squared error but is less affected by the occasional wildly incorrect prediction. It has all of the benefits of Huber loss; unlike Huber loss, It is twice differentiable everywhere. Quantile Loss Use Quantile Loss when predicting an interval instead of point estimates. Quantile Loss can also be used to calculate prediction intervals in neural nets and tree-based models, and It is robust to outliers. Advantages Disadvantages It is robust to outliers.
Quantile loss is computationally intensive. It is beneficial for making interval predictions as opposed to point estimates. This function can also be used in neural networks and tree-based models to determine prediction intervals.
Quantile loss will be worse if we estimate the mean or use a squared loss to quantify efficiency.

A. Datasets
• Electricity Load Dataset -Dataset contains electricity consumption of 370 points/clients. • PEMS-SF Dataset -Dataset contains 15 months worth of daily data (440 daily records) that describes the occu-pancy rate, between 0 and 1, of different car lanes of the San Francisco bay area freeways across time. • Corporación Favorita Dataset -Grocery dataset containing dates, store and item information, whether that item was being promoted, and the unit sales. • Volatility Dataset -Dataset contains daily nonparametric measures of how volatility financial assets or indexes were in the past. Decomposition is mainly used for time series analysis, and as a tool for analysis, it can be utilized to guide forecasting models for your issue.

B. Performance Metrics
For this experiment setup, we evaluated the performance of all loss functions using P10, P50, and P90 metrics.
An upper and lower bound for forecasts can be provided via quantiles. An 80% confidence interval is a range of values that can be obtained, for instance, by utilizing the forecast types 0.1 (P10) and 0.9 (P90). In 10% of cases, the observed value should be less than the P10 value, and in 90% of cases, the P90 value should be higher. You can anticipate that the actual    value will fall inside those ranges 80% of the time by creating forecasts at p10 and P90. The shaded area between P10 and P90 in the following figure represents this range of values. Fig. 19. Illustration of Metrics used for evaluation of model performance.
• P10 (0.1) -The true value is expected to be lower than the predicted value 10% of the time. • P50 (0.5) -The true value is expected to be lower than the predicted value 50% of the time. This is also known as the median forecast.
• P90 (0.9) -The true value is expected to be lower than the predicted value 90% of the time. We have implemented Temporal Fusion Transformers [1] architecture and performed experiments using parameters in XVII. We have used 1000 training and 100 validation samples and performed experiments using 14 loss functions. Other loss functions were either resolved into our chosen loss function or were not fit for the listed datasets.

V. CONCLUSION
Loss functions play a critical role in determining a good fit model given the objective. Its' not possible to determine a universal loss function for complex objectives such as time series forecasting. There are a lot of factors such as outliers, skewness of data distribution, ML model requirements, com-  putational requirements, and performance requirements. There is no single loss function that is applicable to all types of data. In academic setting, which focuses mostly on model architecture and data type, loss function can be determined by the data-set properties used for training, such as distribution, boundaries, etc. In this work, we have also attempted to structure the cases for which a particular loss function can be useful, such as in case of outliers in the datasets, Mean Squared Error is the best strategy; however if there are fewer outliers Mean Absolute Error would be a better choice than MSE. Similarly, Using LogCosh is the better approach if we want to keep it balance and if our objective is based on percentile Quantile loss would be a good choice.
In this study, we have summarized 14 well-known loss functions for time series forecasting and developed a tractable form of the loss function for improved and more accurate optimization. In future, we will use this work as a baseline implementation for time series forecasting applications.