{'fraction_non_alphanumeric': 0.022376599634369287, 'fraction_numerical': 0.005338208409506399, 'mean_word_length': 4.925476603119584, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Study on the Deep Learning based Prediction of Production Demand by using LSTM under the State of Data Sparsity

Rapid market condition changes and growing uncertainties require accurate product demand forecasting to reduce the risks of business operations. So far, recurrent neural networks (RNNs) have been introduced as the representative models for forecasting time series data. However, it showed limited performance when learning time pattern occurring over a long period of time. This study performs deep learning-based production forecasting analysis with Long-term short-term memory (LSTM) and dropout algorithm as the alternative for the conventional time series analysis methods and the exponential smoothing method. Our result provides the reference for sales target setting, facility investment and production planning, inventory control, supply chain management, and marketing strategy establishment.


Introduction
Accurate product demand forecasting enables companies to strategically implement the key elements of overall corporate management, such as supply chain management (SCM), facility investment planning, and maintenance of appropriate inventory levels [1].
Recently, recurrent neural networks (RNNs) have been introduced as models for forecasting time series data such as written language and speech sound, and product demand. However, conventional RNN model showed limited performance when learning the specific patterns in time series data occurring over a long period of time. To overcome this problem, long short-term memory (LSTM) has been proposed. The concept of memory introduced by LSTM enables patterns recognition in time series data over a long period, which has led to improvement in the RNN performance.
However, prediction using LSTM based deep learning model has hardly been successful so far, either. This is due to the fact that time-series data for event forecasting of interest generally has a sparsity problem. For example, for data occurring once a year, there exist only five samples for five years. In addition, the demand patterns five years ago are highly likely to differ from the present demand patterns. So, the representativeness of the sample also decreases.
Meanwhile, various learning methods have been proposed along with LSTM in order to increase the forecasting ability. Among the algorithms, the most commonly used one is dropout. The dropout rules out some nodes in a deep network with the starting probability (for example, 0.6) during the learning process. Dropped-out nodes are randomly selected each time. The output value for the current connection strength (weight) is calculated with dropped-out nodes. Once completing learning, the output value for new data is calculated by multiplying the output values of all nodes by 1 -previous probability without omitting the nodes. This approach improves the stability and accuracy of learning [2]. In this idea, this study attempts to apply the LSTM with dropout method with the aim of verifying its applicability on the demand forecasting. The result of this study shows how dropout actually works and how corporations utilize the experimental results through demonstration.

Changes in the methods of time series analysis
Deep neural networks (DNNs) use more intermediate layers than conventional neural networks to improve the capacity to represent data. Currently, representative models of deep learning include convolutional neural network (CNN) models and recurrent neural network models, and recurrent neural networks are known to be suitable for learning and reasoning for time series data.
A recurrent neural network (RNN) is a neural network structure used for modelling sequence data. The basic idea underlying the RNN model is that since each data after the present data in the sequence provides new information, the information is used to update the present state of the model, When a sentence of a text is read, the data of the present state is updated with each new word, and this state is dependent on the previous words as well as the newly introduced word.
RNN models are based on the concept of a chain structure, and there are various types of RNN models depending on how data are maintained and updated. As the word 'recurrent' in the term suggests, RNN is composed of a kind of 'loops.' Although RNN processes the current input data in connection with the previous input data, it has a limit in remembering the input data that was processed very long time ago. According to Bengio et al. [1994], when learning for long sequences of input data is performed using BPTT (Backpropagation Through Time) in a RNN model, the vanishing gradient problem or the exploding gradient problem may occur. These problems occur because the gradient value vanishes or increases exponentially because of the recurrent weights that are repeatedly multiplied during the process of learning of even the data of the distant past.
The methods to solve the vanishing gradient problem, extended RNN models such as Long Short-Term Memory (LSTM) or gated recurrent units (GRU) have been proposed as fundamental solutions. LSTM was first introduced in 1997 and is now one of the most widely used models in natural language processing. GRU, which was proposed in 2014, is a simplified version of LSTM. Both models were introduced to solve the vanishing gradient problem and are known to handle long sequence data effectively.

Long Short-Term Memory (LSTM)
Gradient descent, which is a learning method of RNN, uses the change of gradient. One strength of RNN is that it connects the present task to previous information. However, RNN has a limitation in considering long-term dependency of data due to the vanishing gradient problem, which refers to the loss of major information as it goes through several time steps when processing long sequences in the learning process. Therefore, studies on LSTM have been conducted to solve the long-term dependency problem of RNN.  Figure. 1. The core of LSTM is a continuous cell state coming through the gate. The continuous cell state is also called the conveyor belt. The information coming through the conveyor belt is transmitted without any changes. LSTM can add or delete information through the input gate, forget gate, and output gate. In short, the gates perform the role of selectively transmitting data and thereby remove previous data to continue learning. LSTM performs calculation using gate vectors.
ƒt is the forget gate vector and serves as the weight to remember the previous cell state, and it is the input gate vector and serves as the weight to acquire new information. On the other hand, ot is the output gate vector, and performs the role of selecting output candidates. ⅹt is the input vector and, ht is the output vector. ct is the cell state vector, W and U are parameter matrices, and b is a vector, respectively. ƒt, it, ott are gate vectors. Two types of activation functions are used in LSTM: σ signifies a sigmoid function, and tanh represents the hyperbolic tangent function [4]. Among various algorithms that enhance the learning capabilities of deep networks, dropout is the algorithm that has received the most attention and is currently commonly used.

Recurrent neural network and dropout
In the initial version of dropout proposed for RNN and LSTM, recurrent connections are maintained as they are and dropout is applied only to non-recurrent connections (Figure 2 and Figure  3). This method is to omit some part of connection lines instead of dropping out nodes as in the conventional dropout. As a result, this technique led to improved performance in tasks such as handwriting recognition, but showed poorer performance in phoneme recognition, compared to other normalization techniques such as weight noise injection [2].
Dropout for RNN and LSTM, that is, rnnDrop distinguishes it from existing dropout algorithms for RNNs is to apply dropout not to connections but to nodes themselves. RnnDrop is also different from existing dropout methods for RNNs in that rnnDrop does not generate a new dropout mask for each time frame, but a dropout mask is generated once for the entire sequence and the same mask is applied to all time frames [2]. This method is more natural than conventional dropout methods for RNNs in that one dropout mask is used for one sequence, just as one random dropout mask is used for one image. Moreover, in order to train an RNN model, the recurrent layer is generally unfolded to apply backpropagation (backpropagation through time or BPTT), as shown in Figure 4, and at this time, unlike other dropout methods, rnnDrop allows the same neural network connection structure to be maintained in the unfolded neural network at each time step even after its application.

Date collection and preprocessing
Corporation H makes available the data of production plans, the previous day's production performance, and the inventory quantity to its partners through its website. The production plan refers to the planned production quantity that Corporation H plans to produce on the retrieval date and the next day (the company shares the hour-by-hour updates of the production plan, but this study reflected the total daily quantity). The production performance of the previous day represents the amount of materials used for production of Corporation H on the previous day. The inventory quantity is the quantity of materials that Corporation H has in its possession. The production plan, inventory quantity, and the previous day's production performance distributed by Corporation H on the retrieval date were collected from January to June 2018. In addition, the shipment quantity sent from D company to the customer company, which is the target variable in this study, was collected from January to June. The data was sorted and arranged by product category to be analyzed as shown in Table 1. Assuming that D company is notified by Corporation H today that the next day's Demand is 170, Act represents D company's performance of delivering parts to Corporation H's warehouse in consideration of BeAct and stock as well as the Demand. With respect to Act, BeAct and stock which occurred because D company did not produce products according to Demand even though Corporation H distributed the information of Demand, were also taken into consideration.

Data analysis
In general, Corporation H has independent factories for each type of engine. A production line of each factory manufactures several products of the same engine product category. Since products in the same product category are expected to have a strong impact on each other in terms of demand, this research set product category as the unit of analysis. Accordingly, during data pre-processing, data was divided according to the product category, and demand forecasting was performed separately for each product category.
The input layer was composed of Demand, BeAct, stock, SDemand, SAct, SBeAct, and Act, and the hidden layer was set as a single layer considering that events were sparse and there was small data available for use. DNN models were trained by randomly sampling 80% of the response-to-demand data, and the experiments were repeated 52 times to derive verification statistics by applying trained models to the tests of the remaining 20% of data [5].    The Google TensorFlow library was used and the DNN model was set through the optimization process. The LSTM model was trained by randomly sampling 80% of the 153 matchup datasets, and the trained model was applied to the remaining 20% of the matchup data to calculate the amount of evapotranspiration, and the obtained value was compared with the ground observation data. These steps were considered to comprise a single experimental procedure, and this experiment was repeated 200 times to derive verification statistics [5].
As shown in Table 2, Adam (Adaptive Moment Estimation) was used as the optimizer to minimize the loss function and improve the learning speed. Also, ReLU (Rectified Linear Unit) was used as the activation function that solves the vanishing gradient problem of the loss function that can occur in the backpropagation process (Agarap, 2018). In this study, the DNN model was built by optimizing the weights and bias set using the backpropagation algorithm, and then, dropout was performed again to improve optimization [2] (Choi & Min, 2015). Table 3 shows the result.
In the situation where data sparsity exists, it can be seen that the prediction accuracy of the model improves as the dropout is not applied to the LSTM model.    Figure 6 shows the accuracy when the LSTM without dropout model was derived by randomly selecting 39 data from a total of 52 product time series data, and the test was performed using the remaining 13 product data. The LSTM without dropout model shows an accuracy of 66.8%.

Conclusion
The results of the study indicate that LSTM, which connect the present task to the previous data using neural network methods to predict the results, is a suitable technique for demand forecasting than simple exponential smoothing. Also, dropout was applied to prevent complex co-adaptation of training data and reduce overfitting in neural networks. As shown in Table 3 and Figure 6, no dropout leads to more accurate forecasting compared to the conventional LSTM method.
Dropout has been shown to be useful in the case of relatively large deep network models with a lot of training data. However, it is less effective in the case of models with sparse data. Therefore, there is a need to conduct research on the effectiveness of dropout algorithms under conditions of data sparsity.