{'fraction_non_alphanumeric': 0.03697553415255175, 'fraction_numerical': 0.015830842466511678, 'mean_word_length': 4.710493046776232, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Machine learning application for support for automated control systems users

The article presents the results of the analysis of determining the possibility of Machine Learning (ML) using for solving the problems of incident classification of users on the example of enterprise resource planning (ERP) systems of JSC Russian Railways and choosing a rational method for solving this problem. The presented problem is a special case of the Natural Language Processing Problem (NLP) with the use of neural networks. The article covers findings of classification of user queries: binary and multi class, based on semantic analysis. In 2018, 1.4 million incidents were reported from users of ERP systems, only 25% of them are suitable for ML applications. Resolving other incidents requires the participation of a support specialist. Implementation of the software will reduce the waiting time for incident processing by 64% of the average. This will increase staff productivity by 5-7%. This paper explicitly proves that using convolutional neural networks for user incident classification problem has high potential in JSC Russian Railways.


Introduction
Interest of the President of the Russian Federation and the Government in the field of information technology (IT), strongly promotes the introduction of innovative techniques like Machine learning (ML) in Russian state-owned companies, including JSC "Russian Railways". Machine learning is already widely used in driving solution for the following problems: ad targeting and sorting articles by category, • telecommunications network management, • information security monitoring, • spam filtering.
• classification of user incidents [1]. The solution suggested in the article [1] is a full-fledged service desk platform with ML elements. Some companies already operate their own service desk systems. Russian Railways is one of these companies. Because of high price of the IT solution for a full-fledged service desk platform with ML elements, the We suggests implementing their own software. As an additional remark it is worth mentioning that the software mentioned above can be integrated with other systems.

Task
In this paper , the authors analyze the possibility of using ML to solve the problem of classifying user incidents of automated control systems of JSC "Russian Railways". This is necessary to provide users with responses to incidents. For example, in 2018, users created 5.6 million incidents. A quarter of them are incidents for ERP systems and these hits make the datasets for further analysis. Automation process with the help of machine learning techniques helps to improve cost and time efficiency.
In a sample of 36 thousand queries, the existing model was able to classify positively 80% of incidents with a probability higher than 75%. The implementation of the developed software will reduce the waiting time for the solution of the considered incidents of users of ERP systems of JSC "Russian Railways" by an average of 10.1 hours or 64%. This will save in total 60 thousand hours.

Approach
As already mentioned, the system includes four groups of automated business processes . It is proposed to define business processes for each of the groups of business processes. The paper deals with three of groups.
The problem of multi classification is follows: it's necessary to defined a set of pairs "object, class" × based on knowledge of the finite training sample of observations = {( 1 , 1 ), … , ( , )}. It is required to construct an algorithm : → , which is able to classify an arbitrary object ∈ . In the technological problems being solved, the "object, class" pair is "text, class", where the class is determined based on what type the text belongs to in terms of semantic content. In other words, the classical problem from the field of Natural Language Processing (NLP) is solved [2,3,4]. To solve the set technological tasks, it was decided to use deep neural networks, in particular, convolutional and recurrent. This solution is based on the success of these algorithms to solve text classification problems [5][6][7][8]. A separate neural network was trained for each task. The algorithm for training and selecting a neural network model is shown in figure 1. In block 1, texts are divided into words and other components (tokenization) and punctuation and other symbols are eliminated (normalization) [9].
In block 2, data is cleared from words that do not carry any semantic load (stop words, in Russianthese are interjections, conjunctions, etc.) using a ready -made library [10]. At this stage, words are lemmatized using a ready-made library [11], i.e., words are reduced to the semantic and canonical form of the word. For example, "prices -price". In block 3, data is pre-processed to convert it to numeric attributes. The following methods were used to convert text to a number:" word bag " (BOW) [12], TF-IDF [13,14], N-Gramm (N-gramm) [15], and Embedding. The first three methods of converting text to numbers were used for classical machine learning algorithms, and the last one was used for deep neural networks.
In block 4, algorithms are trained using various optimization methods. The amount of training data is small for NLP tasks. Therefore, the training was performed using cross-validation. The main models used were: • logistic regression (LogReg), • nearest neighbor method (kNN), • single-layer neural network (NN), • convolutional neural network (CNN), • recurrent neural network (RNN).
In block 5, the quality of each of the machine learning algorithms is evaluated. Quality control is performed on a validation sample that did not participate in the training. Metrics are used for evaluation: • Accuracy (percentage of correct algorithm response), • Precision (the proportion of the really correct choice of a class to the total number of definitions of the same class by the algorithm in the test sample), • Recall (the percentage of objects with the specified class found by the classifier among all objects with the specified class in the test sample), • F-measure (the harmonic mean between Precision and Recall). In block 6, the analysis of each algorithm in combination with the text processing method was performed.
The following data sets were used for each task: • in Task 1, we needed to build a classifier that divides the sample into 16 classes. The class distribution is shown in Figure 2. 45831 records were used for deep neural network training, and 19642 records were used for testing. • in Task 2, we needed to build a classifier that divides the sample into 13 classes. 45778 records were used for deep neural network training, and 1,920 records were used for testing. • in Task 3, we needed to build a classifier that divides the sample into 16 classes. 137496 records were used for deep neural network training, and 58927 records were used for testing. The Embedding method is used for preprocessing data and converting it to numeric attributes. To limit the feature vector used when converting text to a number using Embedding, a limit of 31, 41, and 36 words was selected for each task, respectively. The restriction was chosen by the authors based on the fact that the quantile at the level α = 0.95 for the distribution of record lengths is provided for the following numbers of words X α (1) = 31, X α (2) = 41, X α (3) = 36. The distribution of samples by class and the distribution of the length of rows used for training for task 1 are shown in figure 2.

Results
The results of training on the test set are presented in Tables 1 and 2. The main models used were: • convolutional neural network (CNN); • recurrent neural network (RNN) [25].
In Table 1, the quality of the algorithm is measured using the Accuracy metric [8].
There is a problem of unbalanced classes in the sample. Therefore, it is necessary to use a more objective quality metric-F-measure [5]. The results of the algorithms are presented in Table 2.    For classes determination in Task 1 and Task 2, convolutional neural networks, trained on an unbalanced sample, showed the best results from the point of view of the F-measure. For classes determination in Task 3, the best one was a convolutional network trained on a balanced sample. As a result of research on the data obtained, the following conclusion was made: to improve the value of quality indicators, additional data collection on minor classes is required.
The results of the classification will be used to determinate type of guidance materials which need users experiencing difficulties. Depending on the percentage of users who can use the materials to solve their own issue, the effect will be different. However, not all business processes were defined with sufficient accuracy (BP with accuracy below 50% is excluded from the calculation). An example of effect calculating for Task 1 is shown in Table 3 based on statistics on the input flow of user incidents for 2018. Similarly, the effect is defined for the other two Tasks (2, 3) and summed.