{'fraction_non_alphanumeric': 0.029681732016362365, 'fraction_numerical': 0.005637034819914197, 'mean_word_length': 4.858270017533606, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Methods to identify the destructive information

The article discusses the methods that ensure the detection of destructive messages in the Internet environment. It is shown that to detect them the approaches based on the analysis of the syntactic patterns; analysis of semantic information included in the text, and correlating it with the text corpus; crowd sourcing methods; identification of the patterns of user behavior in social networks; the consideration of the additional information etc., were applied. Good results can be achieved under certain conditions: an access to traffic social networks and other online news resources, opportunities to organize or to get results from crowd sourcing, etc., when there are the restrictions on these conditions the detection of destructive messages can be performed based on indirect signs. The results of the detection of destructive news in the corpus of news reports in Russian, posted on the websites of the Republic of Kazakhstan are shown.


Introduction
Over the past decade and a half or two, the volume of constantly generated information materials has increased. These materials are instantly becoming publicly available due to their placement in the Internet, and they can become both a positive and a negative impact on society. In order to ensure the sustainable development of both individuals and society as a whole, it is necessary to minimize the possible destructive impact of media content. As noted in [1], new protective measures obtained in the course of interdisciplinary research aimed at limiting the spread of false information are needed. Of course, the solution to this problem lies not only in the field of information technologies, but the information technologies are the important tool in ensuring the reliability of information in the media space.
Let's describe the signs of news of interest from the point of view of the organization of the information system of media monitoring in the interests of maintaining social stability and sustainable development of the individual and society. First, we discard the non-resonant news, which are published on resources with a small media coverage or having for a certain period of time a small number of reposts. Obviously, such news cannot have a serious impact on social stability.
Secondly, the resonant news are tested for social significance, focusing primarily on the subject of news, as the news with insignificant topics also have a weak impact on social stability, regardless of their other characteristics.
Third, if the news is socially significant, we check its reliability. This is the most difficult part of the decision-making process, which is shown in Figure1. If the news is unreliable (fake), then we classify it as interesting from the point of view of users of the information system of media monitoring in the interests of maintaining social stability.
Finally, fourth, even formally reliable news, but bearing the signs of bias (unjustifiably high degree of generalization, excessive emotionality, politicization, etc.) will also be classified as of interest from the point of view of the potentially dangerous destructiveness.

Overview of problem solving methods
Let's describe the approaches used to identify the news that satisfy the above criteria.
The methods for determining the resonance of news are quite obvious and depend on whether the creator of the monitoring system is guided by the information provided by the reviewed site or has access to its traffic.
In the first case, as a rule, you can see only the number of comments. At the same time, to count the number of persons who read the news or left comments taking into account the repetition, it is necessary to use separate means of analyzing the web page: in addition, if the site does not provide authorization of users by means of electronic digital signature (EDS) or SMS messages [2], then a reliable count of commentators is impossible: the same person can leave comments using different names or even anonymously.
An external user can see only the number of readings if the site owner has opened this indicator. In this case, it is necessary to check whether repeated visits from the same computer are not taken into account -in this case, the rate of visits is unreliable, because the interested user can easily "catch up" with the attendance.
The number of approvals/disapprovals can be seen if the site is equipped with a voting mechanism. At the same time, if the authorization of users via EDS or SMS-messages is not provided, the indicator is not reliable, because "cheating" is possible. A researcher at whose disposal is the website traffic, has the very different possibilities. In this case, it can, using information about the URLs of visitors, track the number of unique visitors and unique commentators, the location of visitors -with accuracy not only to the country but also to the city, etc.
Thus, the reliability of the assessment of the resonance of the Internet publication increases with access of an explorer to Internet traffic, if there is no access, more or less reliable judgment on the resonance of a publication can be made only on the basis of the number of comments, and after further analysis of the uniqueness of the commentators.
The topics of social importance are usually determined by interviewing experts who possess the data on the state of public opinion, which, in turn, can be obtained through sociological surveys of the population.
The most difficult task is to identify the unreliable, in other words, "fake" news (according to, for example, [3] -they are "news stories, invented with the intention to deceive").
Due to the actuality of the topic of identifying false reports, intensive work in this direction is carried out by groups of scientists around the world. For example, in the work [4], the main recommendations for the formation of the body of texts were formulated, among which one can distinguish a homogeneity in the length of the news and the style of presentation. The works [5,6] describe the practical results of the usage of machine learning to identify unreliable news. A sign of unreliability are, according to the authors, more positive facts and greater confidence in the above. In these works, the authors did not use any ontology for semantic analysis, limiting themselves to morphological and syntactic analysis. Sources of unreliable news were as news, really collected from various sites, as fake messages created by volunteers on the basis of truthful news. As a result, the corpus of texts was formed, consisting of an equal number of truthful and unreliable news. Frequency characteristics of N-grams, syntactic and psycholinguistic features of the text are used as analyzed characteristics of the texts. The usage of the support vector machine (SVM) method allowed to achieve classification accuracy of 75-85 % on different text corpus. In [7] the authors identified the words that correspond to more reliable and unreliable texts.
The regularities inherent in some classes of unreliable news are analyzed in [8]. The authors suggested that unreliable texts contain fewer nouns, numbers and quotations, redundancy of adverbs and repetitions. As a result, an algorithm was implemented, which in 70% of cases correctly determined unreliable news based on these features.
The so-called "hybrid" approach is promising, which implies, along with multi-level linguistic analysis of the text, fact checking via the Internet, for example, by querying the ontology DBpedia or Google Relation Extraction Corpus (GREC) [9].
The formation of text corpus for training classifiers in the case of unreliable news is quite a difficult task. To reduce the complexity of markup in [10], a method of partial markup and establishing a connection between articles in an array of untagged texts is proposed.
This approach allowed to achieve the accuracy of more than 70 % with only 2 % of marked texts. Let's note that after increasing the number of marked texts to 30 %, the accuracy of the classification reached only 75.43 %. This approach seems to be one of the most promising due to the low requirements for the body of texts and good results obtained by indirect use of semantic information of texts.
According to [11], the hoaxes, which are also a special case of fake news, as they are created with the intention to mislead readers, are possible to be classified with a high degree of reliability using social networks. In this work, the identification of hoaxes is performed by analyzing user messages in Facebook. It is reported that if it is knowing which users prefer unreliable messages, it is possible to achieve classification accuracy of about 99 % even if only 1 % of messages are marked.
The following approach, which is presented in [12], uses "wisdom of crowds" and is also based on the analysis of the behavior of users of social networks. By highlighting and analyzing of the comments about the news, the unreliable news was revealed. The following factors were used: firstly, if the article was devoted to a local event, the comment from a user geographically close to the place of events was considered more reliable, and secondly, if the user was inactive for a long time, his comment was considered less reliable. In addition, it is noted that the speed of false news distribution in  [13], and the nature of news distribution on the network is significantly different for false and true [14].
Let's review the works devoted to the identification of fake news in Russian.
The review of methods of detection of lies in the written text is given in [15], however, it is mainly about "live written" speech, including Internet communications, but not about the media. One reviewed study notes that "liars produce longer texts, use more words associated with channels of perception (see, hear), use fewer pronouns for self -referencing, more for naming other people. In addition, motivated liars avoided causations, and unmotivated liars did a lot of negations".
In the article [16] the similar signs of fakes are noted: "With confidence the lexical markers of fakes can be called that are common to advertising and the media: "you'll be amazed", "endangered", "world discovery", "something scary", "disturbing the minds" -these are just some of the titles are teasers from Martketgid".
Finally , with the work with Russian written texts corpus [17,18], which was specially made for the study of language features of texts in Russian, and contains deliberately distorted information, the fact that the models of lie detection in the text can be significantly improved by considering the personal characteristics of their authors (gender, age, psychological profile) was revealed.
However, the professionally created fakes that imitate the style of the media, completely devoid of emotional, personal, etc. characteristics. An example of this is the portal "IA "Panorama"" [19], specializing in fake news. Although the portal does not hide such specialization, but there are cases when after 1-2 reprints fake news generated by "Panorama", were spread by quite serious media.
At the same time, [15] emphasizes: "When we are choosing the characteristics of the text that can be markers of lies, it is necessary first of all to rely on those that can be extracted from the text automatically. At the moment, the most reliable are morphological parsers, a certain accuracy is given by syntactic parsers. Semantic analyzers to date give a lot of errors and have been excluded from the study".
Finally, the identification of biased news involves the use of techniques similar to those used in the identification of fake news, since both of these classes of news are characterized by the usage of similar manipulative technologies.

Method description
The classification of articles described above is possible in two ways.
First, on the basis of expert assessments, it is possible to compile dictionaries characterizing the belonging of the text to each of these classes, after which for each text under study the set of relevant parameters is determined and then they are aggregated in order to assess the possible belonging of the text to one of the classes.
Secondly, if there is a marked body of articles, it is possible to train an automatic classifier, which could calculate the belonging of the text to the corresponding class automatically, based on a particular formal model of the text.
Each approach has its own drawbacks. The first version of the classification of media texts is distinguished by a certain subjectivity, due to the fact that both the parameters and their mutual importance are determined by experts.
The second variant of the classification requires a significant amount of marked body of texts. Given that the parameters of the texts that affect the conclusion of the expert are not defined, it may be necessary to use a corpus suitable for deep learning methods (hundreds of thousands of texts). The formation of such a corpus is a long and laborious process. Therefore, at the initial stage of implementation of the system, we use the first approach, which can be called parametric.
Generally the process of text and media analysis consists of the following steps:  A list of parameters that determine the belonging of the text to one of the above classes is formed.
 The comparative importance of the parameters is estimated.
 For each text to be considered, a parameter estimate is calculated. On the basis of the received estimates of texts the assessment of mass media as a whole is formed. Evaluation of the above-mentioned parameters of media texts can be based on natural language processing (NLP) methods: significant progress has been made in the development of models and algorithms for text analysis. In turn, the aggregation of heterogeneous parameter values is often performed using multi-criteria decision making (MCDM). Let's consider the main features of NLP and MCDA in solving the problem of multi-criteria evaluation of the media.
One of the methods which is used productively in the field of NLP is thematic modeling (thematic analysis). This is a method based on statistical characteristics of document collections, which is used in the tasks of automatic abstraction, information retrieval and classification [20]. The idea of this approach is to intuitively understand that the documents in the collection form groups in which the frequency of occurrence of words or combinations of words varies.
The basis of modern thematic models is a statistical model of natural language. The probabilistic thematic models describe documents by a discrete distribution on a set of topics, and topics are described by a discrete distribution on a set of terms [21]. In other words, the topic model determines which topics each document refers to and which words form each topic are formed it. The clusters of documents which are related to a set of topics allow, in particular, to solve the problems of synonymy and polysemy of terms [22].
The probabilistic latent semantic analysis (PLSA), additive regularization (ARTM) [23] as well as a very popular latent Dirichlet allocation (LDA) [24,25] are used to build a mathematical model of the corpus of documents.
Each topic can be considered as a separate parameter of the text or the media as a whole, the value of which is determined by the mentioned conditional probabilities. Where it is necessary to make a decision on the basis of a variety of heterogeneous parameters and alternatives, a number of methods are used to ensure that the mutual importance of the parameters is taken into account and that their values are aggregated as one or a small number of estimates. By their nature, such tasks belong to the field of multi -criteria decision-making, widely used in decision support system (DSS).
The method of analysis of hierarchies (AHP) [26] has shown its effectiveness in solving many practical problems [27]. Therefore, in this paper, this method is used to assess the weight of parameters in the process of classification of media texts.
In turn, the thematic model built with the use of LDA provides multimodal evaluation of media texts by calculating the conditional probability distributions of documents by features (p1), topics (p2) and classes (p3). The implemented method is notable for the fact that the thematic model created by cluster analysis (teaching without a teacher) is then applied in combination with expert-defined classes and features. Thus, the semantics of the required distribution is set by the user (expert), although the initial thematic analysis depends only on the body of documents. The application of the thematic model provides, in our opinion, the reduction of errors associated with the calculation of compliance of texts and dictionaries parameters.
The initial data and the resulting conditional probability matrices are shown in Figure 2, where MMS (mass media sources) is a set of text sources. Media (MMS) are the source of m documents (papers) that are obtained through the application of data collection systems (process 2). The resulting body of documents M is divided into thematic clusters T (process 1). Experts form classes C (process 4) and define properties or parameters of classes Q (process 3). Properties are described by dictionaries of words, expressions (dictionaries) or procedures of their identification in the text (features).
Using a set of corpus topics, firstly, we obtain a discrete distribution of conditional probabilities of articles and topics -p2(k|m), where k ∈ T, m ∈ M. Second, we obtain a conditional distribution of dictionaries (parameters) and topics p 1 (k|q), where q ∈ Q, that is, we determine to what extent the parameter describes a specific topic. Thirdly, using the analytical hierarchical process (AHP) we calculate the importance of the parameters for the classes (for each class) and obtain p 3 (c|q), where с ∈ С. Then, using p1 and p 3 , we expect the conditional distribution of subjects by grade -p 4 (k|c). Knowing the probability distribution of the topic by class (p 4 ) and the probability distribution of the article by topic (p 2 ), you can calculate the distribution of the article by class p5(m|c). In turn, the distribution of the article by features or dictionaries -p6(m|q) depends on p1 and p2.

Figure 2. The definition of conditional probabilities
The LDA algorithm is used to calculate the set of topics of the corpus T. The Jacquard coefficient [28,29] is used to calculate the probabilities p 1 and p 2 . To calculate the probabilities p 3 -AHP.

Adaptation of pymorphy library for a work with pre-reform spelling
The algorithm described above (referred to below as an aggregator) is used to calculate the distribution of media texts by parameters (p 6 ) and the distribution of articles by classes (p 5 ). At the same time, the experts have created the dictionaries of signs: Facts (657 lexical units), Negative tonality (2344) where the columns (from the left to the right) correspond to the parameters listed above, and the rows (from the top to the bottom) to the classes: "Socially significant, "Objective", "Reliable". Let's note that this matrix is not complete. Not all parameters for the corresponding classes are defined. Zero means that this parameter is not used to determine the probability p5. For example, it can be seen that the social significance of the text is determined by five criteria, while reliable using only one (facts).
A set of articles by the Tengri News news agency (https://tengrinews.kz/) was used to evaluate the capabilities of the aggregator. The corpus contains 1800 articles marked up by experts, what made it possible to compare the results of the aggregator with expert estimates. Figure 3 shows the tables where 5 news with the largest and 5 with the least probability of matching the parameter or class are displayed (aggregator score). The expert ratings are given in the column Expert Score.