{'fraction_non_alphanumeric': 0.033034759144832435, 'fraction_numerical': 0.009021091791604796, 'mean_word_length': 4.79764880125891, 'pattern_counts': {'":': 0, '<': 6, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Science, Technology and Innovation Policy Indicators and Comparisons of Countries through a Hybrid Model of Data Mining and MCDM Methods

: Science, technology and innovation (STI) policies are of great importance for countries to reach their sustainable development goals. Numerous global databases have many indicators that measure and compare the performance of STI policies of countries. However, many problems arise regarding how to identify, classify and systematically analyze these indicators in order to measure, monitor and improve the performance of STI. The study includes a literature review on global problems and new trends in STI policies, while mentioning the necessity of an internationally comparable STI indicator set, current STI indicator studies and efforts, and studies for each continent. In light of these, all the indicators selected are introduced in detail. The strengths and weaknesses of the countries in the study in terms of evaluation indicator values are indicated. After determining the indicator weights objectively with the entropy method, 40 countries are compared with TOPSIS, VIKOR, PROMETHEE I-II, ARAS, COPRAS, MULTIMOORA, ELECTRE, SAW and MAUT methods. In addition, countries that show similarities with each other are evaluated by cluster analysis, which is one of the data mining classiﬁcation methods. This study offers a new and original approach with MCDM methods on this subject. Considering all the results obtained in the study together, these rankings are compared among themselves and with the rankings speciﬁed in the Global Innovation (2019) and Global Competitiveness (2019) indices, and it is seen that the results are consistent. In addition, it is possible to update and publish this study every year with updated data.


Introduction
Developments in science and technology and strategies based on innovation have become the basic elements of productivity increase and competition at both country and company level. Science, technology and innovation (STI) are very important for all countries because of their sustainable growth effect and solutions to the energy, food security and climate change. Therefore, STI is a significant topic for both sustainable growth and the achievement of political goals. STI has been on the agenda of both developed countries and developing countries in recent years. Some of the main findings of the 2018 Industrial R&D Investment Scoreboard, published by the European Commission on 17 December 2018, are shown in Figure 1 [1]. As it can be understood from Figure 1, the biggest difference that separates developed countries from developing countries is the knowledge gap between them. As long as the knowledge gaps are eliminated, it is possible to close the development and income gap. Figure 2 shows some key figures about growth rates based on R&D [1]. With the structural change trends in production brought about by technological developments, companies are now strong as long as they can adapt to global value chains, and also countries have competitive power as long as they have companies with global value chains. Today, science and technology are rapidly globalizing, open innovation systems and collaborations are becoming widespread, and new technologies increase the speed of knowledge dissemination. In a world where competition is rapidly increasing and STI is the most decisive actor, countries need to shape their policies accordingly. Therefore, STI indicators are very important to evaluate the current status of countries and to compare them with each other in terms of STI. Therefore, large global databases such as OECD (Organization for Economic Development and Cooperation), UNDP (United Nations Development Program), ITU (International Telecom Union), Eurostat, the World Bank and the statistical offices of the countries have many indicators that measure and compare the Science, Technology, and Innovation Policy (STIP) of countries. Such efforts produce indicator systems to reveal the state of the capabilities and capacities of nations and update existing ones. STI indicators and regional and international comparisons are considered among the important guides of governments in policy formulation on issues such as economy, welfare, and development. The developing competitive environment increases the need for companies and countries to manage the complexity surrounding the STI policy process. However, many problems arise regarding how to determine the importance of these indicators, how to classify them, and how they should be analyzed systematically in order to measure, monitor, and improve the performance of STIP. Therefore, this is an issue that attracts the attention of experts, policymakers, academics, and investors. Designing strong STI indicators and frameworks that can be used in regional and international comparisons should be seen as a priority for governments to form national and international policies [2,3].
This study includes all the topics that are outside of governments' policies, whether they are aware of it or not, and that is essential for a sustainable STI policy, by presenting a holistic system approach beyond what countries directly implement with each policy and strategy. Thus, it recommends a sustainable system approach to managers who are decision-makers in policies in order to develop holistic solutions and strategies while working on complex issues that they have to deal with on a global scale. By making use of this STI policy proposal, which consists of multi-dimensional factors, decision-makers can see their strengths and weaknesses, and have the opportunity to see and evaluate the status of their competitors.
In the literature, there are studies that many researchers compare and rank countries in terms of their performance in various subjects with MCDM and multivariate statistical methods. These studies evaluated countries in terms of R&D, innovation, technology, regional development, competitive advantage, trade and macroeconomic indicators. The main criteria used in comparing countries in terms of R&D and STI performance in the literature are as follows [4]: Patent applications made by non-residents (number/year); patent applications made by residents (units/year); trademark applications made directly by non-residents (number/year); trademark applications made directly by residents (number/year); trademark applications made by non-residents (number/year); trademark applications made by residents (number/year); total trademark applications (units/year); number of researchers in R&D (per million people); the ratio of R&D expenditures in GDP (%); high technology export amount (USD); high technology exports (percentage of manufacturing products exports); ICT goods exports (percentage of total goods exports); and the number of articles in scientific and technical journals.
Some similar studies on this subject are as follows. Lin, Shyu [5] used a descriptive analysis with descriptive statistics under the innovation policy framework proposed by Rothwell and Zegveld. This study also informed a comparative policy analysis across China and Taiwan. Chaurasia and Bhikajee [6] aimed to analyze India's economic growth performance, STI investment and health improvements in comparison with Brazil, China and Singapore. Sun and Cao [7] analyzed the dynamics of China's science, technology and innovation (STI) work, emphasizing the importance of studying science, technology and innovation (STI) activities in China in understanding international competitiveness in the knowledge-based economy. Erdin and Özkaya [8] assessed the Association of Southeast Asian Nations (ASEAN) countries in terms of the criteria of sustainable development index. They used the TOPSIS method to compare and rank them. Salam, Hafeez [9] compared lowmiddle-income countries while evaluating the dynamic relationship between technology adaptation, innovation, human capital and economy. Blažek and Kadlec [10] examined the interrelationships and problems between the knowledge bases, R&D structure and innovation performance of European regions, and made regional assessments about their situations. Canbolat, Chelst [11] used the MAUT method to evaluate Mexico, Czech Republic, Poland, South Korea and South Africa in terms of global competitiveness and survival, research and development, government regulations and economic factors for establishing a production facility on a global level. They tried to decide which country was more suitable. Kang, Jang [12] compared the national innovation system between the US, Japan and Finland to improve the Korean Deliberation Organization for national science and technology policy.
Manyuchi [13] studied the use of innovation indicators in South Africa's science, technology and innovation policy making and introduced the institutions that support them. Özbek and Demirkol [14] assessed the European countries using AHP, ARAS, COPRAS, and GRA (Gray Relational Analysis) methods with macroeconomic indicators. The best performing country in the evaluation was Germany, while Greece was in the last place in the ranking.
In the first revision meeting of the OECD's Science, Technology and Innovation Policy index, two common issues agreed upon by the delegates and committee members working on this issue are the inability to be sure of the method to analyze the indicators in the most accurate way and whether collecting information only with surveys produces a result consistent with the facts. Therefore, this study aims to create an appropriate comparison framework by taking advantage of the most up-to-date existing STI policy indices and to compare countries using these indicator values with multi-criteria decision-making methods (MCDM) and cluster analysis. Thus, a comprehensive MCDM approach is presented to the STI policy comparisons and evaluations, which has not been done yet in the field. Therefore, the study is expected to add an important novelty to the literature.
In the study, the weights of the indicators are determined with the Entropy method, which is one of the MCDM methods. Then 40 countries whose all indicator values are available are compared with TOPSIS, VIKOR, PROMETHEE I-II, ARAS, COPRAS, ELECTRE, SAW, MAUT, and MULTIMOORA methods. In addition, countries that show similarities with each other are evaluated by cluster analysis, which is one of the data mining classification methods. The selected countries are evaluated within the framework of 10 dimensions and 115 criteria. These criteria were determined as a result of the evaluation of the OECD, the World Bank, the Global Competitiveness Index, and the Global Innovation Indices, which conduct studies and publish reports on science, technology, and innovation. Criteria, indicators, descriptive information about indicators and sources from which data are obtained (SCImago [15], Indexmundi [16], OECD and Group [17], Unesco [18], World-Bank [19], TradingEconomics [20], Schwab [21], Dutta, Lanvin [22], ITU [23], IMF [24], ILO [25] and Numbeo [26]) are shown in Appendix A.
The rest of the study is organized as follows: Section 2 explains the proposed methods. Section 3 presents the obtained results. Section 4 presents the discussion, and Section 5 presents the conclusion.

Methodology and Data
All the data obtained have been verified in more than one source and evaluated by comparing them among themselves. The sources and values of all indicators used in the application are included in the appendices of the study.
While the list of the evaluated countries with some descriptive information is presented in Table 1, the definitions of indicators and dimensions is included in Appendix A. In this study, the Entropy method is proposed in order to avoid subjective evaluations while weighting of indicators to be used in the evaluation of STI policy performance .  After determining the indicator weights, the data of the indicators of the countries are  analyzed with TOPSIS, VIKOR, PROMETHEE I-II, ARAS, COPRAS, MULTIMOORA, SAW, MAUT and ELECTRE methods and the performances of 40 countries are evaluated and are compared with each other. Also, the countries classified by using the data mining cluster analysis. All results are evaluated among themselves for consistency.
Existing multi-criteria decision-making approaches can cause confusion of users from time to time due to the complex calculation steps and the solutions they produce. Each of these methods has its own strengths and weaknesses. Due to these differences in the structures of MCDM approaches, the problem of obtaining different rankings with different methods arises in the evaluation of the same problem, also known as inconsistent problem ordering. This is the biggest criticism of these techniques. The main reasons for the occurrence of these differences are as follows: use of different weights, using a different approach to determine the best alternative, the effort to measure goals, and using parameters that may affect the result differently. Currently, there is no specific standard rule for the application of multi-criteria assessment methods and the interpretation of the results obtained. Table 2 was created by the author based on Brauers and Zavadskas [27]. The methodology used in this study combines two approaches to estimate STI performances of selected countries. These approaches are Clustering method and MCDM methods. The proposed methodology in this study includes three phases as stated: Data understanding and collection from indexes; Data preprocessing; Modeling and data analyzing.
Data preparation and data normalization were included in data preprocessing phase. In the data preparation step, some uncompleted data have been omitted. Next, a normalization process is required to put the fields into comparable scales. This process is due to the different scales of STI inputs. In this paper, a min-max approach was used which recalled all record values in the range between 0 to 1. Then, weights of STI variables were calculated by Entropy method and the normalized data of STI have been weighted by these weights. Then, based on weighted STI values countries were clustered by K means clustering. Finally, the clusters of countries were ranked using MCDM methods. Afterwards, the clusters formed in the clustering analysis and the rankings obtained from the MCDM results are evaluated together and the results are compared. When a similar analysis is made for the countries that are not included here, by considering the countries in the study, it can be determined to which cluster a country belongs. Research framework of the study is shown in Figure 3.

Shannon Entropy and Objective Weights
Two different weight methods are used in MCDM methods, namely objective (objective) and subjective (subjective). Subjective weights are obtained by directly benefiting from the opinions of decision-makers like other MCDM processes. In objective weighting, it makes use of the quantitative features of the criteria. Entropy method, as one of these objective weighting methods, can be applied under conditions where decision matrix values are known [28]. Shannon and Weaver [29] proposed the concept of entropy, a measure of uncertainty in information formulated in terms of probability theory. The concept of entropy is a suitable option for our purpose, as it enables the measurement of relative contrast densities of criteria representing the original information conveyed to the decision-maker [30]. This method has been used in many areas such as spectral analysis [31], language modeling [32], and economics [33].
Shannon has developed an H measure that provides the following properties for all pi in the estimated common probability distribution (P) [34]: H is a positive continuous function, If all p i are equal (p i = 1 n ), then H must be a monotonic incremental function of n.
For all n ≥ 2, H(p 1 , p 2 , . . . , p n ) = h(p 1 + p 2 , p 3 , . . . , p n ) + (p 1 + p 2 ) H p 1 p 1 + p 2 , Shannon showed that the only function that meets these properties is as follows: Shannon's entropy method is explained as a weighting calculation method with the following process steps [35][36][37][38]: Step 1: Creating the Decision Matrix In the first step of the entropy method, the decision matrix is first created similar to other multi-criteria decision making methods.
Step 2: Obtaining the Normalized Decision Matrix In order to convert the criterion scores into common units, the criteria are normalized according to their benefit or cost characteristics. In this step, Equation (4) is used as follows: In this equation, i = alternatives; j = criteria; r ij = normalized values; x ij = benefit values of the i.alternative for j.
With the normalization process specified in the second equation, the normalized decision matrix defined by the Equation (5) is formed: Here, P ij represents normalized values while a is benefit values.
Step 3: Calculate the entropy measure of each indicator using the following equation: k = entropy coe f f icient {(ln(n)) −1 }; P ij = normalized values; E j = entropy value.
Step 4: (d j ) uncertainty value is calculated by Equation (7) Step 5: w j weights are calculated as the importance of j criterion by using the Equation (8), The sum of entropy probability values is always equal to 1.

Ranking of Countries Based on MCDM Methods
In the rest of this section, the steps of the methods used in the study are explained mathematically.

TOPSIS (Technique for Order Preference by Similarity to Ideal Solution) Method
TOPSIS (Technique for Order Preference by Similarity to Ideal Solution), proposed by Hwang and Yoon [39] and developed by Lai, Liu [40], is a method that aims to reach the most ideal solution among the available options [41,42]. The positive ideal solution increases the utility metrics while decreasing the cost metrics. The negative ideal solution does the opposite. TOPSIS is generally defined in five steps [43]: Step 1. First of all, normalization is done to the decision matrix. Using the r ij values calculated here, the R matrix is obtained: Step 2. By applying the weighting process stated below to the matrix in the first step, the v ij matrix is obtained with the v ij weighted normal values. w j represents the weight of the J-th criterion or indicator.
Step 3. In this step, positive ideal (A*) and negative ideal (A − ) solutions are determined: When indicator j is a benefit indicator: When indicator j is a cost indicator: Step 4. The deviations of all alternatives from positive and negative solutions (discrimination criteria) are obtained individually using the following equations using the m-dimensional Euclidean distance: Step 5. In this step, the relative proximity to the ideal solution is determined. The relative proximity of the Ai alternative with respect to A* is defined by the following equation. Then, sort results in descending RC i .

VIKOR (Vise Kriterijumska Optimizacija I Kompromisno Resenje) Multi-Criteria Optimization and Compromise Solution Method
VIKOR is a method proposed by Opricović [44], which aims to reach a compromise order and a consensus result within the framework of determined weights. The term consensus in this definition refers to a consensus of decision makers between options, that is, the determination of a joint decision. The process steps of the VIKOR method are as follows [45]: Step 1. Calculating the positive ideal solution f i * and negative ideal solution f i − . I 1 is a benefit indicator, I 2 is a cost indicator.
Step 2. Calculate the S j and R j of the scheme. W i represents the weight of index i.
Step 3. Calculate Q of each scheme.
While v in the equation of Q j represents the relative importance of the majority of the criteria, namely the maximum group benefit, the value of 1 − v indicates the relative value of the opponents' minimum regret, ie the weight. By ordering the values of S, R and Q ascending, the order between the alternatives is obtained. In this evaluation process, three rankings are obtained. Two conditions must be met for the result to be valid.
Condition 1 (C1)-(Acceptable advantage): There must be a distinct difference between the best alternative and the closest alternative. A1 has the smallest Q value i.e., the first best alternative, while A 2 is the second best alternative. The acceptable advantage in this case is shown as follows; Condition 2 (C2)-(Acceptable stability): In order to ensure the stability condition for the compromised solution found; A1 alternative with the highest Q value must have received the highest value from at least one of the S and R values.
Step 4. Only after these conditions are met, the alternative with the smallest Q value can be considered as the best option.

PROMETHEE
PROMETHEE method is a multi-criteria decision making (MCDM) method that enables the analysis of alternatives to be evaluated using preference functions selected according to the criteria. This assessment for alternatives is obtained by performing paired comparisons [46]. PROMETHEE I method, which was introduced for the first time by Mareschal, Brans [47], performs partial ordering, while the PROMETHEE II method performs full ordering. In addition, later Mareschal and Brans [48] proposed the GAIA (Geometrical Analysis for Interactive Aid) method in 1988, which supports the PROMETHEE method and can obtain graphic presentations. PROMETHEE method consists of 4 steps as follows [49][50][51]: Step 1. Evaluate the n solutions (a1, a2, . . . , an) in A under m criteria C k , and get the decision matrix X = (x ik ) (i = 1, 2, . . . , n; k = 1, 2, . . . , m). When G k (d ij ) = 0, there is no difference between scheme a i and scheme a j . When G k (d ij ) = 1, scheme a i has definite priority over scheme a j .
Step 2. Based on the weight (W) provided by the decision maker, a multi-criteria preference ranking index (H) is calculated.
Step 3. The positive and negative directions of the order of A i 's preference are defined by Φ + (a i ) and Φ − (a i ), respectively.
Step 4. The exact ranking of the alternatives is obtained according to the values of Φ(a i ).

ELECTRE (Elimination and Choice Translating Reality English) Method
ELECTRE (Elimination and Choice Translating Reality English) method is an MCDM method proposed by Benayoun, Roy [52]. ELECTRE is a method that sorts and selects alternatives according to their paired comparison advantages in terms of each of the evaluation criteria. It has an eight-step process [53]. The steps of the ELECTRE evaluation process are as follows: Step 1. This process converts the elements of the decision matrix into dimensionless comparable elements by applying Equation (30) Thus, the normalized matrix X is shown as where m presents the number of alternatives, n shows the number of criteria, and x ij is the normalized preference measure of the i-th alternative with regard to the j-th criterion.
Step 2. Construction of weighted standard decision matrix (Y): The importance of evaluation factors may be different for each decision-maker. In order to reflect these significant differences to the ELECTRE solution, the Y matrix is calculated. The decision-maker must first determine the weights (w i ) of the evaluation factors. 0 ≤ w 1 , w 2 , . . . , w n ≤ 1 and the correlation coefficients of normalized interval numbers are between 0 and 1.
Then the elements in each column of the X matrix are multiplied by the corresponding w i value to form the Y matrix. Therefore, the weighted matrix which is derived from the normalized matrix is shown in Equation (33): . . . . . . . . .
Step 3. Determining the set of concordance (C kl ) and discordance (D kl ). The Y matrix is used to determine the fit sets. The decision points are compared with each other in terms of evaluation factors and the sets are determined by the relationship shown in the formula: The formula is based on the comparison of the superiority of the row elements relative to each other. The number of concordance sets in a multiple decision problem is (m.m−m). The k = l condition should be provided for k and l indices when creating concordance sets. The number of elements in a set of concordance can be the maximum number of evaluation factors (n).
For example, in order to be able to decide the C concordance set for k = 1 and l = 2, the elements of row 1 and 2 of the Y matrix are mutually compared with each other. When there are four evaluation factors, the C 12 concordance set will have, at most, four elements. For instance, if the comparison results of rows 1 and 2 are as follows: y 11 > y 21 , y 12 < y 22 , y 13 < y 23 and y 14 = y 24 . The condition in formula Equation (5) will fit for the values of j = 1 and j = 4, and the C 12 concordance set will be defined as C 12 = {1,4}. The ELECTRE method has a discordance set (D kl ) which is complementary to each concordance set (C kl ). In other words, there are as many discordance sets as the number of concordance sets. The discordance set elements consist of j values that do not belong to the complementary concordance set. In the example, concordance set is C 12 = {1,4} therefore discordance set is D 12 = {2,3}.
Step 4. Construction of concordance (C) and discordance (D) matrix. The concordance index c kl is the sum of the weights related with the criteria included in the concordance set. ance matrices (D) Concordance sets are used to create the concordance matrix (C). The matrix C is a mxm matrix and does not have a value for k = l. The elements of the C matrix are calculated by the relationship shown in the formula: j∈C kl w j f or j = 1, 2, 3, . . . , n.
The discordance matrix (D) shows the degree that a particular alternative A k is worse than a competing alternative A l . The elements of the discordance matrix (D) are calculated by Equation (36): Moreover, both of these two mxm matrices are not symmetric.
Step 5. Determine the concordance and discordance dominance matrices. The concordance dominance matrix (F) is a mxm matrix and the elements of the matrix are obtained from the comparison of the concordance threshold (c) with the elements (c kl ) of the concordance matrix. The concordance threshold value (c) is obtained by the formula m shows the number of decision points in the formula. More specifically, the value of c is equal to the product of the total value of the elements of C matrix and 1 m(m−1) . Based on the threshold value, the elements of the concordance dominance matrix F are decided by c kl ≥ c ⇒ f kl = 1, c kl < c ⇒ f kl = 0 (38) it also shows the same decision points on the diagonal of the matrix, so it has no value. In a similar way, the discordance dominance matrix G is described by using a threshold value d, where d could be explained as Step 6. Construction of the aggregate dominance matrix (E). Here, the E is a mxm matrix depending on the C and D matrices and it consists of 1 or 0 values.
Step 7. Determining the order of importance for decision points. The rows and columns of the E matrix represent the decision points. For example, if the matrix E is calculated as This indicates that the second alternative is preferred to the first alternative, the third alternative is preferred to the first alternative, and the third alternative is preferred to the second alternative by using both the concordance and discordance criteria. In this case, if the decision points are expressed with the symbol A i (i = 1, 2, . . . , m) the order of importance for the decision points will be in the form of A 3 , A 2 , and A 1 .
Step 1. The decision matrix formed by the x ij values is obtained.
Step 2. Normalized values are obtained by dividing each value in the decision matrix by the sum of the column to which it belongs.
Step 3. The weighted normalized decision matrix D 'consisting of d ij elements calculated by multiplying the weight value (w j ) of each evaluation criterion with the normalized decision matrix values is obtained.
Step 4. The sum of the weighted normalized decision matrix values of the benefit and cost criteria is calculated. S i + represents the sum of values in the i weighted normalized decision matrix of the utility criteria, while S i − represents the total value of the cost criteria. The formulas for calculating these values are shown in Equations (44) and (45).
Step 5. In this step, the relative importance value (Q i ) of each alternative is calculated.
Step 6. The highest relative priority value is determined.
Step 7. The performance index (P i ) value of each alternative is obtained.
Performance index value (P i ) which is equal to 100 is determined as the best alternative in terms of alternative evaluation criteria. The COPRAS ranking table is obtained by ranking the performance index value of each alternative in descending order.

ARAS (A New Additive Ratio Assessment) Method
A New Additive Ratio Assessment (ARAS) is a method suggested by Zavadskas and Turskis [58] in order to solve MCDM problems. The ARAS method compares the utility function value of each alternative with the utility function value of the optimal alternative [59]. The process of the ARAS method consists of 4 steps [58]. The first three steps of the method are the same as the COPRAS method. In the last step of the ARAS method, the optimality function value of each alternative is calculated, and thus it is possible to evaluate the alternatives.
Si represents the optimality function value of the i-th alternative. It is equal to the sum of all criterion values for each alternative.
The alternative with the largest S i value is defined as the most efficient alternative. Also in this step, K i utility degrees are obtained by dividing each S i value by S 0 optimal function value.
The relative efficiency of the utility function values (K i ) of each alternative is determined with K i , which takes values in the range of [0,1]. An ARAS ranking table from the best alternative to the worst ranked alternative in terms of criteria is obtained by ordering the Ki values in descending order.

Multimoora (The Multi-Objective Optimization by Ratio Analysis) Method
The MOORA method was developed and proposed by [60]. MOORA-Ratio Method. In the ratio method, the initial decision matrix values of the alternatives are normalized based on each criterion. As stated in the formula below, each data is divided by the square root of the sum of the squares of the values in the criteria to which it belongs. In this process; x ij : the value of alternative j in terms of criterion i; j = 1, 2, . . . , m; m is the number of alternatives included in the analysis; i = 1, 2, . . . , n; n is the number of criteria included in the analysis; x ij *: represents the normalized value of alternative j in terms of i criterion [62].
According to the optimization approach of the method, normalized values are summed up in maximization, subtracted in minimization as expressed in the formula [62]; i = 1, 2,..., g, values are utility criteria whose values are desired to be large; If i = g + 1, g + 2,..., n are the (cost) criteria whose values are desired to be small. j = 1, 2,..., m shows alternatives. y j *; It is the total ranking value of alternative j. While ranking the alternatives by using these values, the alternative with the highest value is determined as the best alternative. On the other hand, the alternative with the lowest y j * value is determined as the worst alternative in terms of evaluation criteria [62].
Reference Point Approach. In the MOORA Reference point approach, the best available criterion values for each criterion are determined and these values are used as reference points in the evaluation. r represents the reference value of the ith criterion. d ij represents the distance from the reference point of the criterion to which each weighted normalized value calculated in the previous analysis belongs. The deviations of the normalized values given in the decision matrix from the reference series are calculated according to the formulation given in the equation. Then, the maximum values of these distances for each criterion are determined. The alternative with the smallest of these largest values is determined as the best alternative in terms of the criteria evaluated. Meanwhile, the final evaluation score of the ith alternative is represented by P i [62].
The full multiplicative form of MOORA. Brauers and Zavadskas describe this approach for MOORA analysis in the following Equation [63]: U i represents the degree of use of the ith alternative. As seen in the equation, the product of the values of the benefit criteria of the relevant alternative forms the numerator, while the product of the values of the cost criteria forms the denominator [63].
Multi-MOORA Approach. Multi-MOORA is a new MOORA approach proposed by Brauers and Zavadskas in 2010. Multi-MOORA is a method in which the MOORA ratio, reference point and full multiplicative approaches are evaluated by making a dominance comparison [27].
Absolute dominance means achieving the same rank in all MOORA approaches applied. The Multi-MOORA order, which can be an example of the concept of absolute dominance, is (1-1-1).
In general dominance expression, it is expressed as the dominance of two of the three approaches applied. Assuming an order such as a < b < c < d: (d-a-a) to (c-b-b); it is possible to evaluate that (a-d-a) has a general dominance over (b-c-b) and (a-a-d) over (b-b-c) [64].

SAW (Simple Additive Weighting) Method
This MCDM method proposed by Churchman and Ackoff [65] is also known as the Weighted Sum Model in the literature [66]. Compared to many other MCDM methods, it is a frequently preferred method because it has a simpler calculation process [67]. The equations used in the SAW approach are as follows [68][69][70]: Step 1. Normalizing decision matrix values Normalization process differs depending on whether the criteria are benefit (maximization) or cost (minimization) criteria. The formulas applied in this step are listed below [66]: Step 2. Determination of preference values for each alternative. The total preference value (S j ) for each alternative is calculated by multiplying the criteria weight for each criterion with the normalized values of the relevant criterion obtained in the first step.
w ij : Weight of the relevant criterion.
The large value indicates that the relevant alternative should be preferred more. The relative value (S j %) of each alternative is obtained by proportioning the S j value of the relevant alternative to the total S j of all alternatives.
The alternative with the largest S j % value is the first alternative in the SAW ranking table and is identified as the best alternative among alternatives.

MAUT (Multi-Attribute Utility Theory) Method
Multi-attribute Utility Theory (MAUT) approach is one of the MCDM methods that enables the determination of the best alternative in terms of criteria by allowing qualitative and quantitative criteria to be evaluated together [71,72]. The operation process of the MAUT approach consists of two steps [73]. In the first step, the decision matrix elements are normalized.
Step 1. In the normalization process, the values of each criterion are first converted so that the best value is one (1) and the worst value is zero (0). Thus, all values must be in the range [0, 1]. This transformation is done using the following Equation [73]: Definitions of variables in this formula are shown below: x + i : The largest value of the relevant criterion.
x − i : The smallest value of the relevant criterion. x : Current value of the cell under calculation.
Step 2. In the second step after normalization process, the utility values of each alternative are calculated. The formula used in the calculation of these benefit values and the definitions of the variables used are given below [73]: The utility value of the alternative in terms of the relevant criteria. W i : weight value of the relevant criterion.

K-Means Clustering Algorithm
K-means algorithm was developed by MacQueen [74] which aims to find the cluster centers, (c 1 ,..., c K ), in order to minimize the sum of the squared distances (Distortion, D) of each data point (xi) to its nearest cluster centre (c k ), as shown in Equation below where d is some distance function. Typically, d is chosen as the Euclidean distance. The steps of K-means algorithm are shown as follows [75]: (1) Initialize K centre locations (c 1 , ..., c K ).
(2) Assign each x i to its nearest cluster centre c k .
(3) Update each cluster centre c k as the mean of all x i that have been assigned as closest to it.
(5) If the value of D has converged, then return (c 1 ,..., c K ); else go to Step 2. Figure 4 shows the weights of the criteria of the STI framework obtained by the Entropy method. As a result of entropy calculations, while the criterion with the highest importance is the management criterion with a value of 14.583%, the dimensions that follow this dimension in order are the development of human capital: education (13.743%); financial and market sophistication (12.538%); economy (11.785%); R&D investment and research workforce (11.53%); energy, mining and green technology infrastructure (9.828%); information and communication technology (ICT) (7.775%); creative outputs (7.337%); institutions (7.32%); scientific publications and citations (3.561%). If evaluated specifically in terms of indicators, Table 3 shows the order of entropy importance weights of STI indicators in descending order. The most important indicator is the intensity of local competition (1.038%). This is followed by ICT and business model creation (1.037%), trade, competition and market scale (1.037%), business environment (1.036%), market coverage (1.036%), PISA (Programme for International Student Assessment) scales in reading, mathematics and science (1.035%), ICT and organizational modeling (1.035%), foreign market size (1.035%), government online service (1.034%), e-participation (1.034%), scientists and engineers (1.033%) and accessibility to the latest technologies (1.033%).  When the administration, which is determined as the biggest weighted criterion with the entropy weighting, is evaluated, the government officials and politicians who perform poorly in this criterion should make government spending effective, increase transparency in government policies, to be fair in the decisions they make. Because the countries with the worst scores in the analyses performed have very low values in these indicators.

Results
In the SAW analysis, Relative values (S j % ) are calculated for each country. The ranking table of the SAW method is obtained by ordering these values in descending order. The country with the greatest value is the best country in terms of STI criteria. Relative values and rankings are shown in Table 4. While Switzerland is the best country in the ranking, the other countries in the top five are Sweden, Singapore, Finland, and the United States of America. On the other hand, the five worst-performing countries are South Africa, Mexico, Greece, Turkey, and Brazil, respectively.
In the TOPSIS analysis, TOPSIS ideal (S i *), negative ideal (S i − ) and relative proximity to Ideal solution (C i *) values for each country were calculated, and they are shown in Table 5. Table 6 presents the TOPSIS ranking of countries in terms of STI performances. According to the ranking obtained as a result of TOPSIS analysis, While Switzerland is the best country in the ranking, the other countries in the top five are Singapore, Sweden, Finland and the USA, respectively. On the other hand, the five worst-performing countries, Greece, Russia, Turkey, Mexico and Brazil.   In the VIKOR analysis, the weighted and normalized Manhattan distance (S i ), the weighted and normalized Chebyshev distance (R i ), and the compromise value (Q i ) for each alternative (country) have been calculated. These values are shown in Table 7 below.
Then, calculations are made by considering five different maximum group utility (v) values. Here, 1 − v also represents a minimum of individual regret. While applying these strategies, there may be a compromise problem with the value of v = 0.5, where v = (n + 1)/2n (v + 0.5 (n − 1)/n = 1). Because the first criterion related to R is also included in S. S* and S − values represent the maximum and minimum values in the Si column, while the R* and R-values show the maximum and minimum values in the Ri column. The rankings obtained according to each Qi value calculated for five different v values are shown in Table 8.
In the VIKOR analysis, acceptable advantage (1st Condition) and acceptable stability (2nd Condition) conditions are provided only for v = 1. The country rankings were obtained by ranking the Qi values calculated according to this value in ascending order.
In the ARAS method, the priority values (Si) and benefit values (Ki) of all countries were calculated, and they are shown in Table 9. When the percentage value (% Ki) of the utility value is ordered in descending order, a table is obtained indicating the order from the best country to the worst country in terms of STI. According to the ranking in Table 9, the top five countries in terms of STI are Switzerland, Sweden, Singapore, Finland, and the United States, respectively. The five countries with the worst scores are the Russian Federation, Turkey, Greece, Mexico, and Brazil.   Ranking of the countries according to the calculated COPRAS benefit degrees is presented in Table 10. According to the ranking, the country with the highest benefit rating is Switzerland. Other countries in the top five of the ranking are Sweden, Singapore, Finland, and the United States, respectively. The five countries with the worst performance score are the Russian Federation, Turkey, Greece, Mexico, and Brazil. Visual PROMETHEE, an easily applicable program, was preferred in the application of PROMETHEE analysis. The program is an important multi-criteria decision support program designed for the implementation of the PROMETHEE method.
The weights used in the analysis are the weights obtained from the entropy analysis. In the PROMETHEE method, if there is no priority among the criteria for decision makers, the first type, namely the usual preference function, is preferred. Therefore, the preference function for all criteria has been determined as the first type (ordinary) function in order to make an evaluation using only the determined Entropy weights, regardless of subjective evaluations (without prioritizing certain value ranges for any criterion). In the analysis, when the ordinary type preference function is preferred, the values part of the parameters q (indifference value), p (absolute preference threshold) and s (intermediate value or standard deviation between p and q) are left blank.
A partial ranking of countries in terms of STI criteria is obtained with the PROMETHEE I method. When the results of PROMETHEE I analysis obtained from the Visual PROMETHEE program are evaluated, it is seen that Switzerland is more dominant than other countries. Sweden and Netherlands follow this country in dominance of other countries, respectively. The final and complete ranking is obtained with the PROMETHEE II method.
The PROMETHEE II method, obtained through the program, performs the full ranking process between countries with the net superiority value (Phi) calculated by using negative (Phi−) and positive superiority (Phi+) values. These PROMETHEE II results show positive advantage value, negative advantage value, net superiority value and the ranking of countries. According to this analysis, Switzerland ranks first among other countries as the country with the highest net Phi value in terms of STI criteria. The top five countries following Switzerland are Sweden, Netherlands, Finland, Singapore and Denmark. According to the PROMETHEE II analysis, the total performance scores of the countries are shown in Table 11. In Figure 5, the representation of PROMETHEE II analysis on GAIA plane is presented. Alternatives (countries) in the GAIA plane are shown as points, and criteria as vectors. Also, the Decision Stick is indicated by π on the GAIA plane. The distribution of the countries to be listed in the GAIA plane is given in Figure 6. Among the countries that are tried to be listed, Switzerland, Sweden, the Netherlands and Finland are in the direction of the best compromise solution because they are in the direction of the decision stick. Mexico, Brazil and Turkey which are situated in the opposite direction of Switzerland are the countries in the worst position in the analysis. It can be said that countries located close to each other and clustered together on the GAIA plane have similar profiles in terms of STI criteria. Similarly, it can be said that the differences are large between countries that are far apart from each other on the plane. When evaluating in terms of criteria, criteria in the same direction are defined as compatible with each other, while criteria in the opposite direction are considered as opposite criteria. Based on the criteria on the GAIA plane and the positions of the countries, it is possible that decision makers may increase the number of comments made above. Showing the single criteria net flows of the countries together reveals the profiles of the countries. Countries that are close to each other in the plane are presented as examples in Figure 6 to show profile graphics in terms of criteria. It is seen that countries that are close to each other have similar profile graphics. The ranking of countries according to the MOORA ratio approach in terms of STI criteria is included in the Table 12. A ranking table is obtained from the best ranking country to the worst by ranking the y i * values obtained in the previous stage in descending order. According to the ranking, the country with the best condition in terms of STI criteria is Switzerland. The countries that rank among the top five countries and follow Switzerland are Sweden, Singapore, Finland and the United States, respectively. On the other hand, the last five countries are South Africa, Turkey, Greece, Mexico and Brazil. The final ranking of the countries obtained with the MOORA reference point analysis is shown in the Table 13. According to the MOORA reference point approach, the country with the best score in terms of STI is Switzerland. Along with Switzerland, the top five countries are Portugal, South Korea, Germany and the Czech Republic, respectively. The last five countries on the list are Singapore, Indonesia, Israel, United Arab Emirates and Qatar. According to the full multiplicative form of MOORA approach, the ranking of the countries and their scores are shown in Table 14.  According to the ranking, the country with the best value in terms of STI is Switzerland. Other countries in the top five are Sweden, Netherlands, Denmark and the United Kingdom, respectively. The five countries with the worst score are Turkey, United Arab Emirates, Indonesia, Mexico and Qatar. Table 15 presents the MULTIMOORA ranking obtained using the three MOORA methods.  In the ELECTRE method, firstly, the consistency matrix is obtained. After creating the consistency matrix, the inconsistency matrix is created. Then, the necessary evaluation matrix values for all cells are obtained. In the last step, the dominance values are calculated by collecting the row and column values of each country using the evaluation table. Finally, the values obtained by calculating the row and column difference for each country are listed in descending order. Thus, ELECTRE ranking of countries is obtained [76]. All of these values and the order are presented in Table 16.  According to the ranking, the country with the best value in terms of STI is Switzerland. Other countries in the top five are Sweden, the USA, Singapore and the United Kingdom, respectively. The five countries with the worst score are Russia, South Africa, Mexico, Indonesia and Brazil.
In the MAUT method, the country with the highest benefit value is the best country in terms of STI criteria. Therefore, country rankings obtained as a result of the MAUT analysis are the same as the rankings of the SAW analysis. Benefit values and country rankings of MAUT analysis are shown in Table 17, which was created by using Table 4. In the cluster analysis, the dendrogram structure of the countries was reached by analyzing the data belonging to 115 criteria and 10 dimensions by hierarchical clustering analysis which is one of the data mining classification methods. The dendrogram is shown in Figure 7.
With the evaluation of the dendrogram, it was decided to divide the countries into 3 groups with k-means clusters analysis. In this way, we have the opportunity to see countries that are similar in terms of evaluation criteria. Table 18 shows the number of cases in each cluster. Table 18. Number of cases in each cluster.   Table 19 presents the cluster memberships of countries according to K-means clusters analysis.  Table 20 lists the results of ranking using TOPSIS, VIKOR, PROMETHEE, ELECTRE, ARAS, COPRAS, SAW and MAUT. Although there are some differences in the rankings of MCDM methods, the rankings show significant consistency in general. MOORA reference point and full multiplicative form of MOORA are used to obtain MULTIMOORA ordering. Therefore, only the MOORA ratio and MULTIMOORA rankings are taken into account among the MOORA rankings.   Table 21 shows the rankings according to the Global Innovation and Global Competitiveness Indices. When the rankings in Tables 20 and 21 are evaluated together, it is seen that the rankings are quite consistent.

Number of Cases in each Cluster
Country comparisons and evaluations show that economic growth and the spread of a wealth system based on knowledge, transparency in management, sustainable infrastructure in technology and human resources education create significant differences between countries. The countries at the end of the rankings have very low values in these indicators.
Switzerland ranks first in the rankings for all methods. Sweden ranks second in all rankings except TOPSIS ranking, third in TOPSIS ranking. Singapore is in the top three in all rankings except PROMETHEE II and ELECTRE. It is in the fifth place in the PROMETHEE II ranking. According to the evaluation results in terms of the performance of STI indicators, the countries in the top ten by achieving the best performance scores are Switzerland, Sweden, Singapore, Finland, USA, Denmark, Netherlands, Germany, United Kingdom and Ireland, respectively. When the countries with low performance scores are evaluated, Brazil ranked last in the rankings obtained from TOPSIS, VIKOR, ARAS, COPRAS, SAW, MAUT and ELECTRE methods; In the PROMETHEE II ranking, it is second to last.
Mexico ranks last in the MOORA ratio approach, MULTIMOORA and PROMETHEE II rankings. When evaluating poor performing countries in general, South Africa, Greece, Turkey, Mexico and Brazil are at the last five in almost all rankings.
Also, the countries in the last ten order show significant consistency. According to the analyses made, countries in the last ten places are India, Thailand, Hungary, Indonesia, South Africa, Russia, Greece, Turkey, Mexico and Brazil. While the order of the ARAS and COPRAS methods is exactly the same, the order of the SAW and MAUT methods is exactly the same. In general, when all the rankings of all methods are evaluated, the country rankings are similar.
The countries with lower scores in the analysis and at the bottom of the lists have very low values in indicators that evaluate the trade volume, innovative policies to improve market share, competitive advantage, industrialization intensity, technological development of production processes, and the added value of all sectors, especially industry, technology exports and intellectual property rights. According to the values of indicators, in these countries, the private sector does not make the expected investment in terms of R&D. Most of the investments made by commercial enterprises also consist of public incentives. Therefore, it is suggested that companies should increase their R&D expenditures.

Discussion
The most important limitation of this study is the absence of a comprehensive database that would enable all countries or a significant majority to be evaluated together. In order to solve this problem, it is necessary to determine common indicators and to create a common database. The novelty of the study is that it both tries to show the current situation of 40 countries with relevant indicators and proposes an integrated decision support system to improve this situation. STI policy indicators include many different areas and are in conflict with each other. This feature shows that multi-criteria decision-making methods should be used in this field. The starting point of this study is the need for the STI policy framework and the need to use new methods in this field. Using MCDM methods and data mining cluster analysis together in this field is a novelty proposed by the study. In general, countries are tried to be evaluated with descriptive statistical methods in studies on a similar subject. It is seen that the results obtained by analyzing the 40 countries determined with 10 criteria and 115 criteria using 10 MCDM methods and cluster analysis are generally consistent. Subjective assessment is not used in any of the calculation processes of these MCDM methods.
When all these results are evaluated together, the results obtained from this study, which was carried out by taking into account more indicators and dimensions, are consistent with the results of the indices that conduct global research and evaluate with their own methodologies.
The PROMETHEE method is a little more advantageous than other methods in terms of visually evaluating both the similarities of countries and similar and different country groups in terms of indicator values. Also, the Multi-MOORA method gets a ranking based on the rankings made by three MOORA methods. Therefore it stands out a bit more than other methods in terms of reliability. In the overall assessment, the rankings of all methods are quite consistent.
Since MCDM methods such as the Analytic Network Process (ANP) and Analytic Hierarchy Process (AHP) are based on subjective evaluations, very different results can be obtained in different analyses for the same indicators. If even the criterion weights are determined by subjective evaluations, the results obtained by the same methods for the same indicators will be different. In this study, the MCDM methods and cluster analysis were used, with the criteria being objectively weighted and not requiring any subjective evaluation, and only processing and evaluating the criteria values. In other words, the evaluations were carried out in a completely objective framework. Due to the computational differences that the methods have, there are some differences in the rankings obtained. However, when the results obtained from all methods are generally evaluated, the results are consistent with each other and the other global evaluation rankings.
Determining and comparing the most developed countries in terms of STI, has been discussed in many studies. In the Introduction section of the study, MCDM studies carried out on this or similar subjects in the last five years were examined. Considering the number of studies, the methods they used, and their applications, it is seen that the application of this study brings a novelty to the field with its scope, methods, and the number of countries it includes. As a future study proposal, different methods may be applied to solve this problem and the outputs may be compared with this article. Also, it is recommended to repeat this study in the following years in order to see the performance of these countries in terms of STI policies.

Conclusions
The article evaluates 40 countries, mostly European, based on 115 indicators related to (STI) science, technology and innovation. In addition, while the article aims to demonstrate the current status of these countries, it also aims to offer an integrated decision support framework in order to contribute to this field. Indicators used in the application of study include very important topics in terms of sustainable economy and development of countries. The fact that a country has a bad score in terms of STI also means that the stability and sustainability of that country's productivity, development, market share, and employment are under threat.
With the evaluation of the dendrogram in cluster analysis, it was decided to divide the countries into 3 groups with k-means clusters analysis. In this way, we have the opportunity to see countries that are similar in terms of evaluation criteria. When the countries in the obtained clusters are evaluated, it is seen that the countries with similar scores in the rankings obtained by MCDM methods are together. Consequently, it is observed that the results of the clustering analysis from the data mining classification method with the MCDM rankings show quite consistency.
Among the top ten countries in the overall assessment in Table 20, only Ireland is not among the top ten countries in the list of the Global Innovation Index. Ireland ranks twelfth on this list. Switzerland and Sweden are again in the first two places of these rankings, and they show consistency with the results of the study. When we compare the Global Competitiveness Index with our analyses results, all countries in the top ten, except Ireland, are among the top ten countries in this index. Japan, which is in the top ten in this index, generally ranks 11th or 12th when the results of the MCDM methods are evaluated. When the countries that show poor performance in terms of STI indicators are evaluated according to the results of the study with the rankings of these methods, all of the ten countries with the worst performance, except Hungary, are also in the last ten in the Global Innovation Index. According to the list of the Global Competitiveness Index, all of these countries except Thailand are among the last ten countries. Slovakia, which is among the last ten countries in the list instead of Thailand, is ranked 11th from the last in the results of the study, while Thailand is ranked 11th from the last in the index ranking.
It is seen that Northern European countries such as Sweden, Finland, Denmark, United Kingdom, Ireland and Norway come to the fore in the rankings based on STI indicators. These countries are generally among the top ten countries in all rankings. Therefore, this region seems to be ahead of other regions in terms of STI performance and policies in the global sense. Also, Switzerland is the leading country in the rankings. The Netherlands and Germany are other European countries that rank in the top ten. In general, the countries in the top ten and not in the European continent are the United States of America and Singapore.
When the results of the study and other global indices are evaluated, it can be said that the European continent is ahead of other regions in terms of STI policies and performances. These countries have very high values compared to other countries, based on the following indicator values used in the study and normalized by compressing between 0-1: the quality of the education system, competitive advantage, environmental performance, innovation capacity, ICT and business model creation, GDP per capita, R&D expenditure of companies, access and use of ICT, regulatory environment, patent applications, political stability and absence of violence/terrorism, participation and accountability, employment in information-intensive services, university-industry cooperation in research and development, accessibility to the latest technologies, R&D expenditures of the first three global companies, quality of mathematics and science education, economic cluster development.
When the last ten countries of the rankings are evaluated, there is no specific region or continent that stands out. However, in the rankings, the same countries are generally in the last rows. These countries are India, Thailand, Hungary, Indonesia, South Africa, Russia, Greece, Turkey, Mexico and Brazil. These countries have very low values in terms of these following indicators used in the study and normalized by compressing between 0-1: creative goods export, unemployment, researchers, total gross R&D expenditure, innovation, competitive advantage, high technology exports, renewable energy consumption, international scientific cooperation, internet access in schools, ICT and business model creation, ICT services export, investment, average monthly net income, participation and accountability, political stability, patent applications, added value of the manufacturing sector, high technology and medium high technology production, university-industry cooperation in R&D, and quality of scientific research institutions. Furthermore, these countries perform very low performance in terms of education. These countries have insufficient education expenditures per student; students obtained low PISA scores in reading, mathematics and science; they do not have the necessary trained workforce in the field of science and engineering; it is observed that the education system and universities' quality evaluations at the global level cannot meet the necessary criteria and get low scores.
Almost all of the countries at the bottom of the rankings perform poorly on issues such as GDP per capita, real GDP growth, average monthly net income, unemployment, and female labour force participation. Consequently, these countries should produce policies that will give priority to the fields of administration, development of human capital: education, finance and market development, R&D investment and research workforce, which are the criteria with the largest entropy weight in the study. The results of the study shows that societies and management approaches should be more inclusive. The study also covers sustainable environment and renewable energy issues with 16 indicators in the dimension of 'energy, mining and green technology infrastructure'. It emphasizes the need for a sustainable productivity and efficiency policy by addressing education, creative sectors and R&D. Thus, it has the vision to shape the future with its assessment dimensions.