{'fraction_non_alphanumeric': 0.031512926418846565, 'fraction_numerical': 0.015029830197338228, 'mean_word_length': 4.419481865284974, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Robust DEA efficiency scores: A probabilistic/combinatorial approach

In this paper we propose robust efficiency scores for the scenario in which the specification of the inputs/outputs to be included in the DEA model is modelled with a probability distribution. This proba- bilistic approach allows us to obtain three different robust efficiency scores: the Conditional Expected Score, the Unconditional Expected Score and the Expected score under the assumption of Maximum Entropy principle. The calculation of the three efficiency scores involves the resolution of an exponential number of linear problems. The algorithm presented in this paper allows to solve over 200 millions of linear problems in an affordable time when considering up 20 inputs/outputs and 200 DMUs. The approach proposed is illustrated with an application to the assessment of professional tennis players.


Introduction
Data Envelopment Analysis (DEA), as introduced in Charnes et al. [1978], assesses relative efficiency of decision making units (DMUs) involved in production processes. DEA models provide efficiency scores of the DMUs in the form of a weighted sum of outputs to a weighted sum of inputs. These scores are the result of an evaluation of each unit within a technology which is empirically constructed from the observations by assuming some postulates such as convexity, constant or variable returns to scale and free disposability. The selection of the inputs and outputs to be considered in the analysis provides a description of the underlying technology, thus becoming one of the key issues of model specification in DEA. In practical applications, the prior knowledge and experience may lead the analyst to select some variables considered as essential to represent this technology. However, as discussed in Pastor et al. [2002], there are often other variables whose inclusion in the model the analyst is not always sure about. This situation can be addressed in different ways. The methods for the selection of variables constitute an important body of research to deal with this issue. The idea is to complement the prior knowledge and experience with information provided by the data, so that these methods may help make a decision about the candidate variables. In this line, the F-tests developed by Banker [1993,1996] and those in Pastor et al. [2002] allow to statistically evaluating the role of inputs/outputs. These tests are empirically analyzed (and compared among themselves) with simulations in Sirvent et al. [2005]. See also Natajara and Johnson [2011], which provides comparisons between some of the methods widely used to guide variable selection: the tests in Pastor et al. [2002], Principal Component Analysis (PCA-DEA), regression-based tests and bootstrapping. To the same end, Wagner and Shimshak [2007] propose a stepwise approach. In Li et et al. [2016] it is proposed a method based on the Akaike's information criteria (AIC), which mainly focuses on assessing the importance of subset of original variables rather than testing the marginal role of variables one by one as in many other methods.
Correlation either between efficiency scores and variables (for their incorporation into the model) or between variables (in order to remove redundant factors) has also been used for the selection of variables, although it has been widely deemed as a criterion of limited value. See Jenkins and Anderson [2003] for discussions. This latter paper proposes instead an approach based on partial covariance. Eskelinen [2017] compares the approach in Jenkins and Anderson [2003] with that in Pastor et al. [2002] in an empirical retail bank context. Unlike previous research, Edirisinghe and Zhang [2010] develop a method for the selection of variables that employs a reward variable observed exogenous to the operation of DMUs. See also Luo et et al. [2012], which uses the concept of cash value added (CVA) for choosing variables.
A different approach, which is the one we follow in the present paper, is based on efficiency scores which are robust against the selection of the variables, while at the same time taking into account the inherent uncertainty on the inclusion of some inputs/outputs in the models. In the literature, some authors have undertaken some exploratory work to examine the robustness of results as they relate to the production function specification. Roll et al. [1989] present one such study in which ten different combinations of outputs are tried out with three inputs to evaluate the efficiency of maintenance units in the Israeli Air Force; Valdmanis [1992] tests ten different specifications of a hospital efficiency model, and Ahn and Seiford [1993] test four different variable sets for each of the DEA models considered in their study on higher education sector efficiency.
For his part, Smith [1997] investigates the implications of model misspecification by means of a simulation study. See also Galagedera and Silvapulle [2003] for another simulation study with large sample which, in particular, analyze the effects of omission of relevant input and inclusion of irrelevant input variables on technical efficiency estimates.
The approach we propose goes one step further than the exploratory work done in the papers just mentioned, in the sense that we not only examine the efficiency scores that result from several combinations of inputs and outputs but we take into account all the scenarios associated with all of the specifications of inputs and outputs that could be considered once a given set of candidate variables is determined. In addition, we allow for the uncertainty regarding the inclusion of variables in the DEA models. The idea is the following: The inclusion of an input/output in the set of selected variables is modelized here through the probability of that variable being considered in the DEA model. For example, if such probability is 0.8, this could be interpreted as saying that 80% of experts would include the corresponding variable in the DEA model. As a result, each specification of the inputs and outputs to be included in the DEA model has a probability of occurrence and, therefore, the efficiency score of a DMU would be a random variable, which takes as values the DEA efficiency scores associated with each specification of inputs and outputs with some probability.
The robust efficiency score of a given DMU is then defined as the expected value of that random variable.
The consideration of all combinations of inputs/outputs gives rise to an exponential number of problems that must be solved. To solve such large number of problems, an efficient algorithm is needed. In this paper an exact algorithm is developed, which allows us to solve over 200 millions of linear problems when considering up 20 inputs/outputs and 200 DMU's. This algorithm reduces the time and the number of problems to solve in half, approximately.
We illustrate the use of the proposed approach with an application to the assessment of professional tennis players. The Association of Tennis Professionals (ATP) assesses players through the points earned in the different tournaments they play during the season. Therefore, ATP assesses the competitive performance of players. However, ATP also provides statistics regarding their game performance. For instance, its official webpage reports data regarding 9 game factors such as the percentage of 1st serve points won or the percentage of return games won. Obviously, it would be interesting to have available an index of the game overall performance of players that aggregates into a single scalar the information provided by the statistics of the factors that are considered. The DEA approach we propose provides a score of the player game performance which is robust against the selection of game factors that is considered for the analysis. Ruiz et al. [2013] also deal with the assessment of game performance of tennis players, but with an approach based on the cross-efficiency evaluation that consider all of the 9 game factors available in the ATP statistics.
The paper is organized as follows: In Section 2 a short introduction of DEA, through the original CCR (Charnes, Cooper and Rhodes) model, is presented.
In Section 3 we define the the robust efficiency score, and Section 4 presents three robust DEA efficiency scores for a probabilistic specification of the inputs and outputs. The exact solution algorithm used for the calculation of the robust scores is described in Section 5. In Section 6 the proposed algorithm is used for obtaining the robust scores in a case study. Finally, some conclusions and outlines for future work are given in Section 7.

DEA Efficiency Scores
In a DEA efficiency analysis, we have n DM U s which use m inputs to produce s outputs. Each DM U j can be described by means of the vector (X j , Y j ) = (x 1j , . . . , x mj , y 1j , . . . , y sj ), j = 1, . . . , n.
As said before, the DEA models assess efficiency with reference to an empirical technology or production possibility set which is constructed from the observations by assuming some postulates. For instance, if we assume convexity, constant returns to scale, and free disposability (which means that if we can produce Y with X, then we can both produce less than Y with X and Y with more than X), then it can be shown that the technology can be characterized as the set The original DEA model by Charnes et al. [1978], the CCR model, provides as measure of the relative efficiency of a given DM U 0 the minimum value θ 0 such that (θ 0 X 0 , Y 0 ) ∈ T . Therefore, this value can obviously be obtained by solving the following linear programming problem.
which is the so-called primal envelopment formulation of the CCR model. Thus, DM U 0 is said to be efficient if, and only if, its efficiency score equals 1. Otherwise, it is inefficient, and the lower the efficiency score, the lesser its efficiency.
The model in Banker et al. [1984], the so-called BCC model, is that resulting from eliminating the constant returns to scale postulate and allowing for variable returns to scale in the production possibility set. Its formulation is the linear problem resulting from adding the constraint n j=1 λ j = 1 to (1).

Robust DEA Efficiency Scores: A probabilistic approach
Throughout the paper we suppose that we have a set of candidate variables to be included in the efficiency model (1), C = {z 1 , . . . , z q }, which can be either inputs or outputs. It is assumed that the probability of including in (1) a given candidate variable, say z c , is p c . Thus, the inclusion of each of the variables in C into (1) can be determined by means of some independent random variables B c distributed Bernoulli, Be(p c ), c = 1, . . . , q. As a result, all the scenarios associated with all the possible specifications of (1) are determined by the random vector B = (B 1 , . . . , B q ). If we denote by p = (p 1 , . . . , p q ), then the probability distribution of B is Let θ b 0 be the efficiency score of DM U 0 provided by (1) when the specification of the model is determined by b ∈ {0, 1} q . We denote by Θ 0 the random variable which takes the value θ b 0 . Then, the expected efficiency score of DM U 0 is For consistency, we define θ These expected values can be seen as DEA efficiency scores which are robust against the selection of variables that is made for the efficiency model.

The specification of p
The key for obtaining the expected efficiency scores (2) is in the specification of p. Three different approaches to deal with this issue are proposed below.

Using expert opinion
The probability of selection of candidate variables can be determined by using information from experts, if available. The values p c s can be set reflecting the personal belief of a given expert regarding the importance to be attached to the corresponding variables z c s in the underlying production process. Alternatively, these probabilities can be estimated. If several experts are asked to give their opinion about whether or not to include a given z c in (1)

Maximizing the Entropy
The definition of entropy was introduced by Shannon [1948]. In the probabilistic context, the entropy H(p) is a measure of the information provided by p, where high values of entropy corresponds to less information o more uncertainty, i.e., the maximum principle Entropy is used in Statistics to obtain a value for the parameters with the least informative distribution assumptions. If information from experts is not available and the probabilities of selection of candidate variables are unknown, we can obtain the value of p that maximizes the entropy in the context of the approach previously set out for providing robust DEA efficiency scores.
The Entropy function associated with the discrete random variable Θ 0 is: Lemma 4.1. [Guiasu and Shenitzer [1985] and Conrad] Suppose that a random variable X takes exactly values with positive probability. Then H(X) ≤ log .
Proof. Applying Lemma 1, it is sufficient to prove that H(p * ) = log 2 q , because 2 q is the number of possible realizations of variable Θ 0 (that is, the number of scenarios determined by all the possible selections of inputs/outputs).
This corollary shows that the average of efficiency scores across all the scenarios resulting from all the specifications of model (1) is the one associated with the specification of p that maximizes the entropy.

A Bayesian approach
In this subsection we develop a Bayesian approach as an alternative for the specification of p when it is unknown. This means that the probabilities of selection of candidate variables are assumed to be random variables in [0, 1].
Denote by P = (P 1 , . . . , P q ) the random vector consisting of the independent random variables associated with the probability of selection of each of the candidates. Let f be the joint probability density function of P, which can be expressed as We need to introduce the following two elements for the subsequent developments Definition 4.1. The unconditional probability function of Θ 0 is defined as Alternatively, the unconditional probability function of Θ 0 can be reexpressed as That is, P (Θ 0 = θ b 0 ) can be seen as the expected conditional probability to p.
Definition 4.2. The unconditional expected efficiency score of Θ 0 is defined as The following two propositions hold Proof. It follows directly from (6).
In a Bayesian approach, the probabilities are often assumed to follow a beta distribution. In that case, the unconditional expected efficiency score is the following Proposition 4.4. If P c follows a beta distibution, β(α c , γ c ), c = 1, . . . , q, then Proof.
Corollary 4.2. If P is a random vector consisting of independent random vari- Note that in this case the unconditional expected efficiency score is again the average of efficiency scores across all the scenarios resulting from all the specifications of model (1), like the maximum entropy expected efficiency score.
It can also be considered the case in which we do not distinguish between the probabilities of selection associated with all the candidate variables. If we assume P 1 = · · · = P q = P to follow a distribution U [0, 1], and we denote by E U (Θ 0 ) the unconditional expected efficiency score in that case, then (4.2) and (4.3) become in the unconditional expected efficiency score presented in the next results: Corollary 4.3. If P 1 = · · · = P q = P follows an uniform distribution U [0, 1], then the unconditional expected score of Θ 0 , E U (Θ 0 ), is the expected of the conditional expected score E(Θ 0 |P).
Note that in this case the weight attached to each score θ b 0 depends on the 1-norm of b. It exists a relationship between the probability P 1 = · · · = P q = P and the probability of each subset of C when the value of the the 1-norm of b is fixed. The unconditional efficiency expected score E U (Θ 0 ) is the weighted average of efficiency scores where the weight attached to each of them is given by the number of all specifications with the same number of inputs/outputs.

Summing up
To end this section, we summarize the results obtained. We have proposed a probabilistic/combinatorial approach that provides DEA efficiency scores which are robust against the selection of inputs/outputs to be included in the model. This approach considers all the scenarios associated with the possible specifications of the DEA model together with their probabilities. The key is obviously in the specification of such probabilities. If they can be set by using expert opinions, then the DEA efficiency scores will be obtained as in (2). Otherwise, if information from experts is not available, then an interesting choice is 1 2 q b∈{0,1} q θ b 0 , that is, the average of DEA efficiency scores across all the scenarios. The developments have shown that this score is that which results both when p is set by maximizing the entropy and when a Bayesian approach is adopted by assuming the probabilities of selection of candidates as independent random variables distributed uniform in U [0, 1]. In particular, the entropy is maximized when the probabilities of selection of candidates are all equal to one half. Having a look at (9), we realize that E U (Θ 0 ) somehow summarizes the values E p (Θ 0 ) across the different values of a common p (with p = 1/2 among them).

Algorithm
In order to compute the efficiency scores we have defined for each subset of inputs/outputs b ∈ {0, 1} q and for each DMU, we need to solve (2 q − 1) · n problems, where n is the number of DMUs to evaluate. To represent the subsets of C, we consider that C is ordered and use a list of q binary numbers. If q = 4, the number 1001 represents the subset with the first and the last elements of the ordered set C. Figure 1 shows the exploration tree, where each node in the tree represents one of the 2 q = 16 problems, and only 4 inputs/outputs have been considered.
Each node is enumerated according to the order of exploration. For example, node 12 will be explored after node 11 and corresponds to the computation of the score when b = {0101}. In order to reduce the number of problems to calculate, we propose the following enumeration algorithm for each DM U 0 , see Figure 2. The main contributions of the algorithm are summarized in three points: 1. The set of inputs/outputs C is sorted in such a way that the scores of the first nodes are hopefully large. If an score equals the upper bound given by θ C 0 = θ b=(1,··· ,1) 0 , then it is not necessary to continue the exploration.
Step 1.3 of the algorithm.
2. At each iteration, the solver uses the optimal solution of the ancestor node in the tree as a feasible solution. Each solution in the exploration tree is a feasible dual solution for the next problem. Running the dual simplex algorithm with start basic solution allows to reduce the computational time.
Step 3 of the algorithm.
3. At each iteration, the resulting score is compared with the score from set C. If both are equal, the exploration tree is cut.
Step 5.1 of the algorithm.
We have tested the algorithm using randomly generated instances. We have used a uniform distribution in [50, 100] to generate the inputs/outputs values.
The results are shown in Table 1 avoids to solve more than half of problems. Nevertheless, we can see in Table 1 that computational time required by our approach is very low when the number of input/output candidates is no larger than 10. This means that, in many of the situations we usually find in practice, computational time will not be a limitation in order to yield robust DEA efficiency scores (and, consequently, in those cases, the use of the algorithm will not provide any important benefit in terms of time).
Our experiment was conducted on a PC with a 2.5 GHz dual-core Intel Core i5 processor, 8 Gb of RAM and the operating system was OS X 10.9.

Combinatorial DEA Efficiency Scores: A case study
To illustrate the approach proposed, we apply it in an example on the assessment of profesional tennis players. The Association of Tennis Professionals (ATP) provides statistics of the game performance of players, in particular regarding the game factors reported in Table 2. there is no reference to resources consumed. We only include in the models a single constant input equal to 1, which means that every player is doing the best for playing his game. Thus, we will be actually evaluating the effectiveness of player game performance instead of efficiency. It should also be noted that, in the case of having one constant input, the optimal solutions of (1) satisfy the condition n j=1 λ j = 1. Therefore, in these special circumstances, the specification of returns to scale is not particularly relevant (see Pastor [1995, 1999] for details and discussions).  We present the results obtained by distinguishing between whether information from experts is available or not.
Suppose that some experts are asked to give their opinion about whether or not to include each of the factors listed in Table 2 in a DEA model aimed at providing measures of effectiveness of the game performance of players. We might have observed that all the experts agree in considering y 4 and y 9 , 80% out of them would include y 2 , y 3 , y 5 , y 6 , y 7 and y 8 , while only 40% would be in favor of incorporating y 1 . This would be showing that the most important game factor in the opinion of experts are those that have to do with winning games (y 4 and y 9 ), whereas y 1 , which does not have a direct influence on the result of the matches, is viewed as a less relevant game factor. With such information from experts, the probabilities of selection of candidates can be estimed as follows: p 1 = 0.4, p 2 = 0.8, p 3 = 0.8, p 4 = 1, p 5 = 0.8, p 6 = 0.8, p 7 = 0.8, p 8 = 0.8 and p 9 = 1. The efficiency scores (2) associated with that specification of the probabilities (together with the corresponding standard deviations) are recorded in Table 4.
We can see that the top 3 players in the ATP ranking also achieve the maximum rating when evaluated with the robust DEA efficiency scores: Djokovic, Federer and Nadal. Note, however, that Isner, which is also scored 1, occupied the 19 th position in the ATP ranking. This shows that the assessment of Isner is better when game performance instead of competitive performance is evaluated, for this specification of p. In contrast, this analysis reveals that other players with lower efficiency scores, like Nishikori (0.961), perform better in competition (he ranked 5 th ).
As discussed in Section 4, if information from experts is not available, the scores E e (Θ 0 ), which coincide with E U (Θ 0 ), and the scores E U (Θ 0 ) may yield useful information. These two scores are reported in Table 5, together with the corresponding standard deviations (which are somewhat larger than the standard deviations of the scores in Table 4. As said before, E U (Θ 0 ) somehow summarizes the values E p (Θ 0 ) across the different values of a common p. Table 6 reports the scores E p (Θ 0 ) for some values of p. This information can complement that provided by Table 5. Note, in particular, that the scores E e (Θ 0 ) are actually the ones under column p = 0.5 in Table 6.
Table 5 also shows that the top 3 players of the ATP ranking are the ones with the larger values of E U (Θ 0 ): Djokovic (0.9899), Federer (0.9855) and Nadal (0.9934). In order to provide a graphical display of how the scores change as p varies, we have depicted graphically the values in the rows of Table 6 corresponding to these three players (Figure 3). We can see that Nadal outperforms Djokovic and Ferderer irrespective of the value of p. Frequently, the scores E p (Θ 0 ) are larger with p. Note that a low p implies a low probability of input/output selection, also for the inputs/outputs that may benefit the player under assessment, while if p is high, the best factors for each player will be in the DEA analysis with high probability.
We have also analyzed the case of Isner, in particular against that of Mayer. Figure 4 shows that, while the scores of Mayer remain stable for different values of p, these are quite smaller for low values of this probability. For instance, Isner, which has a very specialized game based on his service, is penalized when p is low because p is the probability of selecting his best factor, the factor y 1 .

Concluding Remarks
We have proposed a probabilistic/combinatorial approach for the assessment of DMUs in DEA by using efficiency scores which are robust against the selection of inputs/outputs. Robust efficient scores are defined on the basis of the information from expert opinion, by using the entropy principle and following a bayesian approach. Computing these scores requires solving an exponential number of linear problems. In order to do so, we have developed an exact algorithm that reduces approximately half the time and the number of problems required. The method has been presented within a conventional framework wherein efficiency is assessed by means of the classical radial DEA models.
Obviously, the approach proposed can be extended for use, for example, with external factors (non-controlled variables) whose incorporation into the analysis is to be considered, provided that an appropriate efficiency model is chosen in order to deal with such type of inputs/outputs (see Aristovnik et al. [2013] and Aristovnik et al. [2014] for some empirical work that considers external factors).
In that sense, the extension of the method for use with other DEA models is also straightforward, because it only need from them the efficiency scores they provide. In particular, we can derive robust efficiency scores from non-radial DEA models (by using, for example, the enhanced Russell graph measure in ), which would be of special interest when models that minimize the distance to the efficient frontier are used (see Aparicio et al. [2013]). In the latter case, the development of an heuristic algorithm for reducing compu-tational time would become a key issue, due to the complexity of the efficiency models involved.