{'fraction_non_alphanumeric': 0.023411051393378346, 'fraction_numerical': 0.010634431842090163, 'mean_word_length': 4.626170235661251, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Characterisation of Grasp Quality Metrics

Robot grasp quality metrics are used to evaluate, compare and select robotic grasp configurations. Many of them have been proposed based on a diversity of underlying principles and to assess different aspects of the grasp configurations. As a consequence, some of them provide similar information but other can provide completely different assessments. Combinations of metrics have been proposed in order to provide global indexes, but these attempts have shown the difficulties of merging metrics with different numerical ranges and even physical units. All these studies have raised the need of a deeper knowledge in order to determine independent grasp quality metrics which enable a global assessment of a grasp, and a way to combine them. This paper presents an exhaustive study in order to provide numerical evidence for these issues. Ten quality metrics are used to evaluate a set of grasps planned by a simulator for 7 different robot hands over a set of 126 object models. Three statistical analysis, namely, variability, correlation and sensitivity, are performed over this extensive database. Results and graphs presented allow to set practical thresholds for each quality metric, select independent metrics, and determine the robustness of each metric,providing a reliability indicator under pose uncertainty. The results from this paper are intended to serve as guidance for practical use of quality metrics by researchers on grasp planning algorithms.


Introduction
Many grasp quality metrics have been proposed in the robot grasping research literature. A recent survey identifies up to 24 different grasp quality metrics designed to quantify the goodness of a grasp [31]. Quality metrics play a principal role in the so called analytical approach to the grasp planning problem, also often referred as grasp synthesis. Grasp planning consists in determining the finger contact locations on an object surface and the appropriate gripper configuration which allow the constraints, forces and torques produced on the object through the contact points to reach one or more desirable properties like dexterity, force-closure, stability or equilibrium and other which has been defined in the literature [4,36,42].
Analytical approaches try to solve the grasp planning problem by using kinematics and dynamics formulations [36]. There exists a vast literature focusing on the development of algorithms to find grasps under a variety of assumptions regarding dimensionality, contact models, object shapes and others [4,36,42]. Often the proposed solutions produce a large number of possible grasps. Quality metrics are the tools that allow these sets of solutions to be ranked. Actually, metrics play a triple role: allowing relative comparisons between grasps, providing an absolute quantitative assessment of the goodness of a grasp, and, finally, serving as evaluation functions in grasp synthesis optimization algorithms [28].
In recent years, though more attention has been paid to alternative data-driven and experimental approaches for grasp planning, many of them still rely on analytical algorithms as an important part of their pipelines. A recent survey on data-driven grasp synthesis [5] points out that most methods assuming a previously known model of the objects perform an offline initial analysis of the possible grasps for each object using analytical techniques, being quality metrics a critical tool to rank grasps.
A limitation of the use of metrics is that each of them is designed to assess a specific aspect of a grasp. Recently, Roa and Suárez [31], in a detailed revision of existing grasp metrics, have distinguished between metrics associated with the contact points and those associated with the hand configuration. The former ones are the most populated, being subdivided into metrics based on the properties of the Grasp matrix, metrics based on geometric relations, and those that consider limitations of the applied forces.
They also reviewed several attempts to define global metrics by combining existing metrics. Combined metrics can be a solution to overcome the specificity of each one. An early correlation study of several quality metrics showed the existence of at least five independence dimensions on the evaluation of human grasps [23]. Roa and Suárez identify two trends for building combined indexes, serial and parallel. On the serial approach one metric is used to generate and select a subset of grasp configurations, then a second metric is used to rank them [16]. A most common approach is the parallel combination of metrics. In this approach every grasp is evaluated by a set of metrics, and the values obtained are added to produce a unique evaluation index [1,6]. Different normalization procedures can be applied to every metric for their combination, as well as different weighting coefficients [7].
But combined indexes are not free of limitations. First, each combinatorial method is still based on a subset of quality metrics, and there are no clear criteria on how to choose them. Some metrics might be measuring the same property while others capture different underlying properties. And second, there is no obvious way of merging metrics since they do not provide values on the same range not even on the same physical magnitudes.
The main goal of the work described in this paper is to answer the relevant questions about practical use of grasp quality metrics. For each metric, which are the practical ranges within a grasp that can be considered good or bad? How sensitive is the evaluation of a grasp with respect to contact location uncertainty? And, which quality metrics are similar or measure different grasp aspects?
To address these questions the paper describes a numerical study of the characteristics of a set of ten metrics selected among the most commonly used, with the purpose of deepening the knowledge on them. This provides factual data to decide about which metrics to select and how to combine them. The assumed focus of this work is on metrics oriented to the analysis of static grasps performed by a robot gripper on rigid 3D objects for holding and transporting them.
The contribution of this work is a set of numerical ranges and thresholds which establish the practical parameters that can be used to interpret the grasp evaluation provided by the selected quality metrics. This work is a complete extension and revision of results presented in [21,22]. As such, the methodology has been completed, revised and improved, and more models of objects has been added to obtain more general results. The specific novelties of this work are the comparison of different grasp generation methods and how they influence the quality measurements, and the extension of the hand and objects set. Using different shaped objects and hands with more varied characteristics, removes the bias of previous studies. As a consequence, some of the results and conclusions of the previous studies have been better established.
The methodology followed in the paper consists in the statistical analysis of the quality values obtained for 10 different metrics used to evaluate the grasps of a wide database, generated on a simulation environment using 126 different models of rigid objects and 7 models of robot hands. The study contains three different analyses: -Variability analysis: The distribution of each metric values is studied when grasping different objects with different robotic hands and with different grasp configurations. The range of values for each metric is established and used to homogenise the scales of the different metrics and to set practical thresholds for good and bad grasps (see Section 4). -Correlation analysis: The underlying relations between metrics are determined (see Section 5). -Sensitivity analysis: Metrics are recomputed when displacements and rotations are applied to grasp postures. The robustness of each metric with respect to inaccuracies in the placement of the finger contact points is determined (see Section 6).
The paper is structured as follows. Section 2 describes the selected metrics. Section 3 details the software platform used to carry out the experiments. Sections 4 to 6 explain the analyses and show their results. Finally, Section 7 discusses the results obtained and Section 8 presents the conclusions drawn and outlines future research lines.

Grasp Quality Metrics
The taxonomy proposed by [31], which includes a review of 24 metrics previously proposed in the robotics literature, is used for selecting the metrics for this study. Metrics are classified into 4 categories depending on their underlying principles. Object and hand constraints are considered in the evaluation through the contact points, using either the Grasp Matrix G, the location of the contact points, or the contact forces that should be applied to resist external disturbances. Additionally, hand constraints are also considered through the position of the hand finger joints or through the measurement of the capability to move an object in any direction with the same gain.
Ten quality metrics are selected based on several criteria, Table 1 presents a summary of the metrics which notation is defined in Table 2. First, task oriented metrics are discarded since this study aims to a general grasp evaluation, and the methodological complications of considerings different tasks would make the study too extensive. Second, variety in the underlying principles of the metrics is preserved. As can be seen in Table 1 up to four categories of metrics are considered. And third, those most commonly used are included.
In an attempt to make them comparable, their definition is adapted; they are normalised considering their maximal and minimal values, so that the best value is 1, and the worst value is 0. A more detailed description of each metric can be found in the Appendix A.

Simulation Framework
The simulation platform chosen to perform the experiments is OpenHand [20], a simulation toolkit developed by the authors in which the selected quality metrics are implemented. It is based on OpenRAVE [10], an open architecture targeting a simple integration of simulation, visualisation, planning, scripting and control of robot systems.

Robotic Hands
The characterisation of the selected quality metrics is performed by simulating the grasp of seven robot hands with different number of fingers and DOF, including common manipulators used in robotics and industrial applications, as well as a prosthetic hand. The selected hands are: the PR2 Hand [44], the Barrett Hand [3], the Schunk SDH Hand [39], the Schunk SAH Hand [39], the Shadow Hand [41], the Model T [46] and the Michelangelo prosthetic hand [29]. Details of each hand are provided in Table 3.

Objects
A wide variety of objects commonly used for everyday tasks are selected for this study: 125 objects from the KIT object database [8], with the addition of a simple sphere. Objects include cereal boxes, drink bottles, glasses, toys, containers for food, figurines, packed food, etc (Fig. 1).

Q C1
Smallest maximum wrench to be resisted [13,19] min w∈CW w / Volume of the convex hull [26] Volume(CW )/Volume max 0 1 Group D: Configuration of the manipulator

Grasp Generation
In pursuance of obtaining a significant set of candidate grasps applicable to an object, a large set of grasp hypotheses needs to be generated and evaluated. A grasp hypothesis is defined as the combination of an object, a hand, an initial position of the hand with respect to the object, and the initial configuration of the hand joints. Generation of a variety of grasp hypotheses for a given object with a given hand is performed using the Grasping Module from the Database Generators available in OpenRAVE [9]. This algorithm provides a parametric approach to generate a number of grasps distributed over the object surface. Among the different algorithms to generate candidate grasps we have chosen this because a number of reasons: it is able to produce an almost uniform distribution over the object surface; it can be tuned easily to produce more or less candidates; it starts from the bounding box of the shape, which limits the candidates to grasps able to approach the object with a gripper; and last but not least, its implementation is fully available.
In short, the algorithm generates a square grid of points uniformly distributed around the object bounding box and projects it over the object surface (Fig. 2). From each of these points, an approach ray is created pointing outwards the object surface. The hand is placed along the approach ray, facing the object at a given distance, and it can be rotated about the approach ray by a given angle.
The algorithm depends on five parameters ( Fig. 3): δ: Distance between the points in the square grid around the surface of the bounding box to place the approach rays. -α: Angle between the approach ray and the normal to the object surface (default value of zero). Depending on the value of α, a set of oblique approaching rays is generated. -θ : Standoff distance along the approach ray where the reference point of the hand is placed. -N r : The angles to rotate the hand (roll) about the approach ray. -H J : Configuration of the hand joints (preshape) before the closing algorithm starts.
Using these parameters, a large variety of grasp hypotheses are generated, and subsequently evaluated with the algorithm explained in the next section. The size of the objects affects the number of hypotheses generated, varying from a few hundreds to dozen of thousands. Section 4 analyses the selection of these parameters. The grasps generated in this study and the result of their evaluation have been made available in an online-repository in GitHub [33]. Maximum angle limits of joint i G J Grasp Jacobian matrix distance max Maximum distance from the object's centre of mass to any point in the object's contour Area max Maximum possible area of the hand calculated as the area of the polygon when the hand is fully opened V olume max Maximum volume of the convex hull of the primitive wrenches

Grasp closure and evaluation algorithm
The OpenRAVE closure algorithm [9] is used for each grasp hypothesis. This algorithm loads the initial hand configuration and sets the joint values according with the information of the grasp hypothesis. Then, it starts to close all of the joints at a constant velocity. When a collision between the object and the hand model is detected, the closure of the proximal joints stops. The remaining joints keep closing until all the fingers have contacted either the object or the hand, or have reached their joint limits. Although it is possible to define different closure velocities and accelerations for every hand joint, the same constant speed is used in this work for all joints.
Once the hand is closed over the object, the contact points are determined and used for the evaluation of the grasp. In the first place, the force-closure condition is evaluated. If the grasp does not meet this condition, it is discarded; otherwise, the quality metrics for this grasp are evaluated. The friction coefficient used had a constant value of 0.4. As most of the metrics are independent of this parameter, we consider adding variations of this value could lead a misunderstanding when comparing the correlations between metrics. The value was selected as it is the most common friction coefficient for human grasping, as pointed in [38]. This value is kept constant through all the experiments shown in this paper.
Quality metrics are implemented in Python using the Scipy and Numpy libraries [40], requiring a matrix of contact points and the information of the hand and object used as inputs.

Variability Analysis
The distribution of the values for each of the selected metrics is studied in this section. This is relevant, as previous studies suggested that these values tend to be contained in very narrow ranges [21][22][23]. In this section, the range of values for each metric is experimentally established and subsequently used for its normalisation, so that all the metrics have a more homogeneous distribution in the range [0,1]. Also, the distribution is used to set practical thresholds for good and bad grasps.

Methods
The methodology consists in generating and evaluating a sufficiently large amount of grasp hypotheses for every pair object-hand on the simulation environment. For each of these selected grasps, the values for each quality metric are obtained and statistically analysed. With these results, appropriate thresholds are selected to normalise each metric in the range of [0,1].
A proper selection of the values for the parameters presented in Section 3.4, especially for δ and α, is required to generate an appropriate set of grasps hypotheses for every pair object-hand. Although [2,32] showed a high efficiency when using orthogonal approach angles for grasp success, considering only The two links in each finger are associated to the same actuator. When a contact is detected, the closure in that finger stops. BarrettHand 6 4 8 The two links in each finger are associated to the same actuator. When a contact is detected in the proximal link, the closure in that link stops but it continues closing the distal links. It has one actuator for the symmetric abduction of the two nearby fingers. Schunk SDH 6 7 8 It has one actuator for each link. When it detects a collision in the proximal link, it also stops the closure of the distal link. It has one actuator for the symmetric abduction of two nearby fingers. Schunk SAH 12 13 13 It has one actuator for every link. When it detects a collision in the proximal link, it also stops the closure in the distal links. It has one actuator for the abduction of the thumb. Shadow Dexterous Hand 5 7 17 It has one actuator for every finger. When it detects a collision in a link, stops the closure of this finger. It has one actuator for the abduction of the thumb and another for the little finger. Abduction for the little is fixed in the experiments on this paper. Model T 8 4 8 It has one actuator for every finger. When it detects a collision in the proximal link, the closure in the distal link continues. orthogonal approach angles could introduce a bias on the grasp set. As a preliminary study on the the influence of the selection of δ and α values on the quality metrics is performed. A subset of 21 different objects and two hands (Barrett and Shadow) is selected. Two δ values are tested: a small value (0.01 m) and a high value (0.02 m) (Fig. 4a). Additionally, three ways for generating the direction of the approach rays are tested ( Fig. 4b): orthogonal to the object surface (α = 0), at a fixed angle (α = π/3), and at a variable angle (α ∈ [0, π/3]). To summarize, four methods are tested with a combination of the selected δ and α values, as presented in Table 4.
The other parameters needed to define the grasp hypotheses are selected as follows, and are kept unchanged throughout the four methods: Using these parameters with procedure described in Section 3.4 a set of grasp candidates is produced . The number of grasp hypotheses generated per object ranges from a few hundreds with the first two methods  to hundred of thousands with the other methods. Thus, in order to have a tractable number of candidate grasps for these former methods, a 1% of the total amount of candidates are randomly selected in this preliminary study.
Next, the grasp quality values obtained using each metric for all the candidates are statistically analysed; median and interquartile ranges of the different distributions obtained from the use of the four different methods are compared in order to select the most appropriate to generate the approach rays.
Once all the required parameters are selected from the results of the preliminary study, a set of grasp hypotheses is generated for each pair hand-object, using the selected parameters. Each grasp is simulated and, using the contact information, the grasp quality is evaluated according to each of the 10 selected metrics. Different options are considered to select appropriate thresholds to normalise each quality metric: 1) Full Range, using the maximum and minimum values obtained for each metric; 2) Soft Statistics, using the mean ± the standard deviation to define the thresholds (to minimise the effect of atypical values that may appear in case of values grouped within smaller limits [22]); 3) Percentiles, using the 10 th and 90 th percentiles of the grasp quality values (to avoid where Q is the grasp quality value and T max and T min the selected lower and upper thresholds for each metric. Finally, the normalised quality values are graded using a four level scale: bad, fair, good, and great quality. The ranges for each grade are presented in Table 5. The grasp falling under 0 are qualified as bad, those between 0 and the 50-percentile value are labelled as fair, those up to 1 are good, and finally those over 1 are considered great. In any case, this classification has no practical implication and is given as a suggestion to sort out grasps.

Results
The median and interquartile ranges of the quality metrics obtained in the preliminary study are summarised in the box plots presented in Fig. 5. No significant differences between the grasp qualities obtained using the different methods can be observed from the box plots. Furthermore, the results do not consistently show better grasp qualities for any of the methods. The variable method to generate approach rays provides a wider range of possibilities for grasping, and it is, thus, selected for analysing the full set of grasps. A summary of the selected parameters used to generate the grasp hypotheses is presented in Table 6.
With this method more than 25 million grasp hypotheses are generated. In order to make it computable, we decided to select only 1.000 random grasps per object and hand, being a total of 882.000 different grasps evaluated within the 7 hand models.
The values for the ten metrics described in Section 2, for the grasp hypotheses generated using the selected parameters for each of the hand-object combinations (126 objects and 7 hands), are summarised in Table 7. The statistical description for each metric includes all information required to obtain all different thresholds proposed. Results corroborate that the Full Range thresholds are much higher than the Soft Statistics and Percentiles thresholds, because of the high amount of atypical values, especially for metric Q A2 . Also, the high standard deviations of metrics Q A2 , Q A3 , Q C2 , and Q D2 result in negative minimum thresholds for the Soft Statistics thresholds. Figure 6 illustrates an example of the distribution of values of the Q B1 metric using both the Soft Statistics and Percentiles thresholds to normalise the values. Although both methods provide a distribution of grasps along the [0,1] range, the Soft Statistics method produce a high amount of outliers grasps, whilst using percentiles provide more grasps in the mid-range. The Percentiles thresholds were, therefore, finally selected to normalise the metrics. Figure 7 shows the distribution of each of the grasp quality metrics using the Percentiles thresholds, obtained from Table 7, to normalise the metrics. It can be seen that most of the metrics are distributed along the [0,1] range, with the notorious exception of metric Q D2 , which presents 50% of the grasp quality values within the range [0, 0.1]. Also note Q C1 and Q D2 have a lot of values under 0, due to their Percentile 10 coincide with Percentile 30.

Correlation Analysis
The purpose of this study is to find out which metrics provide similar evaluations for the wide set of grasps considered in this work. This would indicate that such metrics are basically representing the same underlying property and, consequently, that one of them is sufficient to evaluate that property, while the others are  Table 4. The grasp quality scale varies to better show the results     redundant. As a final consequence, this would allow the reduction of the number of metrics to be computed without losing any relevant information.

Methods
For this study, the same set of grasp generated in Section 4 is used, since it is sufficiently exhaustive and representative of grasping. The grasps are compared without any normalisation method. This is done to avoid distorting the numeric values, as the correlation coefficients could be influenced by this variations. The Pearson correlation coefficient is calculated for each combination of metrics. However, this method is highly sensitive to outliers and, as it was shown in Section 4, this is quite common in data from quality metrics. Thus, the Spearman correlation coefficient is also calculated, as it has been shown to be more robust in front of outliers [14]. Spearman correlation is also interesting since it can measure non-linear mappings between metrics.

Sensitivity Analysis
A sensitivity analysis is conducted to assess the robustness of each quality metric with respect to the uncertainty in the object's pose. The effect of inaccuracies in the positioning of the real robot hand on the quality of the grasp is investigated. This problem may arise when the final execution of a selected grasp does not succeed in placing the contacts in the desired grasp locations.

Methods
An initial set of grasp configurations are selected as reference. Several modifications of the hand position are computed for each reference grasp configuration by introducing a random translation and rotation on each of the hand position axes (see Fig. 8).
The measurement variation of the metric values for a given object is computed using the Sensitivity Index (SI), which is defined as the mean value of the standard deviations with respect to the metric calculated for the reference grasp: where n is the number of grasps for each object and σ x the standard deviation calculated as: where n v is the number of variations of the reference grasp, x 0 the value of the metric of the reference grasp and x i the value of the metric calculated for each variation. Finally, a Global Sensitivity Index (GSI) has been calculated for each metric, as the mean value of the SI per object previously calculated: where n o is the number of objects and SI i is the Sensitivity Index calculated for each object. GSI is expressed in percentage (GSI N ) by using the metric ranges obtained in the variability analysis (see Table 7): where min y max are any of the proposed minimum and maximum thresholds, respectively.
A preliminary study is carried out to determine the number of variations required for the analysis. Two hands (Barrett and Schunk SDH and 10 objects are chosen for this study. For every pair hand-object, 10 reference grasps are selected randomly from those generated in the experiments described in Section 4. Grasp variations are generated by introducing random translations and rotations on each of the hand position axes obtained from Gaussian distributions with standard deviation of 1 cm and 0.1 rad for translations and rotations, respectively (these Gaussian distributions provide 68% of the deviations inside the bell curve). Set sizes from 10 to 300 variations are investigated (with increments of 10 variations until set size of 50 variations, and with increments of 50 variations afterwards). Grasp variations not meeting the forceclosure condition were discarded, as well as variations resulting in an initial collision between the hand and the object in order to avoid unstable grasps. The metrics' values are computed for all variations, and the GSI N is calculated for each metric, using the 10%-90% percentile thresholds determined in Section 4. The number of variations from which the GSI N gets stabilised is used as the set size N var to be considered in the full analysis.
The full analysis considers 126 objects and 7 hand models. A set of 20 reference grasps is randomly selected for every pair hand-object from those generated in the experiments described in Section 4. For each reference grasp, N var grasp variations are generated analogously as in the preliminary study. GSI N values are computed for each metric using the three different methods for setting range thresholds: full range of variation of the metric, percentiles and soft statistics. Quality values are normalised using the thresholds defined by the Percentiles method. For each metric, a distinction between good and bad reference grasps is considered as those for which the metric value is over or below the 50 th percentile threshold (thresholds taken from the Table 5). Finally, an analysis is conducted regarding the different distribution of variations that improve or worsen the value of the metric from their respective reference grasp. Figure 9 shows the GSI N values calculated in the preliminary study for each metric versus the different set sizes considered. GSI N gets stable for sets equal or larger than 50 variations. Thus, the number of variations N var to be used for the full analysis was set to 50. This preliminary study also revealed that only 20-23% of the variations randomly generated meet the necessary force-closure condition (see Table 9).

Results
The full analysis considers, then, a subset of 50 variations for every reference grasp, which makes a total amount of 1020 grasps (references plus variations) per object, 882000 total variations, and 128520 different grasps for each hand. The force-closure analysis performed to discard unstable grasps provides, in average, only 20.06% of feasible grasps (176,900 from the initial set of 882,000 variations).
Q B3 showed to be the most sensitive metric with a GSI N of 35.91%, and Q B1 the less sensitive with a GSI N of 12.76%. Figure 10 shows, for each metric, the GSI N values computed for the reference grasps, distinguishing between good or bad reference grasps. Higher GSI N values are observed for bad reference grasps than for good ones, especially for Q A2 , Q D2 , and Q C2 metrics; very low differences between good and bad reference grasps are found for Q B1 and Q D1 . Also, Fig. 10 shows the GSI N values computed for their variations, distinguishing between those variations that improve or worsen the metric values from the reference grasp. The highest differences for variations from good reference grasps are observed for Q A1 , Q B2 , and Q B3 (15%, approximately), while variations from bad reference grasps are found for and Q A3 , Q B3 , and Q C1 (20-25%, approximately). These differences match the differences between the percentages of grasps in each category shown in Fig. 11 (referred only to those grasps meeting the force-closure condition), with the exception of those observed for variations from bad reference grasps for Q A3 . The values corresponding to these graphs are detailed in Appendix B. Finally, Fig. 12 shows the relations between the quality variation versus the distance from a grasp to its reference grasp. Deviations are clearly different for some metrics depending on the hand considered; lowest deviations correspond to the Shadow Hand for all metrics, while the Michelangelo prosthetic hand produce high deviations for the metrics Q A1 , Q A3 , Q D1 , and Q D1 , as well as PR2 Gripper for metrics Q D1 and Q D2 .

Discussion
Sections 4 to 6 describe the three analyses that have been carried out on an extensive set of simulated data. The amount of results presented in the form of figures and tables is large and a careful review of them is necessary in order to reach useful conclusions. The most relevant of them are discussed in this section. First, Section 4 describes the variability analysis, and the first question addressed is how to generate  Fig. 5 indicate that the four methods produce indistinguishable sets of grasps, as long as these sets are sufficiently large. The results of the complete variability study are summarised in Fig. 7 and Table 7. These results are relevant, as they can be used to assess the room for improvement of a specific grasp, regarding a given quality metric. An important phenomenon observed on these results, also, is the characteristic distribution of most of the values of the metrics which tend to concentrate in narrow ranges with an important number of atypical values above and below these ranges. This phenomenon complicates the selection of upper and lower thresholds for normalisation if the aim is to obtain an almost uniform distribution of values within Fig. 11 Distribution of grasps in each category for each quality metric. In each graph there are three groups of columns. The left one shows the percentage of good reference grasp (value over the 50% value); the middle group for the bad reference grasps; and the third group for all the reference grasps. In each group the blue column is the percentage of the variations in the category; the orange, variations which improve the reference values; and the yellow column variations which worsen the reference values Fig. 12 Variation in the quality metric of each grasp with respect to the distance to its reference grasp. Each point represents a grasp and grasp on different hands are coloured differently. Horizontal lines are drawn to indicate the limits containing 10%, 50% and 90% of the grasps the proposed limits. Several approaches to set those range limits have been tested, and finally the use of 10 th and 90 th percentiles seem to be a good compromised option for all metrics. Some of them, however, still show a non uniform distribution, especially Q D2 and Q C1 , and more refined approaches could be used for these specific metrics. In practical terms a normalisation based on the 10/90 percentiles and the thresholds derived form them seem to be most advisable since it provides the most uniform distributions, but with the limitations discussed above.
The second study is the correlation analysis presented in Section 5, aimed to determine relations between metrics. The results from this analysis are again deeply affected by the presence of atypical values in the metrics. This problem has been addressed by using the Spearmam correlation coefficient. Table 8 shows the results using both the Pearson and Spearman correlation rules. The latter table shows higher correlations than the former. These results indicate that all the metrics based on the algebraic properties of G matrix (Q A1 , Q A2 , and Q A3 ) represent the same underlying principle. Metrics of Group C, shows a weak correlation with this group. Finally, Q C2 also shows a weak correlation with metric Q C1 , from its same group. In short, the ten metrics could be reduced to seven groups composed of five independent metrics Q B1 , Q B2 , Q B3 , Q D1 , and Q D2 , a pair of metrics, Q C1 and Q C2 , and a triplet of metrics, Q A1 , Q A2 , and Q A3 . Because of the weak correlations between the metrics from groups C and A (both groups based on algebraic properties of G), all these metrics could be even reduced to the same group without losing a significant amount of assessment information.
The third and final analysis is the sensitivity study detailed in Section 6, which is aimed to determine the behaviour of the metrics in front of uncertainty in hand positioning. According to Table 10, all metrics present a GSI N over 10%. Globally, the most robust metrics are Q B1 and Q D1 , while Q B3 , Q C1 , and Q A2 are the more fragile ones. More detailed information can be obtained from Fig. 10, that shows the GSI N distinguishing between good and bad reference grasps. The obvious conclusion from this figure is that reference grasps whose values are over 50 th percentile are more fragile than those under 50 th percentile for most of the quality metrics, and especially for the cases of Q A2 , Q C2 , and Q D2 . Metrics Q D1 and Q B1 , which are the most robust ones globally, have a similar robustness independently of the goodness of their quality value. Also, for good reference grasps, the metrics are more fragile for those grasps variations that worsen the quality metric, except for metrics Q D1 and Q D2 ; while for bad reference grasps, the metrics are more fragile for those grasp variations that improve the quality metric.
Once analysed the robustness of the different metrics, we can postulate that Q A1 could be a good choice for representing the triplet of metrics: Q A1 , Q A2 , and Q A3 , as it is the one among them that has the lowest GSI N .
It is also interesting to note on Fig. 12 that quality deviations are significantly different depending on the robotic hand considered. The Shadow Hand presents the lowest deviations for all metrics, while the PR2 Gripper and the Michelangelo prosthesis present the highest deviations for several metrics, which could be related with the low number of actuators of both hands, that greatly reduces their versatility. Grasp quality is, thus, highly dependent on the robotic hand design. Therefore, the design of robotic hands and prostheses could benefit from the grasp simulation and computation of grasp quality metrics if improvement of these magnitudes were used to guide design modifications.
Finally, an important result of this third analysis that should not be neglected is the high rate of grasp variations that do not meet the force-closure condition. Almost four fifths of them are discarded for this reason. We have decided to do so because some of the metrics require the force-closure condition to be met so the metric values make sense. It should be considered if some more relaxed condition could be chosen and how this could affect the results, and more importantly whether results sensitivity are relevant with such amount of variations discarded. To conclude, some comments are necessary regarding the results obtained in this work in comparison to those in previous works [34,35]. In the first place, this work contributes with more reliable thresholds to normalize and classify the metrics since they are based on grasps obtained from a larger number of objects and hand models. On the second place, while the correlation between metrics in Group A are mantained, new ones appear in Group C, though they are weaker, and those between Groups B and D are discarded. Results on sensitivity slightly differ from those obtained in previous studies, being Q B3 more sensitive, and Q D1 more robust.

Conclusion
This paper presents a numerical study on the practical parameters that characterise the use of ten quality metrics for evaluating grasps performed by robot hands on rigid objects. In the first place, the thresholds that indicate the practical upper and lower limits of the quality metrics have been established. These thresholds allow a more realistic normalisation of the metrics values. However, the influence of object's size on these thresholds should be addressed in future work. Second, a correlation analysis has revealed the underlying relations between some metrics and the independence of others, which leads to conclude the existence of at least seven relevant grasp aspects to be taken into account when evaluating a grasp. Finally, the sensitivity analysis has shown the fragility or robustness of the metrics in front of positioning uncertainty.
The results presented in this paper may provide several benefits. First, they provide parameters regarding quality metrics which allow a better use of them in grasp planners, by the setting of data-based thresholds for evaluating and ranking of grasps. Second, the resulting data are a guidance in the selection of metrics in terms of robustness and equivalence between different alternatives. And finally, the results also give good hints in order to build combined metrics, which can capture more relevant aspects of a grasp and, thus, provide a better prediction of the grasp performance.
However, this study has also some limitations that should not be disregarded. First, this work only considers ten quality metrics, while at least 24 have been surveyed [31]. Though some of them were not compatible with the present study, i.e., biologically oriented, combined, and task oriented metrics, others could have been included, but were discarded. Second, this paper is focused on static grasp for holding objects, consequently, no task-oriented grasp metrics or analyses have been considered. An obvious complementary extension of this work would be to include task oriented analyses. Third, only static contacts and states were simulated. Dynamic behaviour of objects during grasping would provide a much more realistic study of the metrics as quality predictors. This should be possible using physics engines in order to simulate dynamic effects of grasps, though the unreliable physical fidelity that kind of engines are able to achieve poses some shadows on this approach.
Fourth, the selection of hand models may introduce a bias, as the metrics could be giving better or worst values depending on the hand used, as was demonstrated by the authors in previous works [34,35]. In any case, we believe that the wide variety of hands and models used is enough the reduce if not eliminate this possible bias. And fifth, the election of OpenRave as simulation tool can also affect the results as information of contact points may differ from those that could be obtained using other simulators. Replicating the study, using other simulators and grasp generation procedures could clarify the concerns.
Finally, the friction coefficient through this analysis has a constant value of 0.4, which adds another limitation. The friction cones produced at the resulting contact points are affected when changing the value of the friction coefficient, which could lead to different evaluations of the quality measures for the same grasp pose. Therefore, the friction coefficient has a major role as a filter to discard more or less candidates. The sensitivity of the metrics to friction coefficient changes has been previously studied to some extent [15,47] and it is not discussed in this work as it is a topic for a whole new paper.
And last but not least, the most obvious limitation is the problem of transferring results from simulation to real robots. There are some studies that compare and analyse the performance of metrics for real applications. In fact, recent experiments have questioned the utility of analytical metrics. In [2,11] the authors show that grasps highly ranked by the -metric (Q C2 in this paper, see Table 1) perform poorly on real robots. It has also been demonstrated that this metric has a high sensitivity to positioning errors [43]. There can be several reasons for this fragility of the grasp quality metrics. On the one hand, the metric used in the above works, Q C2 , has shown to be very sensitive to uncertainty, a key aspect on the execution on real robots. On the second hand, the correlation analysis has shown that there are at least seven independent dimensions for evaluating a grasp. Grasp planners which rely on a single metric and ignore the other dimensions are probably providing incomplete grasp quality predictions.
The results on this paper are exclusively based on static simulation. They might be taken with cautions until a proper validation and correlation with results obtained from a real-world scenario. However, the preparation and realization of such experimentation is complex and out of the scope of this work. We see the results presented here as the first and necessary phase of such a work which we are currently addressing.
We foresee applications which could take profitable use of the results and methodology proposed in this paper. The results on this paper constitute a huge background database to be used as reference for evaluating hand designs. The methodology would consist on testing a hand on a sufficiently large database of objects, and employing a large number of grasps in each case. The results could be statistically compared against the ones shown in this paper, and evidence-based conclusions could be reached regarding the capabilities of the novel design. The methodology and results described in this paper could be easily adapted for this application. A promising preliminary work [21] has been already published following this approach.
where n c is the number of contact points and c i is the location of each contact point. Then the metric is calculated as: This metric has to be minimised and has units of length. We propose its normalization taking into account that its lower limit is zero and the upper limit can be calculated as the maximum distance from the center of mass of the object to any point in the object's contour (distance max ), which can be obtained as the maximum distance from the center to any of the corners of the object bounding box. Additionally, the metric has been adapted to have 1 as its best value:

A.0.5 Q B2 -Area of the Grasp Polygon
This metric is defined as the area of the polygon formed by the contact points [27]. A larger area is argued to give a more robust grasp, i.e. with the same finger force, the grasp can resist larger external torques [45]. It has been used in robotics for three finger hands, but for more than three finger hands, the indicator is extended generating the contact plane by selecting three fingers and the remaining contacts are perpendicularly projected onto that plane. The index is calculated as: Q B2 = Area(Polygon(p1, p2, p3, p4 P , p5 P , . . .)) (8) where p1, p2, p3 are the contact points for the three selected fingers, and p4 P , p5 P , . . . are the projected points of the other fingers onto the plane. This metric has to be maximised and has units of [area]. We propose to normalise this metric taking into account that its lower limit is zero and the upper limit (Area max ) can be calculated as the area of the polygon when the hand is open in a plane with the joints at their maximum abduction limits. Then, the normalised metric can be calculated as: (Polygon(p1, p2, p3, . . .))/Area max (9)

A.0.6 Q B3 -Shape of the Grasp Polygon
This metric is defined for planar grasp polygons and compares how far the internal angles of the grasp polygon are from those of the corresponding regular polygon [17]. For the five fingers of the human hand, a planar grasp polygon is obtained in the same way as for Q B2 . This index is calculated as: where n f denotes the number of fingers, θ i is the inner angle at the i th vertex of the polygon,θ denotes the average angle of all inner angles of the grasp polygon and θ max is the sum of the differences between the internal angles when the polygon has the most illconditioned shape (degenerates into a line) and those of the regular polygon.
This metric has to be minimised and has no units. The lower limit is zero and upper limit is 1. We propose to adapt this metric to have 1 as its best value: Group C: Magnitude of Forces

A.0.7 Q C1 -Smallest Maximum Wrench to be Resisted
The grasp quality is defined as the largest perturbation wrench that the grasp can resist with independence of its direction [13,19]. Only the directions of forces are used and their magnitudes are upper-bounded to 1.
Defining GW S as the set of all possible wrenches w acting on the object, the maximum of w ∈ GW S lies on the boundary approximated as the convex hull over the discretised frictions cones (CW ). Then the quality metric is the radius of the largest sphere centred at the origin, which is contained in GWS: This metric has to be maximised and it has [f orce] units if the torque in w is divided by a parameter ρ with units of [length]. The index depends on the choice of the origin of the reference system used to compute torques. In this work, we use the center of mass of the object and limited the magnitude of the torques to 1 choosing ρ as distance max defined previously for the metric Q B1 . Then, the upper limit of the index is √ 2 and the lower limit is zero. Then, we can normalise this index as:

A.0.8 Q C2 -Volume of the Convex Hull
This metric is defined to avoid the dependence of the previous index on the selection of the origin of the reference system. The metric calculates the volume of the boundary of the set of all possible wrenches acting on the object [26].
The reference system and ρ have been chosen as described in the previous metric. The index has to be maximised and has units of [f orce] 6 . Lower limit is zero and upper limit is no determined so that it is initially not possible to normalise the index in the range 0 to 1. However, we used the Monte Carlo method to estimate the upper limit for each hand.
The method was used to randomly generate wrenches and determine Q C2 for a very large number of iterations. The variables that could be randomised are the contact normals n i = (n xi , n yi , n zi ) and the contact points c i /ρ = (c xi , c yi , c zi )/ρ. For each of their components, their values can vary in the range between [-1,1]. In order to assure that n i is normalised, it has to satisfy the following equation: which can be interpreted as the equation of a circle with radius 1 − n 2 xi . Therefore n yi and n zi can be calculated as: Giving random values to n xi in the range between [1, -1] and to θ in the range between [0,2π], we obtain a normalised value of n i uniformly distributed. In order to give values to c i , initial experiments were performed given random values between [-1,1] to c xi , c yi , c zi . They were represented as a cube centred at the origin and with dimensions 2 x 2 x 2. As it was expected, greater volume is obtained when the contacts approach the surface of the cube. The parameter ρ is calculated as √ 3 from the center of the cube to one of the corners. Having the contact points fixed at the surface of the cube, n xi and θ are randomised using different number of iterations until two consecutive trials give the same results of Volume max with an allowable error. The normalised metric then can be calculated as: Q C2 = Volume(CW )/Volume max .

A.0.9 Q D1 -Posture of Hand Finger Joints
This index metrics how far each joint i is from its maximum limits [25]: where n q is the number of hand joints, y i the current position of the joint and R i is the joint angle range between the reference position a i and either the upper or lower angle limit, used to normalise the index. It is defined as: where y iM and y im are the maximum and minimum angle limits of the i th joint. In this work, the reference position a i was selected as the initial posture of each hand as this will be the more efficient in terms of energy consumption.
The index has to be minimised, so that the grasp is optimal when all joints are at the reference position, having a quality metric of zero, and it goes to one when all its joints are at their maximum angle limits. The metric has been modified to have 1 as its best value: A.0.10 Q D2 -Inverse of the Condition Number of G J The condition number of a matrix is defined as the ratio of its maximum singular value to its minimum singular value. For the Jacobian, the inverse condition number gives a metric of the sensitivity of the magnitude of the end-effector velocity to the direction of the joint velocity vector [18,37]. It is a dexterity metric that considers the capability of the hand to move an object in any direction with the same gain, which implies a good manipulation ability [31]: where σ min and σ max are the smallest and largest singular values of the grasp Jacobian matrix G J [20]. This metric has to be maximised and has no units. Lower limit is zero and upper limit is one, indicating a uniform transformation and a grasp with the maximum quality. Therefore, this metric is already normalised into the desired range 0 to 1. Table 11 details the GSI N values for every metric computed for the reference grasps, distinguishing between good or bad reference grasps and between those variations that improve or worsen the metric values from the reference grasp. This table is related to Fig. 10. Table 12 shows the distribution of the variations of good and bad reference grasps according to whether the variation improve or worsen the reference quality value. This table is related to Fig. 11.