{'fraction_non_alphanumeric': 0.03658215632950259, 'fraction_numerical': 0.01342223002017721, 'mean_word_length': 4.551623376623376, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 4, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Machine Learning Approach for Predicting Human Preference for Graph Layouts

Understanding what graph layout human prefer and why they prefer is significant and challenging due to the highly complex visual perception and cognition system in human brain. In this paper, we present the first machine learning approach for predicting human preference for graph layouts. In general, the data sets with human preference labels are limited and insufficient for training deep networks. To address this, we train our deep learning model by employing the transfer learning method, e.g., exploiting the quality metrics, such as shape-based metrics, edge crossing and stress, which are shown to be correlated to human preference on graph layouts. Experimental results using the ground truth human preference data sets show that our model can successfully predict human preference for graph layouts. To our best knowledge, this is the first approach for predicting qualitative evaluation of graph layouts using human preference experiment data.


INTRODUCTION
Evaluation of graph layouts is a significant problem in graph drawing. A number of quality metrics (or aesthetic criteria), such as edge crossings, bends, drawing area, total edge lengths, angular resolution, and stress, have been proposed for the quantitative evaluation of graph layouts [4]. Consequently, various graph drawing algorithms to optimize these metrics have been developed [4].
Qualitative evaluation on graph layouts is also available, using HCI methodology, using human preference or specific task performance, measuring time and error. For example, edge crossings are shown to be important aesthetic criteria for performing human preference and shortest path tasks on graph layouts [22,27]. Furthermore, large crossing angles are shown to be effective for shortest path tasks on graph layouts, when edge crossings are present [12].
Understanding what layout human prefer and why they prefer a specific graph layout over the others is significant, since it motivates researchers to design algorithms to compute such layouts, and guides users to choose specific algorithms to produce such layouts. However, it is extremely challenging due to the highly complex human visual perception and cognition system involving massively parallel processing using vision and memory in human brain [26].
A series of human preference task performance experiments have been conducted to better understand which graph layout human prefer. For example, Purchase [22] found the correlation between the human preference and fewer edge crossings in graph layouts. More recently, Chimani et al. [2] found the correlation between the human preference and lower stress in graph layouts, and Eades * e-mail: scai5619@sydney.edu.au † e-mail: seokhee.hong@sydney.edu.au ‡ e-mail: jshe9143@uni.sydney.edu.au § e-mail: tongliang.liu@sydney.edu.au et al. [5] found the correlation between the human preference and higher shape-based metrics in graph layouts.
In this paper, we present the first deep learning approach for predicting human preference for graph layouts. More specifically, we propose a CNN-Siamese-based model that can be trained to predict human preference from a given pair of layouts of the same graph. Roughly speaking, CNN models read images rendered from graph layouts and convert them into feature vectors, which are inspired by the processing procedure of the human brain [8]. The Siamese model computes the difference between a pair of feature vectors.
To understand the procedure, we assume that there exists a certain human preference measurement which measures a pair of layouts to a human preference label (that indicates which layout human prefer). The deep model can be regarded as a measurement from the input layout pair to the output prediction. The model is trained to mimic the ground truth human preference measurement by fitting the training data which contains pairs of layouts and human preference labels.
The amount of the ground truth human preference data sets from previous experiments [2,5] is relatively limited and insufficient for training deep models. To address this, we train the model by employing the transfer learning method [21], which exploits data from related problems to help the original problem (e.g., human preference prediction). Specifically, since some quality metrics (i.e., shape-based metrics, edge crossing and stress metrics) have been shown to be correlated to human preference for graph layouts [2,5], we transfer the hypothesis by employing the layout pairs labeled by quality metrics to help train our deep model.
More specifically, we first pre-train our model using qualitymetrics-based pairs described in Section 3.3.2, which consists of different graph layouts of the same graph and labels constructed by employing the quality metrics. Then, we fine-tune our model by using the human preference experiment data. Extensive experiments show that our machine learning approach successfully predicts human preference for graph layouts.
The main contribution of this paper is summarized as follows: 1. We propose the first machine learning approach to predict human preference for graph layouts. To our best knowledge, this is the first approach for predicting qualitative evaluation of graph layouts by exploiting the ground truth human preference experiment data [2,5]. 2. We introduce a transfer learning method to overcome the insufficiency of the available human preference experiment data for training deep models. More specifically, we use qualitymetrics-based pairs to pre-train our model to better understand human preference. 3. Extensive experiments using the ground truth human preference data [2,5] show that our machine learning approach successfully predicts human preference for graph layouts. Specifically, our model achieves average test accuracy of 92.28% for large graphs, and 63.77% for small graphs, significantly outperforming random guessing (i.e., greater than 50%) for the binary human preference problem.
The main reason of the high test accuracy for large graphs (e.g., arXiv:2103.03665v1 [cs.HC] 1 Mar 2021 mesh and large scale-free graphs) is due to the strong preference scores among the layouts in the human preference data (i.e., some layouts have much better quality than other layouts). On the other hand, for small graphs (e.g., small sparse graphs and biconnected graphs), there was no strong preference among the layouts in the human preference data (i.e., all layouts have similar good quality). Therefore, it is more difficult to predict.

Quantitative Evaluation for Graph Drawing
Various quality metrics for the evaluation of graph drawings, called aesthetic criteria, are available [4]. Traditional readability metrics include edge crossings, bends, area, total edge lengths and angular resolution. Consequently, many graph drawing algorithms have been designed to optimize these quality metrics [4].
Recently, new faithful metrics have been developed, which measure how faithfully graph drawings visually display the ground truth structures of graphs. For example, Eades et al. [5] introduced the shape-based metrics, by comparing the similarity between a graph G with a proximity graph G computed from a drawing of G. The stress [4] is a distance faithful metrics, which compare the difference between graph theoretic distance of vertices and the Euclidean distance in a drawing.
Similarly, the cluster faithful metrics [19] compare the similarity between the ground truth clustering of a graph G and the geometric clustering computed from a drawing of G. The symmetry faithful metrics [20] measure how the ground truth automorphisms of a graph are displayed as symmetries in a drawing.

Qualitative Evaluation for Graph Drawing
Qualitative evaluation on graph layouts have been investigated by conducting the HCI-style human experiments, with preference or task performance, measuring time and error. For example, Purchase [22] found correlation between human preference and fewer edge crossings in graph layouts.
Ware [27] found correlation between fewer edge crossings and shortest path tasks in graph layouts, and Huang et al. [12] found correlation between large crossing angles and the shortest path task.
Recent studies [18] found that human untangling interaction task of hairball-like graph layouts is positively correlated with the shapebased metrics [5], while surprisingly negatively correlated with the edge crossings and stress.

Human Preference Experiments in [2, 5]
More recently, a series of human preference experiments have been conducted [2,5]. Specifically, in the human preference experiments, the system showed two layouts of the same graph, randomly chosen from five different graph layouts, including force-directed layouts (such as FR [6]), stress minimization layouts and multi-level layouts (such as FM3 [9]). The data set used in the experiment includes Hachul library, Walshaw's Graph Partitioning Archive, and randomly generated biconnected and scale-free graphs. The task for participants was to choose their preferred layout from a pair of layouts of the same graph, and select their preference score using a slider bar scaled from 0 to 5.
The first experiment conducted at the University of Osnabrück [2] found the correlation between human preference for graph layouts and edge crossings and stress. Namely, human prefer graph layouts with less stress and fewer crossings.
The two follow-up experiments [5] conducted at the Graph Drawing conference 2014 and the University of Sydney, showed that the shape-based metrics are positively correlated with human preference, i.e., human prefer graph layouts with high shape-based metrics.

Deep Learning
Recently, deep learning has achieved great success in various fields, such as computer vision, natural language processing, and speech recognition. The Convolutional Neural Network (CNN) is a representative deep network for image recognition and classification. CNNs are a type of multi-layer neural networks, designed and trained to recognize the nature of images by varying the depth and breadth of a model [16].
Siamese neural networks were introduced by Bromley and Le-Cun to solve signature verification as an image discrimination problem [1]. Koch et al. [15] used Siamese neural networks to rank the similarity between multiple inputs and discriminated input features. A Siamese neural network joins together the highest-level feature representations of twin inputs for image classification problems.
Transfer learning [21] aims to improve the learning performance of a target task (or problem) by borrowing knowledge from related but different tasks. Its core idea is to learn task-invariant data representations [7]. In computer vision, complex deep networks, e.g., AlexNet [16] and VGG [24], are often trained by employing the transfer learning technique to leverage the large-scale dataset Im-ageNet [3]. Specifically, the networks are usually pre-trained on ImageNet first and then are fine-tuned on the datasets of the target tasks.

Deep Learning Approaches in Graph Visualization
A number of researchers used deep learning methods for problems in graph visualization, mainly focusing on quality metrics [10,14,17,25]. For example, Haleem et al. [10] used CNNs to predict multiple readability metrics, e.g., node spread, group overlap and edge crossings, using graph layout images with up to 600 vertices.
Klammler et al. [14] used the Siamese neural network for aesthetic discrimination of paired layouts, where the labeled training pairs consisted of a graph layout D and its deformed layout D using deformation steps (i.e., comparing the original layout with its deformed layout using quality metrics). Therefore, their work predicted a better quality layout based on quantitative evaluation (i.e., quality metrics).
Note that in our work, we use the ground truth human preference experiment data [2,5] for training and testing, where a pair of layouts D 1 and D 2 are computed using two different graph layout algorithms (i.e., comparing two different graph layouts for human preference). Therefore, We predict human preference based on qualitative evaluation (i.e., human preference experiment data).

Model
We present a CNN-Siamese-based model that can predict which layout human prefer from a given pair of layouts. The notable advantage of CNNs is that they are powerful in extracting features from image inputs. The use of the Siamese model is natural since it deals with a pair of layouts to measure their difference. Fig. 1 shows the pipeline of our model, including four essential parts: (a) Input data, i.e., a pair of layouts (note that in the training phase, the input pair also needs a label indicating which layout is better; while in the test phase, the label is not required); (b) Twin CNN-based image feature extractors; (c) Subtraction part of the Siamese model; (d) Output prediction (note that in the training phase, we need to measure and minimize the difference between the predicted value and the ground truth; while in the test phase, the output indicates the predicted human preference label). We now explain each part of the model in detail. (a) Input data To obtain a machine learning algorithm for predicting human preference, we need training and validation data sets to train the parameters. Furthermore, we need to select a model (controlled by hyper-parameters that cannot learn from the training data) with good performance. Specifically, training and validation data sets consist of labeled pairs of color images. Such images are the visual rendering of layouts with size 320 × 320. We label each pair with 0 or 1 during training, where 0 means human prefer the first layout A compared with the second layout B (see Fig. 1(1)); otherwise, human prefer the second layout B. More details about the training and validation data sets are presented in Section 4. Note that the data with human preference labels is limited. We employ transfer learning and the data pairs (image pairs) labeled by quality metrics to help train our model. More information about transfer learning and the data is presented in Section 3.2.

(b) Twin CNN-based image feature extractor
Our twin CNN-based image feature extractor is built on VGG. They convert the input images into semantic feature vectors. The following part of the Siamese model outputs a prediction for human preference based on the semantic feature vectors. We now explain the details of our CNN-based feature extractor.
Convolutional layers are efficient to extract semantic features of an image inspired by the processing procedure of human brain. Multiple hidden layers are essential to increase the expressive ability of the deep network. Max pooling layers retain the most significant features. The Fully connected layer summarizes features for feature subtraction in the Siamese model (see Fig. 1(c)). Fig. 1(b) shows the design of our CNN-based feature extractor.

(c) Subtraction part of the Siamese model
A Siamese neural network consists of twin feature extractors and a joint layer. To compute the difference of twin inputs, we use a subtraction part to join the twin features, which shows a higher test accuracy than the concatenation part. The subtraction part converts the pair of semantic feature vectors output by the twin feature extractors into a single value in the range [0, 1] to predict human preference. Specifically, the two feature vectors are joined into a single long vector, which is reduced to one dimensional by employing a fully connected layer. A Sigmoid activation function further restricts the output value in the range [0, 1]. Note that if the output value ≤ 0.5, the prediction is that human prefer the first layout A compared with the second layout B; vice versa.

(d) Output prediction
We aim to train the proposed deep model to have outputs that are aligned with human preference labels (see Fig. 1(d)). Specifically, to achieve this, we optimize the parameters of the deep model by minimizing the difference between the predicted values and the ground truth human preference labels. The difference can be measured by a loss function. In this paper, we employ the binary cross-entropy loss function.
Since the training sample with human preference could be limited, we employ the transfer learning technique to reduce the hypothesis complexity of the deep model.

Transfer Learning for Predicting Human Preference for Graph Layouts
To train a deep machine to understand human preference, we need a large amount of human-labeled pairs of layouts. However, annotating a large number of layout pairs is usually time-consuming and expensive. Fortunately, we can address the issue by employing the transfer learning technique [21]. Transfer learning transfers knowledge across different tasks to improve learning performance. Typically, if the target task has limited training data, by using transfer learning, we could use the related tasks (called source tasks) that have sufficient training data. It has been shown that some quality metrics (i.e., shape-based metrics, edge crossing and stress metrics) are correlated to human preference [2,5]. The layout pairs with different quality metrics values are relatively easy to compute, which means we could easily obtain example pairs labeled by quality metrics. We thus exploit such data to help training our deep model. More details on how to generate those data are described in Section 3.3.
Because of the highly complex visual perception and cognition system in the human brain, the mechanism that human use to decide their preference for graph layouts could be very complicated. Although some quality metrics are shown to be correlated to human preference, the precise relationship between them remains unknown. To address the above issue, we transfer the hypothesis. Specifically, using the related data (e.g., quality-metrics-based pairs) to pre-train the deep model and then use the target task data (e.g., human preference pairs) to fine-tune the deep model.
To demonstrate the effectiveness of the transfer learning, we compare the following three trained models in our experiments as an ablation study: 1. M: a model trained only on data pairs labeled by using quality metrics. 2. HP: a model trained only on data pairs with human preference labels. 3. M+HP: a model pre-trained on data pairs labeled by using quality metrics and then fine-tuned on data pairs with human preference labels (i.e., a model trained by using the first and second stages).
Note that M and HP are two baseline models to compare with our transfer learning model, i.e., M+HP. In the experiment section, results show that the performance of M+HP is much better than these of HP and M.

Human preference pairs
Human preference pairs are processed from data in the human preference experiments [2,5]. In the previous experiments, given a pair of layouts of the same graph, the task for multiple participants are to choose their preferred layout. The task performance is measured by a discrete preference score ranging from 0 to 5, where 0 means no preference for the pair of layouts, and 5 means the strongest preference score. Note that since human preference may be subjective, different participants may have different preferences for the same pair of layouts. To solve this conflict, we use the weighted majority voting methods to reach a consensus. Specifically, let G i denote the graph where i is graph index. Let D i j and D i k denote two layouts of the graph G i where j and k are the layouts indices. Let P denote the preferred layout. Let S be the preference score of the preferred layout. For the pair D i j and D i k , where j < k, we convert the measurement of the task performance into the human preference label (e.g., 0 or 1), reflecting the consensus of multiple preferences of the same pair of layouts. The human preference label is generated by the following weighted majority voting algorithm: 1. Counting how many participants chose their preference for the pair D i j and D i k . Let N denotes the number of participants. 2. Assigning the weights for the participants to vote. For the n-th participant, if the preferred layout P is D i j , set its voting weight w n to be the preference score of the preferred layout, i.e., S; if P is D i k , set its voting weight w n to be −S.

Labeling the pair according to the weighted majority voting.
If the average preference score ∑ n w n /n > 0, set the label L hp to be 0, indicating that the layout D i j is more preferred by human than the layout D i k ; if ∑ n w n /n < 0, set the label L hp to be 1, indicating that the layout D i j is less preferred than the layout D i k ; for the pair D i j and D i k , if ∑ n w n /n = 0, discard the pair of layouts without labeling.

Quality-metrics-based pairs
To train and evaluate our model, we randomly split graphs and having their corresponding labeled layout pairs into two data sets, i.e., a training data set and a test data set. However, the size of layout pairs with human preference labels can be small, making it hard to train deep models well. We handle this problem by employing the transfer learning technique to pre-train the deep models, which exploits the layout pairs labeled by using quality metrics. For a pair of layouts, it is labeled by exploiting shape-based metrics M sh , edge crossing M c , and stress metrics M st . The works of literature [2,5] show that values of shape-based metrics are positively correlated to human preference; while values of edge crossing and stress metrics are negatively correlated to human preference. Note that quality metrics for layouts can be computed by specific algorithms [5,23]. As there are different quality metrics, we form qualitymetrics-based labels by employing the majority voting method.
In the human preference experiments [5], each graph has five layouts. We compute the quality-metrics-based labels for all possible ten pairs of each graph. Specifically, let D i j and D i k be two layouts of the graph G i , the quality-metrics-based label is generated as follows:

Data Set
In our experiment, 146 graphs and their five layouts from the human preference experiments [5] were used. The graphs range in size from small (25 vertices and 29 edges) to large (8,000 vertices and 118,404 edges), and have different structures such as sparse graph (i.e., graphs consists of cycles and tree), biconnected graph, mesh, and large scale-free graph. We categorized these graphs based on the number of vertex, i.e., large graphs (e.g., mesh and large scalefree graphs in Fig. 2), and small graphs (e.g., sparse graphs and biconnected graphs in Fig. 3). Table 1 shows the statistics of the graph data used in our experiment.  After we pre-process human preference data, as described in Section 3.3.1, we obtain 511 layout pairs with human preference labels. We then compute the ten quality-metrics-based pairs of each graph using the approach described in Section 3.3.2. Therefore, in total, we have 1,460 quality-metrics-based pairs for pre-training, and 511 human preference pairs for fine-tuning and testing.

Model Training
We randomly split the data with human preference labels into training and test sets with a ratio of 7 : 3. Note that the quality-metricsbased pairs are only used for training, i.e., pre-training. Specifically, to split the data, we randomly split the graphs and use their corresponding labeled layout pairs as the training and test sets. In other words, layout pairs belonging to graphs of the training set will not appear in the test set.
To choose the model, We use the five-fold cross-validation method. Specifically, the training data is randomly split into five folds. The model is chosen by alternatively using four folds of data for training and the remaining one fold for testing.

Prediction Results
To examine the variations of human preference on different kinds of graphs, we do the ablation study by dividing our data sets into four subsets according to graph types: mesh (see G 6 and G 10 ) and large scale-free graph (see G 3 and G 15 ) in Fig. 2, sparse graph (see G 0 and G 188 ) and biconnected graph (see G 18 and G 65 ) in Fig. 3. Table 2 summarizes our experiment results with three trained models (i.e., M, HP and M+HP) on different graph types. The number in each cell represents the average test accuracy with the standard deviation after five times random splitting.  To validate whether one trained model outperforms another, we employ the Wilcoxon signed-rank test to compare the proposed models, a non-parametric statistical hypothesis test method.
We run the significance test on the test accuracy of models trained on different types of graphs (i.e., by scipy.stats.wilcoxon function). Table 3 shows the p-values for comparing training variations. The p-value depends on the median accuracy of the first model that is negative against the median accuracy of the second model that is positive; the smaller the p-value, the better the second model. The p-value ≤ 0.05 means the difference is statistically significant.

SUMMARY, DISCUSSION AND FUTURE WORK
Summary: In summary, we show human preference for graph layouts can be predicted by machine learning algorithms. Specifically, the proposed model predicts human preference for a pair of graph layouts with an average test accuracy of 92.28% for large graphs, and 63.77% for small graphs, which are significantly better than random guessing for binary human preference problem. Our main hypothesis that human preference for graph layouts can be learned by machine has been successfully proven. For the three trained models, i.e., M, HP and M+HP, the performance gradually increases significantly, as shown in Table 2, which shows the importance of employing the transfer learning technique during training. For details, see Appendix B.1.
Large graphs: Large graphs have high test accuracy due to the relatively clear preference between layouts (i.e., some layouts have much better quality than others). For mesh, their layouts have distinct shapes, therefore it is easier for human to decide their preference with a high preference score (see [D 5 4 , D 5 3 ] in Fig. 4). Large scale-free graphs are difficult to visualize, i.e., some graph layouts produce much better quality than other layouts with poor quality, therefore human can choose their preference with a high preference score (see [D 3 0 , D 3 3 ] in Fig. 4). For details, see Appendix B.2.
Small graphs: Our three trained models succeed to predict the ground truth human preference value for the small graphs data set (see Fig. 6). However, for small graphs, the human preference was not strong between layouts (i.e., layouts have similar quality), see Fig. 7 and Table 5. For details, see Appendix B.3.
Limitation and Future work: Although our trained models perform quite well on both large graphs and small graphs, we found some limitations. For example, when a pair of layouts has similar visual quality, the preference score is very low, i.e., 0 or 1 (see [D 113 1 , D 113 2 ] in Fig. 8 and Table 6). Moreover, preference scores of the same pair of layouts can vary (from 1 to 4) due to the subjective human preference (see [D 13 4 , D 13 0 ] in Fig. 8 and Table 6). Interestingly, there was an exceptional case where only model M succeeds to predict human preference (see Fig. 9 and Table 7).
To mitigate such limitations, we plan to conduct new human preference experiments with more data sets and visually different layouts, to design a machine learning model for better predicting the human preference on graph layouts. Fig. 3 show the examples of four types of graphs and their five layouts used in the human preference experiments. Fig. 2 shows the examples of large graphs G 6 , G 10 , G 3 and G 15 . Fig. 3 shows the examples of small graphs G 0 , G 188 , G 18 and G 65 .

Fig. 2 and
In the previous experiments [5], the task for participants was to randomly choose from a pair of different layouts of the same graph, and then rate their preferred layout from 0 to 5. In our experiment, we convert the task performance from the preference score into human preference labels (e.g., 0 or 1) by employing the weighted majority voting algorithm.
Participants in the previous two experiments [5] include computer science and graph drawing experts at the Graph Drawing conference in 2014, and students of the course information visualization at the University of Sydney.

B.1 Comparison of trained models
The model with fine-tuning by human preference pairs (i.e., M+HP) performs significantly better than the model without fine-tuning (i.e., M), and the model only exploiting the data with human preference labels (i.e., HP). The comparison results of M+HP vs M and M+HP vs HP show that the transfer learning technique is very useful and that the qualitymetrics-based pairs contain knowledge that can help to predict human preference, consistent with the known correlation results [2,5].
The success of transfer learning indicates that both the qualitymetrics-based pairs and the human preference pairs are important in studying and predicting human preference for graph layouts.
The model trained by human preference pairs (i.e., HP) performs better than trained by quality-metrics-based pairs (i.e., M). This shows that the quality-metrics-based pairs are indeed different from the human preference pairs. Therefore, research on human preference for graph layouts should be investigated further.

B.2 Large graphs
For mesh and large scale-free graphs, the test accuracy is very high, i.e., (86.55 ± 3.2)% and (98 ± 4.47)%, respectively. For instance, Fig. 4 shows two examples that our three trained models (i.e., M, HP and M+HP) succeed to predict the ground truth human preference.  Figure 4: Examples of the test layout pairs for mesh G 5 and large scale-free graph G 3 , where our three trained models succeed ( √ ) to predict the ground truth human preference labels. In each row, the layout on the left is more preferred by human than layout on the right.
To compare the prediction results based on layouts, we further examine the five layouts of large graphs in detail. From the five layouts of G 6 , G 10 , G 3 and G 15 in Fig. 2, we can see that layouts D i 0 , D i 1 and D i 4 are all visually much better than layouts D i 2 and D i 3 . Therefore, it was easy for human to make a preference from a pair between D i 0 (resp. D i 1 and D i 4 ) and D i 2 (resp. D i 3 ) (see Fig. 4).
On the other hand, there are difficult cases for human to decide which layout they prefer, i.e., layouts D i 0 and D i 1 are similar and slightly better than D i 4. In this case, a layout was chosen with conflicts (e.g., for the same layout pair [D 7 0, D 7 1], the preference among three participants was different, see Fig. 5 and Table 4).
Similarly, layouts D i 2 and D i 3 are of similar quality, both worse than the other three layouts, difficult for human to make a selection. In this case, a layout was chosen with a very low preference score (see [D 3 2, D 3 3] in Fig. 5 and Table 4).  Figure 5: Examples of the test layout pairs for mesh G 7 and large scale-free graph G 3 , where our three trained models succeed ( √ ) or fail (×) to predict the ground truth human preference labels. In each row, the layout on the left is more preferred by human than layout on the right.  Figure 6: Examples of the test layout pairs for sparse G 185 and biconnected graph G 42 , where our three trained models succeed ( √ ) to predict the ground truth human preference labels. In each row, the layout on the left is more preferred by human than layout on the right.
Overall, the test accuracy for small graphs is relatively low compared with large graphs. For instance, Fig. 6 shows two examples of layout pairs of sparse graph G 185 and biconnected graph G 42 .
The low test accuracy is because the layout pairs of small graphs are difficult for human to make a preference. This is because all the five graph layout algorithms produced visually similar layouts with similar quality for small graphs (see sparse graph G 188 , and biconnected graphs G 18 and G 65 in Fig. 3). Therefore, human tend  Figure 7: Examples of the test layout pairs for sparse graph G 185 and biconnected graph G 66 , where our three trained models succeed ( √ ) or fail (×) to predict the ground truth human preference labels. In each row, the layout on the left is more preferred by human than layout on the right.  to randomly choose a layout as the preference with a low preference score, e.g., Fig. 7 and Table 5 show that the preference score by each participant was very low for both the sparse graph layout pair [D 185 0 , D 185 4 ] and the biconnected graph layout pair [D 66 0 , D 66 3 ].

B.4 Limitation and future improvement
Overall our trained models perform quite well, nevertheless, there are some unexpected cases. We use three examples to address and analyze our limitations in detail. Fig. 8 shows the instance where all our models fail to predict the ground truth human preference on the test data. Table 6 shows the corresponding preference score.
From Fig. 8, we can see that each layout pair has almost the same visual quality. For example, two layouts of the pair [D 113 1 , D 113 2 ], are visually similar, and the preference score of preferred layout D 113 2 is very low. Due to such a low preference score, our binary classification accuracy for biconnected graph is lower than large graphs.
Furthermore, human preference can be subjective depending on personal preference, resulting in different preference scores, e.g., preference scores of 1 and 4 for the same pair [D 13 0 , D 13 4 ].
On the other hand, Fig. 9 and Table 7 show an example of an unexpected case, where only model M succeeds to predict the ground truth human preference. The preference score for the pair [D 187 2 , D 187 4 ] is 4.  Figure 8: Examples of the test layout pairs for G 13 , and G 113 , where all our three trained models fail (×) to predict the ground truth human preference labels. In each row, the layout on the left is more preferred by human than layout on the right.