{'fraction_non_alphanumeric': 0.032051988104416784, 'fraction_numerical': 0.008669929035608075, 'mean_word_length': 5.2259012539184955, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
KB4VA: A Knowledge Base of Visualization Designs for Visual Analytics

Visual analytics (VA) systems have been widely used to facilitate decision-making and analytical reasoning in various application domains. VA involves visual designs, interaction designs, and data mining, which is a systematic and complex paradigm. In this work, we focus on the design of effective visualizations for complex data and analytical tasks, which is a critical step in designing a VA system. This step is challenging because it requires extensive knowledge about domain problems and visualization to design effective encodings. Existing visualization designs published in top venues are valuable resources to inspire designs for problems with similar data structures and tasks. However, those designs are hard to understand, parse, and retrieve due to the lack of specifications. To address this problem, we build KB4VA, a knowledge base of visualization designs in VA systems with comprehensive labels about their analytical tasks and visual encodings. Our labeling scheme is inspired by a workshop study with 12 VA researchers to learn user requirements in understanding and retrieving professional visualization designs in VA systems. The theme extends Vega-Lite specifications for describing advanced and composited visualization designs in a declarative manner, thus facilitating human understanding and automatic indexing. To demonstrate the usefulness of our knowledge base, we present a user study about design inspirations for VA tasks. In summary, our work opens new perspectives for enhancing the accessibility and reusability of professional visualization designs.


INTRODUCTION
V ISUAL analytics (VA) combines data mining and visualization techniques to help users with data exploration in different applications, such as biology [1], [2], sports [3], and explainable AI [4], [5]. To represent the complex input data, researchers in visual analytics have contributed many advanced VA systems that are equipped with highly customized visualization designs for insight discovery [6]. However, designing effective VA designs is highly challenging and demanding, requiring close collaboration between experienced visualization practitioners and domain experts. The design process comprises multiple stages, including requirement summarization, system design, implementation, and feedback analysis [7].
The key to creating an effective visualization is to map the data and tasks derived from domain problems to visual elements [8]. Recent advance in visualization empirical studies and recommendations has proposed approaches to understand or automate such a mapping process (e.g., [9], [10]). However, these studies focus on recommending common charts and low-level analytical tasks such as finding distributions. These studies cannot work well for visual analytics that deals with complex datasets and tasks [11].
Researchers in visual analytics have contributed many VA design solutions for complex VA problems. The VA designs range from simple and basic charts to innovative designs with customized layouts and compositions. It remains unclear how visualization designs are configured, what tasks different designs can achieve, and how data is encoded with these designs, making it difficult to reuse and reproduce these designs.
Therefore, to facilitate efficient visualization design for visual analytics, we aim to construct a comprehensive visualization knowledge base by collectively considering tasks, data, and visualizations [8]. Given that visual analytics is a large topic, this paper primarily focuses on analytical tasks, data encodings, and visual structures of the designs, which are the key design considerations in visual analytics. Though other artifacts, such as algorithms, interactive linkings, and data flow within/between views, are also important to a VA system, we leave the analysis of these artifacts to future research. For knowledge base construction, we collected VA designs from the research papers published in the IEEE VAST Conference, which is one of the top venues for visual analytics and visualization design. As a starting point, we currently focus on the papers during the past five years (i.e., from 2016 to 2020).
We encounter two major challenges when constructing the knowledge base. First, it is unclear whether referring to existing designs can help users with designing their own visualizations and how to specify the visualizations for efficient design retrieval. Therefore, we first constructed a prototype knowledge base with labels of task types [12], data types [13], and visualization types [14], [15]. With the prototype, we conducted a workshop study with 12 visual analytics researchers, most of whom have published arXiv:2211.02567v1 [cs.HC] 3 Nov 2022 papers in the IEEE VAST conference as the first author. The participants were asked to design visualizations to accomplish tasks adapted from IEEE VAST Challenge 2021 (mini-challenge 2 [16]). With the study, we validated the usefulness of a VA knowledge base and collected the requirements for constructing such a knowledge base.
Second, it is challenging to specify and describe complex VA designs. Different from basic visualizations (e.g., bar charts), a VA design might contain a hybrid use of different visual elements, such as composite visualizations [14] and glyphs [17]. Building upon the user feedback from the workshop study, we formulated a specification for VA designs by extending Vega-Lite, a well-established declarative language with JSON syntax. The specification enables an expressive characterization of visualizations with analytical tasks, data, visual encodings, and layouts. We iteratively labeled the visualization designs by analyzing the corresponding descriptions in the papers.
As a result, we gained 442 visualization designs from 124 VA systems. We conducted an in-depth analysis of the knowledge base and obtained insights into the patterns of visual analytics designs. To further demonstrate the usefulness of the knowledge base, we developed an interface for KB4VA and conducted a user study with 12 participants. Specifically, we asked participants to retrieve designs for specific analytical tasks, data types, mark types, etc. The participants were allowed to explore the retrieved designs freely and requested to explain the visual encodings. User feedback showed that the knowledge base could help them retrieve diverse and useful visual designs, and our specification facilitates their understanding of the visual encodings. In general, the major contribution of this paper is an opensource knowledge base that covers analytical tasks, data types, visual encodings, and compositions of VA designs. Specifically, the contributions of this paper include user requirements for understanding and indexing visualizations in VA, which are derived from a workshop study with 12 VA researchers; an open-source knowledge base for visual analytics, KB4VA 1 , with an analysis of the visualization compositions and property distributions; and a user study to demonstrate the usefulness of KB4VA, where the implications and lessons learned in design inspiration are also reported.

RELATED WORK
This paper is related to studies about visual analytics, task and visualization typologies, and visualization indexing.

Visualization Design in Visual Analytics
Since the analysis problems and data structures are getting more complex in recent years, visual analytics (VA) systems are equipped with more features to fulfill the analytical requirements. Therefore, researchers reflected on the scope and challenges of VA [18] and proposed a series of conceptual models. For example, Sacha et al. [6] have proposed a knowledge generation model to characterize VA systems and their use in sensemaking. According to the model, VA 1. https://kb4va.github.io/ systems should be well integrated into the human knowledge generation loop from hypothesis and action to derive the findings and insights. A VA system is composed of three components, namely, data, visualization, and algorithms, involving the pipeline of information visualization and the process of knowledge discovery and data mining.
To design visualizations that are compatible with the knowledge generation pipeline [6], Munzner [8] has proposed a nested model for visualization design and evaluation. The nested model consists of four stages: 1) domain problem and data characterization, 2) operation and data type abstraction, 3) visualization design, and 4) algorithm design. The first two stages are considered different levels of abstraction of the data and tasks. With the abstracted data and tasks, the design choices of visualizations can be further derived based on the theories in information visualization, such as expressiveness and effectiveness criteria [19] and the rules of visual mapping [20], [21]. The nested model provides prescriptive guidance for visualization experts in constructing VA systems. Inspired by the model, we construct a knowledge base of visual analytics describing visualizations from their analytical tasks, visual encodings, and compositions. Compared to conceptual models, the knowledge base we present is a unique contribution to the community for data-driven analysis and design inspiration to promote research on VA systems.

Analytical Task Taxonomy
Determining analytical tasks is critical in visualization design. High-level tasks are considered a general characterization of the analytical problems [8]. For example, Edge et al. [22] constructed a task typology from the perspective of activity theory. Other studies focus on low-level tasks (or atomic tasks) that are more practical and actionable for design. Amar et al. [12] and Chen et al. [23] proposed categorizations of low-level tasks in the analytics activities, which contain retrieving the value, finding anomalies, etc. Instead of focusing on specific task levels, some research describes analytical tasks from several different dimensions. For example, Schulz et al. [24] proposed a design space for visualization tasks with five dimensions, namely, goal, means, characteristics, target, and cardinality. Similarly, Rind et al. [25] conceptualized the task design space from different dimensions, including perspective, abstraction, and composition. A more well-known taxonomy is the multi-level visualization typology proposed by Brehmer and Munzner [26], which characterizes the analytical tasks from levels of why, what, and how. The why level describes general analytical actions and goals (e.g., explore and identify), the what level indicates the analytical targets (e.g., distribution), and the how level characterizes the method adopted to solve the tasks (e.g., visual encodings and interactions). The why and what levels are usually combined together to describe the tasks and named as action-target model [7], [21].
In this work, we follow the idea of multi-level task typology and consider both high-level and low-level tasks at the same time. Specifically, for why and what, we adopted the action-target model to classify the tasks (e.g., compare distributions). For the how level, we contribute an expressive specification method based on Vega-Lite [13] that describes the visual encodings and visual layouts of the visualizations to facilitate understanding and readability.

Visualization Taxonomy and Specification
The classification of visualizations has been studied for a long time. For example, Borkin et al. [15] classified visualizations into 12 categories, such as Area, Bar and Circle, each comprising multiple sub-types. However, the designs of visualizations for visual analytics usually have novel layouts and complex compositions. Chen et al. [27] have attempted to map each view in VA systems to a specific visualization category in Borkin's taxonomy. However, they discovered that a view might be ambiguous to a specific category because many designs contain various visual components of multiple categories. They reflected on the categorization and proposed to follow Javed et al.'s theory of composite visualization [14] to characterize the visual designs in further research. Based on this reflection, we regard the visualizations in VA systems as composite visualizations and characterize the relations between the components with a hierarchical specification.
Visualization specifications are fundamental in visualization systems, indicating the visual mappings from data to visual channels and the visual layouts. Mackinlay [19] formulated the visualization specification as a graphical presentation problem and adopted relation tuples to specify the data features and visual encodings. Heer et al. [28] proposed using declarative languages to describe and specify the visualizations, which is intuitive for the programmers. After that, Bostock et al. [29] delivered D3, a programming grammar to operate on the graphical elements of document object model pages. To further reduce the burden of visualization specification, Satyanarayan et al. [13] proposed Vega-Lite, a JSON-based declarative programming language, by which users can render a visualization with even several lines of JSON text. After that, specifying visualization using JSON files becomes widely used, and similar languages are evolving, such as ECharts [30]. In this paper, we refer to the syntax of Vega-Lite and extend their specifications to support various visualization designs in VA systems.

Visualization Indexing
Given that visualizations usually have complex structures of visual components, numerous studies investigate the indexing and searching of visualizations. The most intuitive way for indexing is by assigning tags to the visualizations. Many visualization datasets collect visualizations and categorized them by types, such as MASSVIS [15], VizNet [31], VIS30K [32], and VisImages [33]. Assigning tags to visualizations is useful for training machine learning models, but the tags are limited in analyzing visualization configurations, such as visual encodings, compositions, and the corresponding tasks, which are the key aspects for understanding the designs of visual analytics. Many Eyes [34] is a platform for creating and learning visualizations. The platform allows users to upload data, create visualizations, and discuss the results. Besides, many survey papers index visualizations and deliver search engines for readers to review the collected publications and designs, such as tree visualizations [35], geospatial visualizations [36], and dynamic graphs [37]. In this work, we broaden the scope of specific survey topics and exhibit common design patterns without the constraints of application domains.
Computational methods have been used to extract and index visualizations. For example, when retrieving SVG charts, to ensure both the similarity of visual structures and data distributions, Li et al. [38] proposed a method based on graph neural networks for feature modeling. Hoque et al. [39] collected visualizations that are implemented by D3.js and parsed the hierarchical structures of the visualizations. The D3 search is useful when users want to develop a visualization with D3. Differently, our motivation is to help VA researchers understand and design VA designs. The visualizations we collected are bitmap images, whose structures are inaccessible. Moreover, we focus on analyzing visualizations in the context of visual analytics, which poses higher requirements for data labeling. Specifically, it not only requires labeling the meta information like chart positions but the information related to visualization literacy (e.g., visual encodings and tasks). Our efforts form a valuable knowledge base of visualization designs from state-of-the-art VA systems.

KNOWLEDGE BASE CONSTRUCTION
We manually constructed the knowledge base from a preliminary dataset, VisImages [33]. Existing studies have attempted to reverse engineer visualizations from bitmap images for indexing or retargeting them. For example, Savva et al. [41] proposed using image patch features to classify charts and extract chart data. ChartSense [42] classifies the chart types using CNN. Poco et al. [43] extracted color configurations of map visualizations from images and improved the color encodings for better visual effects. For additional chart types, such as bar charts and line charts, Poco et al. [44] proposed a CNN-based method to recognize chart types and marks for redesigning. These methods demonstrate great potential in automatically parsing and indexing visualizations. However, the views in VA systems are composed of multiple visualization types, which are difficult for automatic recognition [33]. More importantly, this work focuses on recognizing visual encodings and structures, which are represented in a nested JSON format. Existing methods cannot achieve the reverse engineering of such kind of semantic information. Therefore, in this work, we choose to manually annotate the data by reading the original paper of the VA designs.
We annotated the data in three steps, as shown in Figure 1. First, we identified the papers that propose visual analytics (VA) systems, whose paper types are commonly referred to as applications or design studies. VisImages [33] consists of bitmap images collected from IEEE InfoVis and VAST, which are top venues for visualization and visual analytics. We chose the papers in the IEEE VAST from 2016 to 2020, the primary venue for VA research. For each paper, we selected one figure containing the complete system interface, which was usually the teaser. With the system figures, we annotated the area of different views with bounding boxes one by one. In most cases, a view is assigned with a specific name for identification. However, a view sometimes consists of several independent sub-views. If the data  To construct the knowledge base, we conducted a workshop study with 12 VA researchers from the visualization community and collected the requirements for understanding and indexing visualization designs. With requirements, we annotated the designs with a declarative specification and adopted cross-validation to ensure their correctness. The annotation of AxiSketcher [40] is shown, for example.
among sub-views are not directly related (such as sharing the axes or connected with visual links), we decompose the view into sub-views for different visualizations. Each view is regarded to be an individual visualization and is the basic analysis unit throughout the paper. After the annotation, we obtained an image collection of 442 visualizations from 124 VA systems.
Second, we constructed a prototype knowledge base for a workshop study in order to understand the necessity and requirements of constructing a VA knowledge base. For the prototype knowledge base, we annotate the task types, data types, and visualization types (Subsection 4.1).
Third, we followed the requirements derived above and re-labeled the knowledge base by using specifications extended from Vega-Lite. The detailed definitions and the processing of the final specification are presented in Section 5.

REQUIREMENT ANALYSIS FOR KB4VA
We conducted a workshop study with VA researchers to 1) understand whether reviewing state-of-the-art visual designs can help visualization researchers in design inspiration and 2) obtain the requirements for understanding and retrieving VA designs.

Prototype KB4VA
The visualizations in the prototype knowledge base are characterized by tuples of their task types, data types, and visualization types. For the task type, we used a task taxonomy of data analysis [12] for annotation. The taxonomy contains ten types including retrieve value, derive value, filter, find extremum, sort, determine range, characterize distribution, find anomalies, cluster, correlate, and compare. To avoid bias during the annotation, we identified a task only when the original authors had mentioned it explicitly. After the annotation, 98.87% (437/442) visualizations contain at least one task type. For the data type, we represent the encoded data by its types in visual encoding channels. The data types include quantitative (Q), temporal (T), ordinal (O), nominal (N), and graph-related (G) data [13]. For a visualization design, we summarize the counts of each data type, such as "Q × 1, N × 2." We further labeled the visualization types of each visualization. For a composite visualization, we characterize its type by decomposing it into several visual components. For example, a scatterplot matrix can be considered as nesting scatterplots into a matrix, which is represented as a tuple: "(scatterplot, matrix)." For the visualization designs in VisImages, the bounding box and the type of each visual component have already been labeled based on the taxonomy of Borkin et al. [15]. Therefore, we use the original labels of visual components directly.

Study Setup
In the workshop study, we asked participants to imitate the process of designing visualization prototypes to accomplish VA tasks. We followed the think-aloud protocol and gathered qualitative feedback from participants.
Participants. We recruited 12 VA researchers who are postgraduate students (6 females and 6 males) are conducting research on visual analytics for various application domains like digital humanities, sports analysis, medicine, urban planning, etc. Ten of them have published papers in IEEE VIS as the first author. We asked participants to report on their experience in designing visualizations for data analysis. Based on the pre-study interview, 6 participants (P1, P3, P4, P8, P10, P11) are Ph.D. students who have more than three years of research for visual analytics, while 3 (P2, P9, P12) have less than one year and the remaining 3 participants have less than two years. Specifically, 3 participants (P1, P3, P12) own bachelor's degrees in design.
Problem. We selected the datasets and tasks with an adequate level of complexity so that visual analytics is required. Therefore, we used the problems in the minichallenge 2 from IEEE VAST Challenge 2021, which is a classic challenge in the field of visual analytics.
Procedure. The whole user study lasted about 60 minutes. Before the study, we first asked participants for their agreement to collect their design process, comments, and results for research use. After that, a study session started with a 15-minute tutorial introducing the prototype knowledge base (i.e., how we define the data representations, task types, and visualization types) and several examples illustrating how to use the interface. During the study, participants were asked to explain how they understood the problems and tasks and what and why visualizations they wanted to design. The participants needed to sketch the prototype visualizations on paper and illustrated how to use the designs to accomplish the tasks. The session ended with post-study interviews where we gathered qualitative feedback from participants.

Feedback on the Use of Knowledge Base
In the post-study interview, we asked participants to indicate how they have used the knowledge base to facilitate their design with a 3-point Likert scale, i.e., "no use" (the participant designed the desired visualization to fulfill the tasks without the help of the system), "moderate use" (the participant did not have an idea initially but got inspiration by multiple designs in the knowledge base), and "full use" (the retrieved designs from the knowledge base can be directly used with minor revision). The result is shown shown in Figure 2(left). In total, we obtained 36 visualization designs from 12 participants, with 16 (44.4%) designed with moderate use and 14 (38.9%) designed with full use of the prototype. The numbers demonstrate the usefulness of the knowledge base for designing visualizations.

Preference for Data, Task, and Visualization
During the study, we surveyed users' preferences for data, tasks, and visualizations for searching visualization designs of interest, as shown in Figure 2(right). "Data" is the most preferred. We discovered that 6 out of 12 participants ranked the data first, and five participants ranked it second. The participants all agreed that data is the most critical factor to consider during visualization design. P4, a senior urban planning analyst, commented that "data and algorithms are the most critical from the perspective of an expert". The other two seniors, P10 and P11, held a similar opinion. P5, a junior, also regarded the data as her priority consideration for visual design, saying that "the same data could be represented with different visual representations". However, knowing how many columns with different data types are encoded seems insufficient. P10 commented, "in real scenarios, data transformation would be performed, and it might be more important to tell how the data are mapped to the visual channels".
"Visualizations" are more preferred by designers. Four participants ranked visualization first, and three of them (P1, P3, P12) are senior designers with bachelor's degrees in design. An advantage of searching by visualization types is the consistency between the expectation and outputs. P1 liked searching by visualizations because "visualizations are very intuitive for understanding and I can imagine what results will appear". Among three dimensions, P3 highlighted his preference for visualization:" when I search by visualization, I prefer an exact match and exclude all designs without my selection". However, before applying the retrieved designs to their own scenarios, users have to understand the visual encodings. Both junior (P2, P3) and senior (P10, P11) Ph.D. students encountered problems understanding retrieved designs. P2 appreciated the textual explanation about the data types and task types, commenting that "the indication of visual encoding helps me to understand the design easily".
"Tasks" are mixed in understanding. Even though two participants ranked tasks first, half of the participants ranked it the last. A common problem is the gap between the original analytical question and low-level tasks. P12 commented "I am fuzzy about the task types, so I prefer to consider how to visualize all the data first". Besides, P10 pointed out that he cannot map tasks such as "obtaining an overview" to low-level tasks. P3 explained that he would have a different understanding of the tasks of the question. Those comments call for continued efforts to classify the tasks in VA systems for searching.

Requirements for the Specification
The workshop study exhibits the limitations of existing taxonomies, which motivates us to continuously improve and propose better visualization specifications. R1: Integration of data and visualization. Data and Visualization are the most preferred by users. Besides, all users' comments on data and visualization mention the relations between data and visual channels, namely, visual encodings. The indication of visual encodings can help users better understand how the data can be applied to the design. Inspired by the comments, we aim to propose an efficient method to describe the visual encodings in VA designs. R2: Specification of visualization composition. Many VA designs are composited visualizations, where composition relationships such as layouts are important in human understanding. For example, a common VA technique is the "glyph scatterplot" which each point is represented by a glyph. Such relationships are hard to describe using existing methods. Additional specifications on such a relationship are required. R3: More detailed descriptions of analytical tasks. Users' difficulties in mapping real analytical questions to lowlevel tasks might be because of the lack of "action" or high-level tasks, such as summarize, compare, and explore [26]. Comparing distribution and summarizing distribution might require visualization designs that are different in visual encodings and layouts. Besides, graph-related tasks are not investigated in depth. Therefore, we decide to incorporate additional task taxonomies with multiple levels to improve the specification.

KB4VA SPECIFICATION
Inspired by the above requirements, we improved the original data specifications. Specifically, we extended Vega-Lite [13], a declarative programming grammar for visualizations, to indicate the visual encoding and composition (R1, R2). In addition, we elaborated on task characterization using the multi-level typology of VA tasks [26] (R3).

Analytical Task Specification
Given that the low-level task taxonomy might be insufficient for users to understand VA tasks, we improve the task specification based on Brehmer and Munzner's taxonomy of VA tasks. Specifically, in their taxonomy, a VA task is described with three levels, namely, why, what, and how. The why level refers to the goals of VA, such as present, compare, and browse. This level also describes human behaviors during analysis, so it is named "action" [21]. The what level explains the analytical "targets" in VA, such as raw data, specific attributes, or data patterns. Our current task taxonomy can be regarded as a subset of these targets. Moreover, the how level describes the methodology used to achieve the "actions" and "targets," including visualization designs and algorithms. In this work, we focus on the specification of visualization designs for the how level and do not consider it a part of analytical task specification.  We use action-target pairs to describe the analytical tasks. The actions are the same as their definitions in the original taxonomy. For the targets, we refer to Amar et al.'s low-level task taxonomies for tabular data [12] and Lee et al.'s task taxonomies for graph data [45]. The classifications of actions and targets are summarized in Figure 3.

Visualization Design Specification
For visualization designs, we aim to identify visual encodings, which are the mappings from data to visual channels and layouts. Specifically, we regard each design as a composite visualization [14], [46] and firstly recognize the overall layouts (e.g., faceting) and decompose it into different visual components. Each component is an independent visualization of specific types, such as bar chart, line chart, Sankey diagram. For the designs of well-crafted glyphs that are not just combinations of different visualization types, we regard them as "others" type. Then we recognize the visual encodings for each component.
We use Vega-Lite specifications as a starting point because their JSON syntax is intuitive for representing visual structures. Specifically, it characterizes a visualization with the fields of "mark" and "encoding". In the field "encoding", the data "field", "type", and "aggregate" are further specified. Moreover, it supports the specification of basic visual layouts, such as faceting, concatenating, and layering.  We iteratively extended Vega-Lite specifications to cover the collected visualization designs and annotated the specification for each design. The process of extension and annotation consisted of four stages. In the first stage, four authors annotated visualizations with original Vega-Lite specifications. We discovered that the Vega-Lite specification did not support the specification of graph-related visualizations, such as Sankey diagrams and tree visualizations, which are common visualization types in VA designs. In addition, complex visual compositions are not supported, such as embedding glyphs in graph nodes. We attempted to extend the specifications based on the failed cases. We extended the original Vega-Lite specifications from three perspectives. First, we added additional data types (e.g., relational data) and regarded "node" and "link" to be the two visual channels of graph-related visualizations. We further specified the properties of the nodes and links, such as positions and widths, under the "node" and "link" labels. An example specification is presented in Figure 5A. Second, to handle complex visual compositions, such as embedding glyphs in graph nodes [47], we have added a composition type "nested". The nested visualizations are represented by specifying the "parent" and "children" components. We also use a key "canvas" to indicate which elements of the parent are the embedded children components. The specifications of the compositions, i.e., "concat," "layer," and "facet," and "nested" composition, are shown in Figure 5B. Third, the mark types supported by Vega-Lite are also insufficient for the representation of VA designs. We have extended the mark types by adding new ones like graph, Sankey, and radar, referring to the typologies proposed by Borkin et al. [15]. It is noted that the newly added mark types are not graphical primitives that are elemental building blocks of the visualization. Instead, some of them are "macros for complex layered graphics that contain multiple primitive marks" [13], which are consistent with the definitions of Vega-Lite. Figure 4 provides an example with complex composition relationships such as nesting pie charts into node-link graphs. In the second stage, we annotated the visualizations independently and further identified the cases that are not covered by the extended specifications. We fine-tuned the specifications and maintained a document providing rules and annotation examples for different cases. The authors synchronized and resolved the disagreements through weekly discussions. After discussion, we updated the document and revised the annotated results for these cases. Third, after a round of annotation, all authors exchanged and double-checked the annotation results. We ensured the annotation was updated with the latest document and resolved disagreements. Finally, an author double-checked the results again for all details. As a result, we obtained KB4VA specifications ( Figure 4) and annotated 442 visualization designs with the specification.
During the annotation, we followed the idea of obedience. Specifically, we annotate the visualization with the original Vega-Lite specification as much as possible. Compared to the original Vega-Lite specification, we extensively expand the properties of composition, marks, encoding types, and data types based on the VA designs we collected. It is noted that the Vega-Lite specification also provides powerful operators for data transformations, such as filtering. However, in VA research, many techniques involve complex data processing methods, such as dimensional reduction, and the classification and identification of these methods are challenging. In this work, we currently focus on visualization designs and only use part of the Vega-Lite specification to characterize visual encodings (e.g., excluding style-related parameters) that help for better visualization indexing and understanding.
Moreover, we followed the idea of minimization, i.e., choosing the one with the least number of duplications, to address the problem when there are multiple solutions to a visualization. For example, the component in Figure 4-B2 can also be regarded as a layered visualization with a density plot, a graph, and a facet visualization whose elements are pie charts from the perspective of implementation. The node positions of the graph and the positions of the pies both repeatedly encode the row attribute and column attribute. Referring to the original descriptions [48], the pies without links would fade out, indicating a one-toone mapping of the graph nodes and pies. Therefore, it would be more appropriate to consider the visualization as a layered visualization composited by a density plot and a nested graph visualization (pie charts embedded into the nodes). For another example, a faceted visualization can be considered as a concatenation of a list of similar visual components. If we represent the chart with "concat", we have to duplicate the specifications of similar visual components multiple times. Instead, it is much neater and more accurate by using "facet" to represent it in the context of data visualization.

DATA ANALYSIS
With the knowledge base, we conduct a data analysis of the "design demographics" [39], [49]. We report on the statistics, including the frequency and co-occurrence patterns for the analytical tasks, visual encodings, and compositions.

Overview
We first analyze the specifications to understand the composition of visualization designs in visual analytics.

How expressive is the original Vega-Lite specification?
We investigated the annotated designs and opted to understand whether these visualizations can be implemented with common compositions (layer, concat, and facet) of basic mark types with Vega-Lite. We discovered that only 38.2% (169/442) designs could be specified with original Vega-Lite specifications. The results indicate that researchers tend to use novel techniques in VA systems to visualize the data, which demonstrates the unique value of our specifications and KB4VA. The limitations of Vega-Lite mainly lie in the limited mark types and the lack of support for graph-related data and nested visualizations. The limited expressiveness of declarative visualization grammars may be a reason for the result that the visualization designs of most existing VA systems are implemented with lower-level Javascript libraries (e.g., D3.js), as they could provide flexible customizations for the designs.
How many VA designs are using composite visualizations? Composite visualizations combine multiple visual components together along specific directions (e.g., "layer", "concat" and "facet" in Vega-Lite) or in a hierarchical manner (i.e., "nested"), which can make the visual components well-organized and easy to interpret. As shown in Figure 6A, 63.8% (282/442) of the VA designs contain composite visualizations. We focus on the number of composition labels in a specification. Figure 6A shows that 34.8% (154/442) of visualizations contain only one composition. 3.6% (16/442) visualizations contain at least four compositions. A visualization can have several types of compositions (four different types in total). Only one visualization has used the maximum number of different composition types, which is four ( Figure 6B). Figure 6B shows that most visualizations have no more than two composition types.
How deep is the composition hierarchy of VA designs? As described in Subsection 5.2, the visualization is represented in a hierarchical JSON syntax with multiple levels of composition. For example, the visualization presented in Figure 4 has a composition level of three. For composite visualizations, 54.6% (154/282) have a level of one, 34.0% (96/282) have a level of two, and 8.1% (23/282) have a level of three. Only 9 visualizations have a level of four, which is the maximum level of the hierarchy. The numbers indicate that most composite visualizations only use one or two levels of composition. Adding more levels of composition demands encoding more data columns, which might go beyond the requirement of analysis scenarios. Moreover, more levels of compositions increase implementation difficulty and visual complexity drastically.
Do more compositions mean more tasks achieved? Investigating KB4VA, we discover that all visualizations have at least one action-target task. The ones with composition achieve 1.49 tasks on average, while the ones without composition are designed for 1.33 tasks on average. Figure 6D shows the average number of tasks vs. the number of compositions. Overall, the number of tasks to be solved will increase with the increase of compositions.

Frequency Analysis
We then report on the frequency of different property values in KB4VA specifications.
Actions. As shown in Figure 7A, present is the most frequent action, followed by compare and identify, and summarize. The results show that many designs are merely used for data exhibition. Besides, comparative analysis, identification of data patterns/attributes, and displaying data overview are popular. For the goal of searching, explore is the most popular one, which stands for exploratory analysis. Interestingly, we did not discover designs that are used for enjoy, which shows the difference between VA and infographics.
Targets. From target distribution ( Figure 7B), we discovered that the most popular target is value for tabular data, which refers to visualizing specific attribute values or metric values that are useful for the domain problems. distribution and correlation are the second and third popular targets. The results demonstrate that understanding the correlations and distributions of the attributes are the key indicators for data patterns in VA. For graph data, links and the whole graphs are the most frequently visualized targets.
Compositions. Composition distribution is presented in Figure 7C. Facet, which organizes visualizations of the same types by rows and columns, is the most used composition type in VA. Concat is the second popular type, which refers to placing visualizations with different types side by side. Nested composition is not covered by the original Vega-Lite specification. Although it has the smallest proportion, it accounts for more than 10% of the knowledge base.
Marks. In a composite visualization, each visual component is regarded as a specific mark type. We display the mark types that have more than 20 records in our knowledge base ( Figure 7D). The distribution demonstrates that bar, point, line, and rect are the most popular mark types, which are also basic mark types in Vega-Lite. For the types that Vega-Lite does not cover, graph and unit [50] visualizations ranks 5 th and 7 th among all types. The type others ranks 9 th , indicating that glyph visualizations are also commonly used in VA designs.
Channels. We show the visual channels with more than 10 records in Figure 7E. The channels x, y, and color are the most popular. The channels link and node are also frequently used because of the graph-related visualizations, such as Sankey diagrams, graphs, and trees.
Data Types. Among all data types, quantitative data and nominal data are the most frequently encoded in the visualizations ( Figure 7F), followed by node data, which is commonly used in graph data.
Aggregate. Aggregate types are labeled based on Vega-Lite aggregation operations ( Figure 7G). Aggregate count and bin are the most frequently used types, which are usually because of the visualization of histograms. Other aggregate types are not frequently used in VA designs. The reason might be that complex data processing methods and metrics are adopted in VA, instead of basic aggregation strategies, such as sum, median, and variance.
Field Names. Field names are the terms used in the original VA research papers describing the data columns. We show the word frequency of field names using word cloud ( Figure 7H). From the word cloud, we immediately discover that the words time, feature, metric, and dimensionality reduction have a relatively large size, indicting the values about these terms are frequently used in VA research.

Co-occurrence Analysis
We further conduct co-occurrence analysis on the knowledge base to provide a general picture of the relations among task, composition, mark type, and visual encodings ( Figure 7I, J, K, L). In this work, we do not compute the conditional probabilities by normalizing the co-occurrence matrices by the sum of the rows or columns because the great distinction among the amounts of different types might introduce a misinterpretation of the probabilities.
Action versus Target. Figure 7I shows the co-occurrence patterns between actions and targets in analytical tasks. From the matrix, we discover that although values is the most popular target for many analytical actions, present distribution has the highest co-occurrence, followed by compare values and present correlation.

Action-Target versus Mark Type.
We further show the co-occurrence between top action-target pairs and mark types, trying to identify popular visualizations for different analytical tasks. In the co-occurrence matrix ( Figure 7J), the top three pairs is (present distribution, bar),(compare values, bar), and (present correlation, bar), which are all about bar. The bar charts are the most popular in the top analytical tasks. However, for the tasks of presenting groups, scatterplots are more popular than bar charts, which are commonly seen in visualizations with dimensional reduction.
Composition versus Mark Type. Mark types exhibit different co-occurrence patterns with compositions (Figure 7K). From the co-occurrence matrix, we discover that the most co-occurring pairs are (concat, bar) and (facet, bar), indicating that bar charts are mostly concatenated with other visualizations or organized by rows and columns. Moreover, (layer, point) ranks third. From the samples, we discovered that scatterplots are commonly overlaid on map visualizations (geoshape) to show the places of interest. For nested visualizations, graph is the most co-occurring one. This is because, in graph visualizations, smaller visualizations are commonly embedded in the nodes or links to show additional information.
Data Type versus Visual Channel. Data types are highly related to visual channels. From the co-occurrence matrix ( Figure 7L), we discover that quantitative data is usually visualized with x and y, while nominal data is more commonly encoded with color. In addition, compared to y channels, link and node are undoubtedly co-occurring with relational data and node data, respectively.

EVALUATION
We conduct a user study to evaluate the usefulness of the KB4VA knowledge base. To allow the users freely explore the knowledge base, we have designed an interface for searching and exploration.

Interface Design
The interface includes a filtering panel ( Figure 8A), a specification view ( Figure 8B), a gallery view (Figure 8C), and a detail view ( Figure 8D).
The filtering panel ( Figure 8A) supports searching visualizations by selecting values for different keys, which is convenient for retrieving the visualization designs that meet the conditions. For example, searching for "bar" can retrieve visualization designs with bar charts. However, the results retrieved with several filters might be unsatisfactory because such keywords-based retrieval does not consider the nested structure of specifications [51]. For instance, searching by the keywords "facet" and "bar" might not necessarily retrieve a faceting bar visualization, but possibly a faceting visualization concatenating with bar charts.
To solve this problem, we develop a specification view ( Figure 8B) that enables users to input specifications with JSON syntax. Then the search engine will recursively match the key and values of the JSON input and retrieve the ones matching both the structures and values. The input also supports wildcard pattern matching by specifying the value with a character "*". However, this search method might not be friendly for novice users. Thus, we keep both methods.
The retrieved results will be displayed in the gallery view ( Figure 8C). Each retrieved design is shown on a card in the gallery. When hovering on the cards, a snippet showing the design preview will pop up. When clicking on the "detail" button on the card, a detail view ( Figure 8D   visualizations of the other views. The specification is also displayed next to the visualization design to help users understand the visualization design. The right-most side provides additional information, including the paper title, figure caption, analytical tasks, and keywords.

Study Design
In this study, our goal is to understand if participants can better understand the retrieved designs using KB4VA specifications and use the system to find design inspirations for VA problems. Specifically, we ask participants to search for visualizations to solve six well-designed VA problems. The VA problems have varying complexity and are related to specific analytical tasks, mark types, composition types, or data types, such as "to find visualizations that encode a three-dimensional dataset with a nominal, a quantitative, and a temporal field" and "to find VA designs for comparing distributions". Given the retrieved results, participants were asked to explore the results and select one visualization of interest for in-depth investigation, that is, to understand the design by reading images, specifications, and other metadata (e.g., titles and captions). Lastly, we ask participants to explain the visual encodings of the visualizations. Participants. We recruited 12 visualization practitioners and designers (U1-U12) who reported to have experience in creating visualizations for data analysis. The participants include 5 females and 7 males with various backgrounds, including literature, computer science, energy engineering, and digital media design. They are used to analyze data with toolkits such as Python, R, and MATLAB for data analysis. In addition, the libraries they use for data visualizations are Excel, Python matplotlib, and Javascript D3. Two of them report being familiar with Vega-Lite, and another two have tried Vega-Lite before. The participants in this user study all have adequate data visualization or design knowledge but with varying expertise in designing more complex VA systems.
Procedure. Each study consisted of two sessions, namely, a training session (15 minutes) and an experiment session (20 minutes). In the training session, we introduced the definitions of KB4VA specification, including taxonomies of analytical tasks, composition types, mark types, data types, and visual channels. Then we introduced the use of the interface. All participants were allowed to use and explore the data freely to get familiar with the interface. In the experiment session, each participant was asked to accomplish tasks. During the whole study, we followed the think-aloud protocol. Participants were requested to speak out about their understanding of visual encodings and their thoughts about the specification or interface when accomplishing tasks. The study ended with a post-study interview session as well as a questionnaire for rating the KB4VA from different dimensions.

Results and Feedback
The overall reactions from the participants were very positive, as shown in Figure 9. Eleven out of twelve participants strongly agreed that KB4VA helps to find useful visualization designs for achieving the VA tasks in the study. Similarly, 11 participants strongly appreciated the diversity of VA designs in KB4VA.
Nine out of twelve participants agreed that KB4VA specification is easy to understand. One participant (U8) gave the reason that "the design specification is easy to read, but I was not familiar with the terms at first. After completing the first several tasks, I gained more understanding of the specifications." One participant (U2) who strongly agreed with the statement said that he was familiar with the Vega-Lite, so he appreciated our extension as very helpful. However, one participant (U3) gave disagreement with this question, commenting that "I really do not like reading codes, although I can partially understand the meaning." The opposite opinions here indicate that although declarative languages were proved to be easy to understand [28], there is a learning curve for some novices. To better explain the visualizations, future research should propose more intuitive presentation methods to explain visualizations (e.g., [52]).
Although it took some time to understand, all participants (strongly) agreed that the specifications are helpful in interpreting visual encodings. A participant (U5) who had not used Vega-Lite before commented, "the specifications are helpful for understanding visual designs". A participant (U4) thought that code was better than text, commenting that "reading specifications is much more efficient than reading captions or even original papers." Another participant (U8) appreciated the hierarchical structures of visualizations, saying that "it reminds me of HTML structures and the composition structures tell me how to implement such a complex visualization design if I need to." When opening the detail view, participants would first try to understand the visual encoding by reading images, which was often difficult in VA designs. U6 said, "I first read the image but did not note that those circles were glyphs until I moved to read specifications. Without the specifications, I could have misunderstood the encoding." Eleven out of twelve participants agreed that the interface is intuitive and easy to use. One holding a neutral opinion said that she was not familiar with the values of filters, and it took time to learn. Most participants strongly agreed that the search results were satisfactory. U1, who has rich experience and expertise in VA systems, said that "I could imagine the outputs," but "other visual components accompanied with the visualizations I expected are inspiring."

LESSONS LEARNED AND LIMITATIONS
We discuss the lessons learned from understanding VA designs and the limitations of our work.

Towards More Effective Visualization Retrieval
During the post-study interview, we obtained insights about designing a more effective search engine system for VA from three perspectives: visualization understanding, progressive query, and design retrieval for users with different expertise.
Exploratory Visualization Design Query. Given the complexity of VA designs, the query should be in an exploratory manner. In the pilot study, P11 complained that the complex results overwhelmed him initially. P10 suggested that the initial results should be simple and easy to understand, for example, retrieving a representative type of visualizations. If the user is interested, more variants with compositions ("genealogy of visual design", commented by P11) could be recommended. The situation got better after fine-tuning the specification since we provided more options for users to query by tasks, visual channels, and compositions. In the second study, U1 said that "the retrieved visualizations were as expected in the types and layouts". The extended specification makes the search for variants a possible option. U2 indicated that novice users can benefit from the initial searching results, saying that "for interesting visual structures, users can copy the codes of specification in retrieved designs and further search by specifications." Furthermore, P11 commented on the overall searching mechanism from the perspective of design idea convergence, saying that "it lacks a methodology to guide the users in visualization design, such as telling users how those designs evolve along the research history." Recording, analyzing, and visualizing the searching logs might be helpful for users to be aware of the searching and design process.
Requirements with Different Expertise. Considering that visual analytics is interdisciplinary of visualization design and data analytics, we witnessed a gap between the users with different application domains in the workshop study. First, from the perspective of data transformation, P3, a senior designer, is expected to query visualizations by data processing algorithms. However, P4, an expert in urban planning, thought that he could easily imagine the data structure of encoded data, so searching by data types is more important. From the perspective of the system output, P4 expected to see the pattern of how real data is visualized with the retrieved visualization. In contrast, P12, an experienced designer, would like to obtain queries in SVG format, with which she could easily modify the styles to facilitate the design. Therefore, future research should better understand the user needs and requirements in developing visualization retrieval systems (e.g., [53]).

Towards Efficient Visualization Understanding
After retrieving visualization designs, the next step is understanding the design, that is, visual encodings.
Visual Explanation of the Visual Encoding. Explaining visual encodings with annotations directly overlaying on the design might be effective for understanding. In the workshop study, both junior (P2, P3) and senior (P10, P11) Ph.D. students encountered problems understanding the retrieved designs. The improved specification with explicit representation of visual encodings lowers the burden of visualization understanding, but users also indicated that it would be better if there is visual cues such as highlighting when hovering over the specification, which is similar to the "inspect" function in web developer tools. However, achieving such a visual explanation requires additional specifications on the positions of different levels of visual components, which is beyond the scope of this work. Leveraging computer vision and natural language processing models to construct the semantic relations between components might be a possible research direction for the explanation [54], [55].
Visualization Designs Comparison for VA Research. In the visualization community, VA designs have long been criticized for their over-crafted designs for specific domain problems [56]. To partially alleviate these concerns, retrieving existing designs and justifying the novelty and necessity of new designs through comparison might be helpful. VA systems are often significantly different as a whole, but they might share similarities in specific visual components and tasks. In this work, we attempt to decompose and index the visualization designs in VA with state-of-the-art information visualization theories, i.e., multi-level typology of visual analytics tasks and Vega-Lite specification for visualizations. Our "anatomy" of the visual designs enables identify common patterns between designs from the perspectives of analytical tasks and visual encodings. Both VA researchers and visualization practitioners in our study (P1, P2, U8, U9, U10) commented that KB4VA could help them to review existing designs and accelerate the design process. KB4VA potentially helps to raise more discussions on the theories of VA designs, e.g., to identify design genelolgy [57] and find common design patterns.

Towards More Powerful Visualization Synthesis
In this work, we investigate the expressiveness of Vega-Lite under the condition of generating visual analytics designs. The results show evidence that Vega-Lite specification is powerful in describing visualizations in VA with adequate extension. Vega and D3 have a more expressive power but at the cost of a more skewed learning curve. The KB4VA specifications still require a practical library to demonstrate the value of abstraction. Future research should propose more powerful visualization specifications that support more advanced visualizations while reducing users' learning curves.

Limitations
As a first attempt to construct a VA design knoledge base from the perspectives of data, tasks, and visualizations, our work has several limitations that warrant future research.
Visual Relationships. Linking between views or visual components are essential features to support users' interactive sense-making. The relationships between different visual components are often reflected by their sizes, positions as well as interactions. For example, the main view may take the most space, or a visualization view in the center may be the main one, and others might be the auxiliary ones. Also, interactions between visualizations can also indicate the relations between them. However, it is difficult to recognize the interactions from VA designs, even in an entirely manual manner, since not all interactions between two views are introduced in the original papers. Moreover, the use of static images hiders our analysis of configurable VA systems (e.g., Turkay et al. [58]), as the configuration frameworks are not reflected in the images. To facilitate better analysis of view relationships, it requires parsing live VA systems and constructing the data flow between views.
Data Transformations. Our specification focuses on the transformed data in the visualizations, which can be different from the original data table. For example, to visualize high-dimensional data with a number of quantitative columns, users might prefer to use a parallel coordinates plot (PCP). However, in our specification, a PCP is regarded as a variant of line charts with nominal fields of the xaxis and quantitative fields of the y-axis. To convert the source data to the representation of encoded data, table transformation operations such as "fold" and "unfold" are required [59], [60]. Moreover, VA usually involves novel models and algorithms for data preprocessing and insight discovery. Thus, future work might study how the data is transformed with models or operations from source [61].
Scalability. In this work, the scalability of annotation is limited because it requires extensive visualization knowledge for annotating such a fine-grained knowledge base. Our work is rooted in the fact that there lack practical rules and guidance in decomposing visualization designs in VA. As a starting point, we manually annotate and finetune the specification iteratively with a workshop study, aiming to construct a solid foundation for the indexing. Such manual efforts were expensive and resulted in relatively small dataset size. In the future, we plan to improve our knowledge base with a combination of machine learning methods. For example, VisImages [33] collects a large number of visualizations from research papers and shows the potential to use computer vision models to decompose VA systems. In addition, there are attempts to extract the visual structures of the designs, such as maps [43], charts [41], [44], and PowerPoint slides [55]. It is possible to adopt deep learning models to detect the positions of visual elements and reconstruct their relations.

CONCLUSION AND FUTURE WORK
We built and published a knowledge base, KB4VA, from visual analytics research papers. The knowledge base features a specification for describing complex VA designs from the perspectives of visual encodings, composition, and analytical tasks. The specification is constructed iteratively through a workshop study with 12 VA researchers. The knowledge base provides opportunities to understand and utilize state-of-the-art visualization designs, which is demonstrated through a user study. However, given that designing a visual analytics system is a complex procedure, we note that our work is the first step towards understanding and analyzing VA systems. We hope that our specification, knowledge base, and lessons learned could provide a helpful foundation for further research.
There are several directions for future research. First, it remains an open research problem to reverse-engineer the VA systems. From the perspective of computer vision, recognizing the position and types of different views [27], [32] and visualizations [33] from bitmap figures is possible. However, it is still unclear how to recognize the composition relationships and group visual elements to form a structural and editable template of VA systems. Another research direction is machine learning-based methods for visualization recommendation in VA. This direction calls for continued efforts to contribute large-scale datasets and novel techniques to enable training machine learning models.