{'fraction_non_alphanumeric': 0.043815475922845196, 'fraction_numerical': 0.015624297468639, 'mean_word_length': 4.545125903764648, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 0, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Table2Charts: Recommending Charts by Learning Shared Table Representations

It is common for people to create different types of charts to explore a multi-dimensional dataset (table). However, to recommend commonly composed charts in real world, one should take the challenges of efficiency, imbalanced data and table context into consideration. In this paper, we propose Table2Charts framework which learns common patterns from a large corpus of (table, charts) pairs. Based on deep Q-learning with copying mechanism and heuristic searching, Table2Charts does table-to-sequence generation, where each sequence follows a chart template. On a large spreadsheet corpus with 165k tables and 266k charts, we show that Table2Charts could learn a shared representation of table fields so that recommendation tasks on different chart types could mutually enhance each other. Table2Charts outperforms other chart recommendation systems in both multi-type task (with doubled recall numbers R@3=0.61 and R@1=0.43) and human evaluations.


INTRODUCTION
Creating charts for a multi-dimensional dataset (denoted as table) is a common activity in many domains such as education, research, engineering, finance, etc. To discover insights and perform routine analysis, people spend a huge amount of time constructing different types of charts to present diverse perspectives on their tables -such as the charts in Figure 2 created for Table 1a and 1b. Both data queries (selecting what data to analyze) and design choices (how to visualize selected data) are made during chart creation [8]. This tedious process requires experience and expertise in data analytics and visualization tools. For example, to compose the bar chart in Figure 2a, one has to first select the left-most three fields/columns from Table 1a, then choose bar chart type, map the three fields onto x and y axis, stack two value series one upon another, etc. To simplify chart composing, a long line of works tried to build machine learning models recommending data queries and/or design choices, such as DeepEye [9], Data2Vis [4], DracoLearn [13] and VizML [8]. However, most of them did not address the single-type tasks that each recommends one specific type of charts for a given table, including less used but meaningful minor chart types (e.g., area and radar charts). They only considered the multi-type task where a ranked list of few major types of charts (e.g., line, bar, scatter       Table 1a and Table 1b. and pie charts) are recommended together. But both single and multi-type tasks should be tackled for real-world scenarios: When facing a table for the first time, one usually has no clear idea about what chart should be created. In this scenario, an assistant could help leverage past common wisdom of what commonly composed charts could be created for the table -which is the multi-type task. For example, in Excel, the "Recommended Charts" button in Figure 1 is expected for this. Later with a clearer intention in mind, the main obstacle is the efforts needed to realize ideas through the complex charting process. Since lots of charting tools put chart type buttons / choices as the top entry points to chart composing (e.g., the chart type icons in Figure 1), guessing and suggesting auto-filling and completion of the details of a chosen chart type -which is the single-type task -could help save time and efforts from users. For example, when the first three fields of Table 1a are selected, after clicking the bar chart icon on Figure 1, a lot of efforts could be saved if the rest of design choices on field mapping and stacking could be done automatically, leading to the bar chart in Figure 2a.
When tackling the single-type and multi-type tasks with both data queries and design choices, there are three fundamental challenges. First, separate costs: It is memory and speed inefficient to design, train and deploy models for multi-type task and single-type tasks repeatedly and independently. Second, imbalanced data: The available data for different chart types are highly imbalanced. Four major types of charts (line, bar, scatter and pie) cover 98.91% of the available charts while others rarely appear because it is hard for non-experts to create them. Lack of data in minor chart types (area and radar) makes it hard to build high-quality models for them. Third, table as context: Selecting and visualizing data from a table depend on not only the data statistics, but also the semantic meanings of the whole table context. Proper models need to be designed to take table context into chart recommendation.
In this paper, we propose Table2Charts framework to learn  common patterns of chart creation -including both data queries  and design choices -from a large amount of (table, charts) examples and to recommend charts for each given table. In  §2, charts  recommendation is formulated as table to sequence(s) problem with next-action-token estimation to fill chart template(s). This formulation allows chart recommendation with partial intent when part of data queries and design choices are already given. Then in §3, as the estimation heuristic for beam searching, we design an encoderdecoder deep Q-value network (DQN) which selects table fields to fill template(s) via copying mechanism. All recommendation tasks share one encoder but have their own decoders, which addresses the separate costs challenge. The DQN is trained using mixed learning on the multi-type task of major chart types. By exposing its encoder part to the diverse source tables of different chart types, it learns shared table representations containing semantic and statistic information of table fields. Then the pre-trained table representations are transferred for type-specific decoders of single-type tasks, relieving the imbalanced data problem.
From the public web, we collect a large corpus of 266252 charts created from 165214 tables in Excel files and use a public Plotly corpus of 67617 charts from 36888 tables to verify the effectiveness of Table2Charts framework in §4. For each chart type, the recall for top-3 and top-1 recommendations are 59.99% ∼ 94.04% and 49.30% ∼ 79.72% on the single-type tasks. The multi-type task of recommending major chart types has 61.84% recall at top-3 and 43.84% recall at top-1, which exceed the baseline methods whose maximal recall numbers are 27.14% and 13.17% respectively. Human evaluation is also conducted to validate the precision of the proposed framework on 500 frequently visited web tables from a search engine. Lastly, through T-SNE visualization, we find that the DQN could learn shared table representations during multi-type task training for later transfer learning, thus improving the performance and saving memory occupation of single-type tasks. All these experiments and evaluations justify that Table2Charts could efficiently learn to help composing charts.
In summary, our main contributions are: • Table2Charts framework is proposed by us to learn human chart composing wisdom. It generates both data queries and design choices in an action sequence for multi-type and single-type chart recommendation tasks with the state of the art performance and efficiency. • To the best of our knowledge, we conduct the largest scale (165k tables and 266k charts from Excel corpus) training with diverse evaluations (on Excel, Plotly, and web table corpora) of chart recommend systems. • We show the feasibility of learning shared table representations (encoding table fields into embedding vectors) for enhancing down-stream data analysis tasks.

PROBLEM
To build machine learning models that learn patterns from large amounts of (table, charts) pairs, and to generate commonly composed charts for a given table, in this section we formulate singletype and multi-type chart recommendation tasks as table to action  sequences generation by filling chart grammar templates. A table here is an -dimensional dataset D which contains data fields F D = ( D 1 , · · · , D ). Each data field refers to an attribute of the dataset with its corresponding header name (attribute metadata) and data values (records). For example, each column from tables in Figure 1 is a data field with its first row as header.
To demonstrate our ideas, as shown in Figure 2, in this paper we pick four major and two minor chart types that appeared in common charting tools such as Excel. The major types are line, bar, scatter and pie charts. The minor types are area 2 and radar 3 charts.

Chart Templates
Although different types of charts exhibit distinct visual effects and behaviors, the essential actions for creating them from tables can be summarized into two categories: Selecting / referencing table fields and running specific charting commands / operations to organize and plot the selected fields. In this sense, a chart can be regarded as a sequence of actions on data queries and design choices.
Definition 1 (Action Space / Tokens). For an -dimensional table D, there are two categories of action tokens A D = F D ∪ C representing core actions of composing a chart: • Field referencing token ∈ F D that indicates a field is selected for composing chart. Then we can define how to represent different types of charts using these unified action tokens. Unlike flexible language modelling in NLP, here action tokens should be organized into a sequence according to specific grammar rules of a chart type. where ⟨grp⟩, ⟨f*⟩, ⟨f+⟩ and ⟨f⟩ are token placeholders: ⟨grp⟩ |= an operation ∈ G, ⟨f*⟩ |= | ⟨f⟩⟨f*⟩, ⟨f+⟩ |= ⟨f⟩ | ⟨f⟩⟨f+⟩, and ⟨f⟩ |= a field ∈ F D , means empty. The first ⟨f+⟩ or ⟨f⟩ segment is the y-field(s) and the second ⟨f*⟩ or ⟨f⟩ segment is the x-field(s). 5 Note that how x and y axes behave also depends on chart type: E.g., scatter and pie charts only allow one y-field; temporal records will be ordered by their timestamps along x-axis on a line chart; etc.

Definition 2 (Chart Grammar Templates
Hard constraints are also included in the template definitions to restrict heuristic beam searching (see §3). These could be any hand-written rules, such as the data type of a field mapping to y-axis is forbidden to be string type. Currently we only set field type and field number limitations and let Table2Charts models to learn the rest. More rules such as the ones in Draco [13] could be adopted as hard constraints to further improve the framework.
With the above definitions, now each chart can be written down as an action sequence. For example, the sequence of the bar chart in Figure 2a (created from There are more detailed charting aesthetics [20] to consider, such as shape, size, color, line width and type, etc. In this paper, rather than considering every detail, we mainly focus on the core parts of data queries and design choices -how to select and compose fields as axes of proper chart type -to study if Table2Charts framework can learn common wisdom in a data-driven way. Meanwhile, for the sake of simplicity, we only deal with database-like tables and referencing a whole field without filtering, aggregation, bucketing or ordering. All the above subtle aspects of analysis may not be well supported by our training data (Excel files on the public web, see §4.1) in both quantity (< 5% charts involve customizing them) and quality (the creators of these files may not be experts on inessential parts of charting). They can still easily be added as new tokens into the action space and grammar templates in the future.

Table to Sequence Generation
Table to charts recommendation now becomes how to meaningfully fill the placeholders of chart template(s). In other words, how to learn common wisdom to generate action token sequences (tokenby-token from left to right) that follow the grammars of the given template(s). Note that in a single-type task the first chart type token is fixed (the generation starts from the second token), while in the multi-type task it starts from the first chart type token.
A common way to solve sequence generation problem is to learn an estimation function for heuristic beam searching (more details in appendix §A.3). Given a table D and an incomplete chart sequence , we can define the valid actions space of the sequence as A D ( ) according to its corresponding template. Follow the language modelling formulation in [24], we choose ( , ) = ( ∈ T + D | , D) as action-value function to guide the choice of next action token ∈ A D ( ). Here T + D is the set of all target chart sequences (the charts that would be adopted by user for D) and their prefixes. So the optimal action-value function * ( , ) = 1 if ′ = and ′ ∈ T + D , 0 otherwise. is the learning target for ( , ). More details of the corresponding Markov decision process can be found in appendix §A.1. 5 X-fields are the fields mapped to x-axis in line, bar, scatter and area charts, to legend in pie charts, and to curved polar axis in radar charts. Multiple x-fields means concatenation. Y-fields are the fields mapped to y-axis in line, bar, scatter and area charts, to the size of slice in pie charts, and to radial axis in radar charts. Multiple y-fields means multiple value series are shown together.

METHOD
An overview of Table2Charts framework is shown in Figure 3.
To approximate * ( , ) for chart generation, in §3.1 we design an encoder-decoder deep Q-Network (DQN) architecture with copying mechanism. Because the exposure bias is severe for sequence generation with templates, in §3.2 we adopt search sampling technique to train DQN during beam searching. Finally, in order to solve the imbalanced data problem and mutually enhance the performance among different chart types, in §3.3 we propose a mix-and-transfer training paradigm for all the single and multi-type tasks.

Filling Templates: DQN with Copying
As shown in Figure  In Figure 4a is our customized CopyNet architecture. As shown in Figure 4a, the output vector of ( , A D ) consists of two parts: "Generate" (for the command tokens) and "Copy" (for the field tokens). The "Generate" part contains the action value estimations for command tokens C, which comes from a full connected layer with a binary softmax applied on the final decoder state . The "Copy" part has variable length of value estimations for F D , which comes from a binary softmax applied on the product of and a non-linear transformation of the memory = {ℎ 1 , ..., ℎ } (the encoder outputs). We adopt GRU [2] in bidirectional and unidirectional ways for the encoder and decoder RNN parts, respectively. Thus is simply the outputs of a bidirectional GRU for F D = ( 1 , ..., ). A decoder state is updated by from the previous state −1 in GRU cell.
is a linear projection from the concatenated vector of three parts: selective read vector , context vector and the token embedding ( −1 ). Selective read vector 6 choose the field representation from for a field token: Context vector is a linear attention between −1 and :  There are several differences between our model and the original CopyNet architecture [7]: First, unlike the typical NLP scenario where vocabulary size is far greater than the length of the copying source, vocabulary (the command tokens) in Table2Charts is small while the universe of table fields is infinite, and there is no overlapping between generate mode (for vocabulary) and copy mode (for table fields). Second, the input tokens to ( , A D ) first go through the feature transformation network (·) in Figure 4b rather than the usual index to embedding matrix in NLP. Third, the output of ( , A D ) is a vector of [0, 1] values for each action, rather than a probability distribution over all actions in the original CopyNet.
Our design of DQN with copying mechanism is naturally suited for tasks generating structures from table fields. It handles the open vocabulary of table field universe and provides a clear division between table representation (encoder) and template filling (decoder). The encoder part takes in the whole table context and generate field embedding vectors as table representations. The decoder part consumes these vectors for next-token estimation. As good * ( , ) estimator, ( , A D ) is then used by Table2Charts as a heuristic function in beam searching to generate multiple sequences.

Fixing Exposure Bias: Search Sampling
A traditional way to train a next-token estimator is through teacher forcing [21] by only sampling the prefix sequences of user-created charts, and comparing the estimated actions with actual user actions. In other words, in teacher forcing the only samples used to train ( , A D ) network come from a corpus of (table, charts) pairs following the format of * ( , ) (see §2.2) with ∈ T + D . As discussed in [14,24], with only teacher forcing, the outcome model could face exposure bias problem which is common in sequence generation. During teacher forcing, the model is only exposed to the ground truth states (target prefixes); While at inference time it has only access to its own predictions. As a result, during generation it can potentially deviate quite far from the actual sequence to be generated, leading to a biased estimation.
To mitigate exposure bias, we take the search sampling approach in [24] to close the gap between training and inference. Inspired by reinforcement learning, the search sampling process adopts ( , A D ) as the heuristic function to conduct beam searching on each table (details in appendix §A.3). Then the expanded states (including negative samples, not in T + D ) will be stored in a replay memory for periodical update of ( , A D ) itself. This process is very effective after the warm-up of the network with teacher forcing. Without search sampling, the model would perform poorly with the customized beam searching process limited by chart templates.

Mixed Training and Transfer Learning
As discussed in §1, for single and multi-type tasks, there exists separate costs and imbalanced data challenges. As shown in Figure 3, our basic idea to solve the challenges is that the table representations (the encoder part) can be shared by several (one multi-type and six single-type) tasks. This exposes the encoder to diverse and abundant table field samples, and reduce the memory occupation and inference time for deploying models of the tasks.
To train the shared table representation encoder and the taskspecific decoders, as shown in Figure 3, we propose a mix-andtransfer paradigm containing two stages: 1) Mixed Training: Mixing samples from all major chart types together and train one DQN model. Its mixed encoder will be transferred to the next stage, while the whole mixed DQN will be used for the multi-type recommendation task. 2) Transfer Learning: Take the mixed encoder from the previous stage and freeze its parameters. Then, for each single-type task, a new decoder is trained with the fixed shared encoder using only the data of this chart type.
Comparing to Separate Training where a whole DQN is trained for each single-type task (using only the data of that chart type), the mix-and-transfer paradigm in Table2Charts has the following advantages: First, smaller memory occupation and faster inference speed, because now DQN models for all tasks share one same encoder, while separate training still inefficiently holds one for each task. This addresses the separate cost challenge. Second, the encoder is exposed to far more samples than each individual chart type can provide. This not only leads to better learning and generalization of the table representation (see §4.5 on how table context is represented), but also addresses the imbalanced data challenge so that only decoder part (which is small comparing to the larger encoder part) needs tuning for minor chart types.

EXPERIMENTS
In §4.1, we first introduce the (table, charts

Chart Corpora
Two corpora -Excel and Plotly -are used for training and evaluation. The Excel corpus is created by us to train and evaluate models, but some baseline models do not provide training scripts. To make fair comparisons, in §4.2 we also evaluate every model on a public Plotly corpus [8] without training or fine-tuning on it.

Excel Corpus.
Our chart corpus contains 113390 (42.59%) line, 67600 (25.39%) bar, 64934 (24.39%) scatter, 17436 (6.55%) pie, 1990 (0.75%) area and 902 (0.34%) radar charts. They are extracted using OpenXML [11] from Excel spreadsheet files crawled from the public web. Following data preparation steps are also taken: 1) Cell Reference Cleansing. X-fields, y-fields and series names 7 are stored as location references to spreadsheet cells (even in another file), which may lead to inaccessible or invalid tables. Charts with these kinds of cell references are removed from the corpus.
2) Source Table Restoration. In spreadsheets, a chart object has no reference to its source table. (Only direct cell references are saved.) To restore the region and structure of its source table, we implement a table detection algorithm [5] according to its cell references. A chart will be dropped if its references are not covered by any detected table, the series names are not in the table header  region, or the y-field references are not in the table value region. 3) Combo Chart Splitting. In the corpus all combo charts are split into simple charts. Several simple charts (even in different types) can be drawn into one combined plote.g., draw a line chart over a bar chart. (Note that simple charts can still have multiple x-fields and y-fields.) In this paper, we focus on simple charts and leave recommendation of combo charts as future work. 4) Table Deduplication. To avoid the "data leakage" problem that duplicated tables are allocated into both training and testing sets, tables are grouped according to their schemas 8 . Then within each group, same (table, chart) pairs are merged. 5) Down Sampling. After deduplication, the number of tables within each schema is very imbalanced -0.23% schemas cover 20% of these tables. To mitigate this problem, we randomly sample at most 10 unique tables for each unique (schema, chart) pair.
After preparing the data, 266252 charts are remained in 165214 unique tables with 98588 different schemas. The schemas (with their tables and charts) are randomly allocated for training, validation and testing in the ratio of 7:1:2.

Plotly Corpus.
We also adopt the public Plotly community feed corpus [8] and sample 36888 tables with 67617 charts (22644 line charts, 20053 scatter charts, 24204 bar charts and 716 pie charts) from it for testing in §4.2. To extract (table, charts) pairs, following the data processing procedure in VizML [8], we download the full corpus (205GB) and adopt data cleansing code from VizML to remove charts with missing data. Also, similar procedures of combo chart splitting, table deduplication and down sampling are applied to the remaining (table, charts) pairs as in the Excel corpus.

Evaluations on Multi-Type Reco Task
As mentioned in §3.3, for multi-type task, a mixed DQN is first trained using samples of Excel major chart types. Then, this mixedtrained DQN is used as heuristic function for beam searching to generate a ranked list of major-type charts for each table. We compare Table2Charts framework on recall and precision with four baselines: DeepEye [9], Data2Vis [4], DracoLearn [13] and VizML [8].

Baselines.
DeepEye (https://github.com/Thanksyy/DeepEye-APIs) provides two public models (ML and rule-based) without training scripts. Thus, we adopt its models without training on our Excel corpus. Because its ML approach works better than its rule-based one on Excel test set, only its ML results are reported in this paper. Data2Vis (https://github.com/victordibia/data2vis) model was originally trained on 11 tables with 4.3k charts. For fair comparison, we re-train its model using our larger Excel training set (see §4.1.1) which is also used by Table2Charts. DracoLearn (https://github.com/uwdata/draco) provides inference API without training scripts. Again, we evaluate it without training on our Excel chart corpus. It differs from the other methods in that it needs human defined rules as constraints and focuses on searching for specified chart components that least violates them. Thus, in Draco we adopt its default rules and specify it to generate chart type, x-fields and y-fields. VizML (https://github.com/mitmedialab/vizml) formulates the design choices into five classification problems and does not recommend data queries. In other words, VizML lacks ability to recommend charts without field selections. Thus, after re-training the classification models on the Excel training set, we only test VizML performance on design choices. More details of experiment setup can be found in appendix §B.

Large-scale Evaluations on Recall.
Recall of user charting actions in three stages -data queries, design choices and overall chart recommendation -are evaluated on Excel (testing set) and Plotly (whole dataset) corpora for Table2Charts and the four baselines. On data queries, we examine whether the recommended fields match user-selected ones. On design choices, we evaluate whether models can recommend correct chart type, field mapping and bar grouping operation given the user-selected fields. On overall chart recommendation, both data queries and design choices are compared with the ground truth. Recall at top-( = 1, 3; R@1, R@3) numbers are adopted as evaluation metrics. They show how a ranked list of chart recommendations matches the user-created charts. More details of recall calculation can be found in appendix §B.2.
The recall numbers are shown in Table 2. We can see that Ta-ble2Charts outperforms the baseline methods for all three stages on both Excel and Plotly corpora. The overall R@1 and R@3 have reached 43.84% and 61.84% on Excel (33.37% and 48.03% on Plotly), which exceeds those of baselines by large margins (at least doubled on Excel). The recall numbers of data queries stage are higher on Plotly than Excel -This is because in Plotly corpus, each table only contain fields which are used in corresponding chart, and thus lower the difficulty of selecting fields. In addition to the results in Table 2, DracoLearn has R@1 < 1% on all stages -It needs humandefined rules as constraints and focuses on searching for charts that least violates them, which leads to weak generalization.

Human Evaluation on Precision.
To evaluate the quality of recommended charts (precision 9 ), we build a labelling website to collect and compare ratings for the top-1 recommendations from Table2Charts, DeepEye and Data2Vis 10 . 500 unique HTML tables crawled from the public web are selected based on query-frequency 9 Precision numbers cannot be calculated from Excel and Plotly corpora because they only have user-created charts but there can be good charts not created by users. 10 VizML and DracoLearn are not included because VizML cannot recommend complete chart with data queries and DracoLearn has weak generative power. We filtered out the tables which marked as "unsuitable for chart recommendation" 11 , and got the distribution of the ratings based on 330 tables left. As shown in Table 3, Table2Charts has the highest average score, the largest amount of good charts (rating=5, rating≥4, rating≥3), and the smallest amount of bad charts (rating ≤2).
To check statistical significance, we further conduct Wilcoxon signed-rank test [19] which is a non-parametric statistical hypothesis test used to compare two related or matched samples to assess whether their population mean ranks differ (i.e. it is a paired difference test

Evaluations on Single-Type Reco Tasks
After mixed training, as discussed in §3.3, the shared table representation encoder is taken and frozen for the training of six decoders for six single-type tasks. For comparison, the separate training (see §3.3) will generate an independent DQN model for each chart type with the same settings as the transfer learning. Also, the mixed DQN from §4.2 is directly tested on single-type tasks of major types.
The evaluation results are shown in Table 4. The mix-and-transfer paradigm ("Transfer") has higher recall numbers than separate training ("Separate") and mixed-only ("Mixed") DQN for all chart types. Table2Charts could handle single-type tasks well by learning shared table context representations. Considering the model size where the encoder and decoder parts are designed with 1.3M and 0.5M parameters (see appendix §B.1.1), "Separate" have 10.8M parameters while "Transfer" only has 4.3M parameters. In this way, Table2Charts reduces separate costs and improves efficiency for model deployment and inference. As for minor chart types, there are huge gaps between the performance of "Transfer" and "Separate" in Table 4. With mix-andtransfer ("Transfer") paradigm, on average R@1 and R@3 increase 21.84% and 17.87%. The main reasons are that the shared encoder could capture and extract the information of table context and field semantics, and the quantity of minor type charts are sufficient to train decoder which is of smaller size. Thus, as discussed in §3.3, the imbalanced data problem is overcome.

Recommendation Case Studies
As an example, in this section we take Table 1a (with user-created Figure 2a and 2b) to conduct empirical studies. When a user does not know where to start, the multi-type mixed model ( §4.2) recommends common types of charts. Its top-3 recommendations are:

1) [Bar](Total Male Students)(Total Female Students)[SEP](Program)[Stack] 2) [Bar](Total Male Students)(Total Female Students)[SEP](Program)[Cluster] 3) [Bar](Total Program Students Percentage)[SEP](Program)[Cluster]
From the above results, we can see that the mixed model successfully recommends the bar chart in Figure 2a as the top-1 result. Our model can identify 1a 2 -1a 3 as a group and use them to create a bar chart with two y-fields. Result 2) is the clustered form of the bar chart in result 1). Result 3) is also useful. Our model can identify 1a 8 should not be a measure in the 1a 2 -1a 3 group. The multi-type model tend to recommend what are commonly composed, thus may lack diversity (e.g., all the above results are bar charts). So one can also put single-type recommendations into the list. (We leave as a future work how to mix the results from multi-type and single-type models together for a balanced recommendation.) When a user has chosen a specific chart type and needs autocompletion help, it is time to use single-type models ( §4.3) for recommendations. In such scenario, our single-type models can recommend all as top-1 the area chart in Figure 2b and the bar chart in Figure 2a. Furthermore, the single-typed model can also recommend the pie chart in Figure 2d, which is also meaningful to show the percentages but not originally created by the user.

Exploring Table Representations
To understand how the embeddings generated by the shared  Figure 5: Visualization of Shared Table Representations.
t-SNE [16]. In the left part of Figure 5, each point represents a field and the color represents its field type. In the figure, we can see the field type information is learnt by the embedding in a meaningful way. For example, date-time fields and year fields are close. One possible explanation is that they both are often used as x-axes in line charts, and thus have similar representations. As depicted in Figure 5, marked by arrows are the points corresponding to the fields 1a 5 , 1a 6 and 1a 8 (which is "Total Program Students Percentage") from Table 1a. They are close to each other because their record values are all percentages. Note that 1a 5 and 1a 6 are closer compared to 1a 8 because their semantics are similar (contain gender information). Some example neighbor fields (based on cosine distance) of 1a 8 are shown in the right part of Figure 5. Similar to 1a 8 , these fields are also percentages that sum up to 1. Two more clusters are shown as examples in Figure 5. In the squared area, there are many fields about countries. E.g., there are four numerical fields (from four tables) with header names "U.S.", "Japan", "England" and "Scotland" showing annual statistics. In the circular area, many fields take the role of index or ID. E.g., located in this cluster are four fields (from four tables) with header names "VerminID", "Index", "category" and "Course Code" and increasing integers. These integers lose the measure property -they are not for mathematical operations / aggregations. Thus, these fields locate very close to string fields (dark blue points) in Figure 5.

RELATED WORK
Analysis Recommendation: For general data analysis and insight recommendation from tables, current systems are mostly based on collaborative filtering [10], statistical significance [17], heuristic and history matching [6,12], or only target for specific analysis [24]. They rarely consider the semantic meaning of table context or tackle the challenges in recommending multiple types of analysis, which are both taken into account by Table2Charts in an end-to-end approach using large-scale human created corpus.
Chart Recommendation is an important branch in analysis recommendation. Lots of visualization recommendation systems heavily rely on hand-crafted heuristics and rules, such as Voyager [22] and DracoLearn [13]. Data-driven approaches are becoming popular in recent learning-based systems such as Deep-Eye [9], Data2Vis [4] and VizML [8]. In §4.2, we discussed DeepEye, Data2Vis, DracoLearn and VizML as baselines. More of their differences with Table2Charts are summarized in Table 5. DracoLearn and DeepEye both learnt from low quality data and depended on complex rule designs. Data2Vis suffered from naive model of characterlevel seq2seq generation of Vega-lite [15] JSON string. VizML only considered design choices without handling data queries.
Structured Prediction: Filling chart templates and generating action sequences is a structured prediction problem. There are lots of related work such as NL QA and Text2SQL [18]. Table2Charts is inspired by [7,24] to design an encoder-decoder architecture with copying mechanism as a function approximator.
Representation Learning and Pre-training: The word embedding [1] and pre-training paradigm [3] in NLP inspired us to learn pre-trained table representations for multiple tasks [23]. The table field embedding could be useful for more down-stream data analysis tasks including recommendation of other types of analysis.

CONCLUSION
In this paper, we propose the Table2Charts framework to solve single and multi-type chart recommendation tasks considering both data queries and design choices. Through copying from table fields, shared table representations are learnt to enhance performance and efficiency for all chart types. We believe the proposed techniques can be widely used for data analysis tasks on tables in the future.

A TABLE2CHARTS FRAMEWORK DETAILS
In this section, we will dive into the details of the table-to-sequence problem formulation in §2.2 with the corresponding Markov decision process. Also, the input token features to the input embedding network of Figure 4b in §3.1 will be listed for your references. Finally, we will describe the companion heuristic beam searching algorithm to the DQN. Core code of Table2Charts and part of the test data can be found at https://github.com/microsoft/Table2Charts.

A.1 Markov Decision Process (MDP)
As described in §2.2, the MDP for chart generation is based on the one for pivot table that was first defined by [24].
Definition 3 (Chart Generation MDP). For a table D, we adopt the definitions in §2.1 and §2.2 to describe the next-token chart sequence generation MDP: is legal}, which can be viewed as a forest with chart type tokens (see §2.1) as root nodes (initial states) of the trees. S + D contains all the prefixes of all the possible legal chart sequences that follow the chart templates.
• Action space A D : The legal actions for a given state are A D ( ) = { | ∈ S + D , ∀ ∈ A D }. • State transition is deterministic. The transition probability from to ′ by taking action ∈ A D ( ) is: • Reward function D is designed to reflect if a user-created sequence is successfully generated: Here G D is a subset of S + D which contains exactly the charts created by user for D. (As first mentioned in §2.2, T + D is the set of all the prefixes of all the target sequences in G D .) • Discount rate = 1 so that the length of a chart sequence has no impact on its rewards.
According to Bellman optimality equation, one can easily find the optimal action-value function (the expected discounted return for the optimal policy): * ( , ) = D ( , , )+ max ′ ∈A D ( ) * ( , ′ ) In other words, * ( , ) equals to 1 if and only if is a prefix of a target sequence. As described in §2.2, the rest of the problem is to learn a good approximator for * ( , ).

A.2 Token Features for Input Embedding
As shown in Figure 4b, token embedding vector consists of: Semantic Embedding. Semantic embedding features are calculated from the header name of a field (e.g., table header or database attribute string). In this work, we adopt FastText [1] with = 200, 000 and = 50 for semantic embedding. If there are more than 1 words in the field name, the embedding of all words are averaged.
Field Categories. There are five types of categorical features which are adopted in this work.
(1) Token type shows the type of a token in an analysis sequence, which includes {PADDING, SEP, FIELD, GRP, Line, Bar, Scatter, Pie, Area, Radar}.
(2) Segment type shows to which segment a token belongs in an analysis sequence.

A.3 Heuristic Beam Searching
In search sampling training and beam searching inference stages, we adopt and customize a drill-down beam searching algorithm [24]. It takes the following steps to generate chart sequences: (1 (2) For each round, the top-scored partial / incomplete sequences in the frontier will be popped and extended as described below. (a) For each state in the beam, greedily drill down (choose with the highest ( , ) to append) until a complete sequence is generated. The complete sequence is put into the result ranking list with ( , ) as its score. Each nonoptimal state ( ∈ A D ( )) from each expansion (one calculation of ( , A D )) is put into the frontier also with ( , ) as its score. (b) No more rounds and stop searching if the number of expansions exceeds . As mentioned in §2.1, to restrict heuristic beam searching and eliminate some nonsense recommendations, hard constraints are defined in chart templates. For example, the data type of a y-field is forbidden to be string type. During training and inference, these hard constraints are also applied by removing illegal actions from A D ( ) for each expansion in the heuristic beam searching.
This also allows users to specify more constraints on searching. For example, a user could select interested fields from a table and the beam searching can use exactly these fields to suggest charts.  Table 6). To choose embedding model and DQN size, we compare their six possible combinations after the teacher forcing training stage on multi-type task. Results show that compared to FastText, BERT increases R@1 for about 2% (from 13.07% to 14.97%), but doubles the training time. Similarly, R@1 of "small", "medium" and "large" models are 13.07%, 13.30% and 15.37% respectively. The "large" model gains about 2% in recalls while the number of parameters is 2.5× "medium" or 6.1× "small" model. To make a trade-off between performance and training costs, we use the FastText embedding and the "medium" model size in all experiments in §4.
The beam searching hyper-parameters ( , ) are fixed to (4,100). For neural network tuning, we use Adam optimizer with ( _ , 1 , 2 , , ℎ _ ) set to (1 4, 0.9, 0.999, 1 − 8, 0.01). Due to GPU memory limitations, for all experiments the batch size is set to 512. During the back propagation at each step, gradients from each process are averaged.
B.1.2 Training Data2Vis. To train and evaluate Data2Vis, we transform Excel and Plotly data to JSON strings. Following data preparation code from Data2Vis, table input is a JSON dictionary containing key-value pairs of field keys to one randomly sampled row, and chart output is a JSON dictionary in a simplified Vega-lite format. For each (table, chart) pair, two samples are generated by sampling two rows from the table. In total, there are 180383 training samples. Data2Vis uses a character-level seq2seq model with strings as input and output. We set its encoder to be a 1-layer bidirectional LSTM with hidden dimension 256, and decoder to be 2-layer LSTM with hidden dimension 128. These choices make sure the size of the model (~1.94M) is comparable to the that of Table2Charts (~1.8M).
Following the training configurations of Data2Vis, Adam optimizer is used again. According to our Excel corpus, 98% of the source table JSON strings have fewer than 471 characters, while 99% target chart JSON strings have fewer than 130 characters. Therefore, the maximum source length and target length are set to 500 and 130. The vocabulary sizes of source and target are 98 and 42. The model is trained for nearly 30000 steps, with a batch size of 16.
B.1.3 Training VizML. As mentioned in §4.2.1, VizML focuses on design choices and does not provide models for data queries. We re-train VizML models on its Mark Type task (corresponding to chart type) and Is on X-axis or Y-axis task (corresponding to field mapping), and change its Mark Type task from 2, 3, and 6 classification to 4 classification (including line, scatter, bar, pie chart). These models make predictions for one field at a time. So only the labelled fields (selected in user-created Excel charts) are kept for training. Field feature extraction process and model hyper-parameters are identical with the original VizML paper and source code.

B.2 Evaluation Details
B.2.1 Calculating Recall at Top-k. In this section, we elaborate how recall numbers are calculated on data queries, design choices and overall chart recommendation tasks in §4.2 and §4.3.
On data queries, field selection is compared with ground truth. Given a table, if the chosen field set of any top-recommended chart matches that of any user-created chart of the table, this table is considered to be successfully recalled w.r.t. data queries. Thus, recall at top-of data queries is calculated as @ = # (Tables successfully recalled) # (Tables) . On design choices, we consider chart type, field mapping (i.e., map the selected fields onto x-axis and y-axis of a chart), and grouping operation (i.e., whether stacked or clustered, only for bar chart). Given a table and one set of its fields, if any top-recommended chart adopts the field set and matches a user-created chart of the table, then the field set is considered to be successfully recalled w.r.t. design choices. Thus, the recall on design choices is calculated as @ = #(User-created field sets successfully recalled) #(User-created field sets) . On overall evaluation, both data queries and design choices are considered. The equation for overall evaluation is the same as that of data queries, while the "table successfully recalled" here means that any top-recommended chart complete matches any user-created chart of the table.
B.2.2 Comparing with Baselines. For fair comparisons among chart reco systems, more evaluation details (in addition to those described in §4.2.1) need to be properly handled. In DeepEye, bar grouping operations are not considered, and several data transformations (e.g., dimension breakdown and measure binning) are recommended. To make sure the recommended charts from DeepEye matches the definitions in our corpora, during evaluation we ignore the grouping operations in ground truth and drop the charts with breakdown and binning operations. In Data2Vis, searching beam size is set to 15 for data query and overall chart recommendation, and set to 30 for design choices. In DracoLearn, same as DeepEye, its weights for soft constraints are not re-trained using our Excel training set. In VizML, we only evaluate it on design choices. Given a field set, if Mark Type and Is on X-axis or Y-axis predictions of all fields in the set match any of user-created chart, then the field set is considered to be successfully recalled w.r.t. design choices. Only R@1 is calculated for VizML because only one result is available. In Table2Charts, and are the same as in search sampling training (see §B.1.1).