{'fraction_non_alphanumeric': 0.040060347318075305, 'fraction_numerical': 0.009806439187237184, 'mean_word_length': 4.717287575701963, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Capture Salient Historical Information: A Fast and Accurate Non-Autoregressive Model for Multi-turn Spoken Language Understanding

Spoken Language Understanding (SLU), a core component of the task-oriented dialogue system, expects a shorter inference facing the impatience of human users. Existing work increases inference speed by designing non-autoregressive models for single-turn SLU tasks but fails to apply to multi-turn SLU in confronting the dialogue history. The intuitive idea is to concatenate all historical utterances and utilize the non-autoregressive models directly. However, this approach seriously misses the salient historical information and suffers from the uncoordinated-slot problems. To overcome those shortcomings, we propose a novel model for multi-turn SLU named Salient History Attention with Layer-Refined Transformer (SHA-LRT), which composes of an SHA module, a Layer-Refined Mechanism (LRM), and a Slot Label Generation (SLG) task. SHA captures salient historical information for the current dialogue from both historical utterances and results via a well-designed history-attention mechanism. LRM predicts preliminary SLU results from Transformer's middle states and utilizes them to guide the final prediction, and SLG obtains the sequential dependency information for the non-autoregressive encoder. Experiments on public datasets indicate that our model significantly improves multi-turn SLU performance (17.5% on Overall) with accelerating (nearly 15 times) the inference process over the state-of-the-art baseline as well as effective on the single-turn SLU tasks.


INTRODUCTION
Task-oriented dialogue systems have received more attention with the widespread application of intelligent voice assistants, e.g., Apple Siri, Microsoft Cortana, etc. Working as the spoken interface between users and machines, Spoken Language Understanding (SLU) plays a critical role in the task-oriented dialogue system [44,49,55]. A typical SLU task mainly includes two subtasks [43], i.e., Intent Detection (ID) and Slot Filling (SF). Given by an utterance expressed in natural language from the user, ID aims to identify the intent of the user (e.g., Navigate), and SF aims to fill the slot for each token in the utterance (e.g., distance, time). Generally, ID works on sentence-level and is treated as a semantic classification task, while SF is a sequence labeling task focusing on token-level. In recent real application scenarios, people are not satisfied with only making single-turn dialogues with the voice assistant and desire a multi-turn dialogue to achieve their demand. Therefore, the multi-turn dialogue-based SLU task is increasingly crucial [14,25,46]. A simple example of multi-turn SLU is shown in Fig. 1  In multi-turn dialogue, the current utterance is closely related to the dialogue history due to the conversational continuity. Thus, SLU models have to deal with both the dialogue history and the current utterance to enhance the performance, which leads to a large amount of extra computation. Due to the above reason, reducing the inference time of multi-turn SLU tasks is entirely meaningful. However, most existing models process dialogue history inefficiently. Some studies [2,3,7] obtain historical information via some lengthy and complicated architectures and thus consume a lot of inference time. Cheng et al. [9] propose RPFSLU framework which employ an RNN-based portable plugin to obtain historical information for state-of-art single-turn SLU models [16,35]. Unfortunately, RPFSLU still needs to calculate the hidden states of each historical utterance in succession, which is low efficiency. Besides, most of the work relies on the RNN-based framework, which models the entire sequence dependency in an autoregressive way, leading to redundant computation and inevitable high latency.
An intuitive idea for accelerating the predicting process is to utilize the self-attention (i.e., Transformer) [45] instead of the RNN as the basic framework in a non-autoregressive way. However, utilizing the Transformer-based model in multi-turn SLU tasks will face the following problems: Missing salient information: To deal with the multi-turn dialogue, many existing Transformerbased models [53,56] simply concatenate all historical utterances and the current utterance together. However, those approaches cause severe problems with increasing dialogue turns. First, simply concatenating sentences creates redundant computations. More seriously, concatenating all utterances together will form a very long sequence. The long sequence may cause the model difficulty to focus on the salient information due to the self-attention mechanism, resulting in a low SLU performance. Therefore, how to design an efficient module to process dialogue history without missing salient information is a major challenge.
Uncoordinated slot problem: Directly utilizing the Transformer framework will reduce the prediction performance of the SF task in each single turn [48]. The reason is that SF is a sequence labeling task whose results utilize the "Inside-Outside-Beginning (IOB)" tagging format. Therefore, SF heavily depends on the strongness of the sequential dependency information among each slot chunk (proved by Ren et al. [40]). However, as shown in Fig. 2 the slot labels are predicted independently and simultaneously in non-autoregressive methods, which reduces the strongness of the sequential dependency and causes the uncoordinated slot problem. Fig. 3 shows an example of the uncoordinated slot problem, where B-city should be followed by I-city, but I-time is predicted by mistake.  Although Wu et al. [48] tries to handle the uncoordinated slot problem via a two-pass mechanism called SlotRefine, this method still has the following weakness: First, SlotRefine supposes all the uncoordinated slot problems are caused by wrong 'I-tags' and only utilize 'B-tags' to correct the 'I-tags' in its second pass period. However, as shown in Fig. 3, when predicting, an incorrect 'B-tag' follows by a correct 'I-tag' also happens. In this case, SlotRefine cannot utilize 'I-tags' to correct the 'B-tags'. Second, the two-pass mechanism needs to characterize text and predict two times in both the training and testing process, which inevitably leads to lower efficiency. Therefore, how to maintain the non-autoregressive efficiency when inferring while avoiding the uncoordinated problem in SLU tasks is a significant challenge.
To solve the above challenges, in this paper, we propose a novel non-autoregressive model for multi-turn SLU called Salient History Attention with Layer-Refined Transformer (SHA-LRT). Our SHA-LRT contains a Salient History Attention (SHA) module, a Transformer encoder with Layer-Refined Mechanism (LRM), and a Slot Label Generation (SLG) auxiliary task. Specifically, the SHA is proposed to capture salient historical information for the first challenge, where a history-attention approach is designed based on the self-attention mechanism. The query (Q), key (K), and value (V) of our history-attention is generated by the current utterance, historical utterances, and historical results, respectively. Since the predicted results of SLU in dialogue history contain essential and specific semantics (proved byCheng et al. [9]), our SHA module could obtain the hidden states of dialogue history with salient historical information. Subsequently, we utilize the hidden states of dialogue history to guide the prediction of the current-turn utterance. Utilizing calculated hidden states but not all utterances for the current utterance prediction avoids redundant computation. Thus our SHA module is quite efficient.
To further solve the uncoordinated problem for the second challenge, we propose LRM and SLG, respectively. Specifically, LRM modifies the hidden states between Transformer layers according to the intermediate predicted ID and SF results to help the final prediction. In practice, LRM predicts the preliminary results of ID and SF by the hidden states of the former Transformer layer, merging the result embeddings with the hidden states and inputting them into the next Transformer layer. We have further proved in experiments that LRM only needs to be employed once and does not need to be used between every Transformer layer, so the cost is much smaller than running the entire model twice. SLG is an auxiliary task that predicts the next SF label according to the utterance sequence and the generated SF labels, which is a sequence-to-sequence (seq2seq) task like machine translation. We jointly train the SLG and the original sequence labeling-based SLU tasks by multitask learning and share t encoder. By SLG, the shared encoder learns more sequential dependency information from the decoder through the cross attention mechanism in the Transformer and further improves sequence labeling-based SLU tasks. Notable, SLG is only employed in the training process and consumes no extra inference time. Thus our model is still a fast non-autoregressive model.
We conduct extensive experiments to evaluate the effect of our complete SHA-LRT. Specifically, we first evaluate our model on the multi-turn SLU dataset KVRET. The experiments results indicate that our model significantly improves multi-turn SLU performance (16.1% on SF F1 score, 17.5% on Overall) while substantially speeding up (nearly 15 times) the inference process over the state-ofthe-art (SOTA) baselines. We further evaluate our model on two single-turn SLU datasets (without the SHA module). The results show that our model still obtains the best performance and the fastest speed compared with the current SOTA single-turn SLU models, which also manifests that SHA is a portable module and our whole model works robustly. We also make the ablation study and verify the effeteness of each module of our model. Finally, we combine our model with the pre-trained language models, and the results imply that the pre-trained language models could further boost the performance of our SHA-LRT.
The main contributions of this paper are presented as follows: (1) We propose the SHA, which takes advantage of the dialogue history efficiently without losing the salient information and evidently enhances the SLU performance for the current utterance. (2) We design the SLG as auxiliary multitasking of SLU, which increases the sequential dependency information of the model to modify the uncoordinated problem while consuming no extra inference time. (3) We propose the LRM, which improves the overall performance of SLU tasks by the interaction of predicted results between Transformer layers with a negligible time cost. (4) Experimental results on three public datasets show our model is superior to both existing SOTA autoregressive and non-autoregressive models in terms of speed and performance, indicating that our model has great potential for real-world application.

RELATED WORK
In this section, we introduce the related work from two aspects.

Single-turn SLU
In SLU, intent detection is usually seen as a semantic classification problem to predict the intent label, and slot filling is mainly regarded as a sequence labeling task. With the developments in deep neural networks, many methods [22,32,33,52] have been proposed to solve these two tasks.
Traditionally, pipeline approaches are utilized to manage the two mentioned tasks separately. These kinds of methods typically suffer from error propagation due to their independent models and prove less effective than the joint models. Motivated by this problem, some work deeply researches the relationship between these two tasks and finds that ID and SF are closely related. Goo et al. [15] propose a slot-gated model that learns the relationship between intent and slot by the gate mechanism. Inspired by [15], some bi-directional networks are proposed [16,28], which dig into the correlation between ID and SF deeper and model the relationship between them more explicitly. In the above work, the interrelated connections between ID and SF are established. Besides, Zhang et al. [54] utilize a hierarchical capsule neural network structure encapsulating the hierarchical relationship among utterance, slot, and intent. Qin et al. [35] propose a Stack-Propagation framework that uses result information of ID to guide the SF task but ignores the impact of SF results on the ID task. Although Stack-Propagation reaches the SOTA performance on two SLU datasets, it surfers a long inference latency caused by the heavy and complex framework. Qin et al. [37] then concern about the impact of SF results in their model but still suffering the long inference latency. Inspired by [50], Cheng et al. [9] propose a portable framework RPFSLU, which contains a two-round predicting period. RPFSLU utilizes the semantic information of the first round prediction results to guide the second round prediction by a represent learning process. However, although RPFSLU is portable, it is not a lightweight model due to its two-round prediction process. The above methods mostly use the autoregressive models (e.g., LSTM and GRU [10]) that suffer inevitable long inference latency. Inspired by the good performance of Transformer in other tasks [23,24,27], Wu et al. [48] then propose a non-autoregressive joint model named SlotRefine for SLU, which speeds up the predicting process and encounters the uncoordinated-slot problem. SlotRefine tries to handle this problem by a two-pass refine mechanism and get some effect. However, it still suffers uncoordinated slot problems caused by the wrong 'I-tag' and is not efficient enough during inference due to its two-pass mechanism that needs to run the whole model twice.
Meanwhile, some work [17,34] tries to enhance the performance via multi-task learning. These multi-task learning methods link the two tasks implicitly via applying a joint loss function. Bai et al. [2] propose an impressive multi-task learning approach for multi-turn SLU by consolidating context memory with a dialogue logistic inference task called DLI. DLI needs no extra labeled data and only needs to be carried out during training, which inspired us to design SLG.
Compared with previous work, our SHA-LRT model obtains the sequential dependency information via auxiliary multitasking and utilizes the interaction of the prediction results between ID and SF via the Transformer's middle states. Thus, our model has high predicting accuracy and fast inference speed, which indicates that our model has great potential for industrial application.

Multi-turn SLU
In multi-turn dialogue SLU, the model is provided with a group of extra context-sensitive dialog statements [31]. To deal with the contextual information, many models incorporating more context from dialogue history [7] or semantic context from the frame [4] are proposed and outperformed the methods without context information. Using dialogue context is shown to gain improvement for an end-to-end dialogue [5]. Besides, Bapna et al. [3] proposed an RNN based model SDEN to improve dialogue context modeling and [2] refined SDEN with SDEN+, which proposed a dialogue logistic inference task for contextual SLU without needing extra labeled data. Then context-aware GCN [36] is proposed to automatically aggregate the contextual information, which frees the SLU model from the manually designed heuristic aggregation function. All these models only concern context utterance but ignore the historical results, which contain abundant contextual information.
Cheng et al. [9] then propose RPFSLU, which attain contextual information from both historical utterance and historical predicted results. Meanwhile, RPFSLU is a portable framework that can easily let nearly all RNN-based single-turn SLU models also be available in multi-turn SLU tasks. However, with the widespread of Transformer [45], Transformer-based models outperform RNNbased models on both accuracy and latency. Due to its basic architecture, RPFSLU is not compatible with Transformer-based SLU models. Compared with RPFSLU, our model is specially designed for Transformer-based SLU models and performs better on both speed and accuracy.

METHOD
In this section, we first introduce the problem formulation of multi-turn SLU and the overview of our SHA-LRT. Then we describe our SHA module in detail. Subsequently, we describe the detail of our basic model and the Layer-Refined Mechanism. Finally, we introduce our Slot Label Generation task.

Problem Formulation
In this section, we introduce the problem formulation for the multi-turn SLU task.
Given X and DH as input, our tasks are composed of Intent Detection (ID) and Slot Filling (SF). Specifically, ID is a semantic classification task to predict the intent label for the whole utterance, while SF is a sequence labeling task to give each token in the sequence a slot label. In our model, the intent label and all slot labels are predicted simultaneously.

Overview
In this section, we describe the overview of SHA-LRT and introduce the relationship between each module in our whole model.  The general framework of our model is shown in Fig. 4, where is the number of Transformer Encoder layers, and indicates we employ LRM in the interval of the -th and the ( + 1)-th Encoder layer. Generally, our SHA-LRT consists of three parts, i.e., a Salient History Attention (SHA) module, a current-turn SLU module, and a Slot Label Generation (SLG) auxiliary task.
Specifically, we first employ the SHA module to deal with the dialogue history. The input of SHA is composed of three parts: the current utterance, historical utterances, and previously predicted results. In SHA, we first employ self-attention to address the current utterance. Then, we utilize the history-utterance-attention and history-result-attention, respectively, to obtain the contextual representation of the current utterance with historical information. Note that the history-utteranceattention and history-result-attention merge the semantic information contained in the historical utterances and results (previously predicted SLU results) based on the influence of each historical utterance on the current utterance, respectively. Thus, the obtained representation of the current utterance is with salient historical information, and we use it as the input of our current-turn SLU module. We will introduce the detail of SHA in section 3.3.
Then, we employ the current-turn SLU module to predict the SLU results for the current utterance. Our current-turn SLU module processes the output of SHA with a layers Transformer Encoder to calculate ID and SF results. Particularly, to solve the uncoordinated slot problem, we utilize Layer-Refined Mechanism (LRM) between two specific ( and + 1) Transformer layers. In LRM, we predict the preliminary results of ID and SF by the hidden states of the -th Transformer layer, merging the result embeddings with the hidden states and inputting them into the ( + 1)-th Transformer layer. We will describe the detail of the Transformer Encoder and LRM in section 3.4.
Finally, we introduce an auxiliary task, i.e., Slot Label Generation (SLG) to further solve the uncoordinated slot problem. The SLG extends the current-turn SLU module with a Transformer decoder (called SLG module) and forms a seq2seq task in an autoregressive way. Specifically, the SLG module aims to generate each of the current-turn slot labels based on previously generated slot labels to obtain sequential dependency information for our current-turn SLU module by the joint loss function. We will describe the detail of SLG in section 3.5.
Notably, both the SHA and SLU modules work in a non-autoregressive way. Furthermore, the SLG task is only carried out during the training process, which costs no extra time for inference. Therefore, our complete model is an effective non-autoregressive SLU model.

Salient History Attention
In this section, we proposed the Salient History Attention (SHA) module to address the salient information of dialogue history.
History-Attention: As we mentioned, dialogue history needs to be significantly concerned in multi-turn SLU tasks. Actually, the historical utterance with higher semantic similarity to the current dialogue should have a more significant influence [25]. Moreover, Cheng et al. [9] proved that the predicted results in dialogue history also contain important semantics and are even more specific than utterances in SLU tasks. Therefore, both historical utterances and historical results should be considered to obtain salient information from dialogue history.
Motivated by the above requirements, we design a non-autoregressive architecture called SHA based on the Transformer Encoder. Specifically, we insert two history-attention sub-layers, i.e., the history-utterance-attention sub-layer and the history-result-attention sub-layer between the self-attention sub-layer and the feed-forward network of the Transformer encoder. The historyutterance-attention and history-result-attention are utilized to obtain semantic information contained in the historical utterances and results, respectively. Besides, we employ residual connections [18] around each of the sub-layers, followed by layer normalization [1]. Similar to Transformer, SHA is an N-layer architecture, and each layer has the same structure. A brief view of SHA architecture is shown in Fig. 5.
Before introducing our SHA module, we first briefly introduce the attention mechanism used in the Transformer. The attention function takes a query matrix Q ∈ R , a key matrix K ∈ R and a value matrix V ∈ R as input, and is defined as where FC is the linear project operation. In our SHA, we first employ the self-attention to encode the current utterance sequence bỹ where H = {h , h 1 , ..., h ∈ R } denotes the hidden states of the current utterance, is the input and output dimension of the SHA layer, and Norm() denotes the layer normalization. In this paper, we set = . Particularly, in the -th (2 ≤ ≤ ) SHA layer, we replace e with the output of the ( − 1)-th SHA layer as the input of self-attention.
To merge the semantic information of historical utterances into the current utterance based on the semantic similarity, we design a history-utterance-attention over current utterance and historical utterances byH To further incorporate the salient information contained in historical results, we inject historical results information into the current utterance according to the semantic similarity between their corresponding historical utterance and the current utterance, and design the history-result-attention byH We then employ a feed-forward network (FFN) to prevent the vanishing or exploding of gradients by where FFN is consists of two linear transformations with a ReLU activation function in between. The SHA module finally returnsĤ = {Ĥ ,Ĥ 1 , ...,Ĥ ∈ R } from the output of the -th SHA layer, which denotes the contextual representation of the current utterance. Subsequently, We mergeĤ with the embedding sequence of the current utterance e, and obtain the input of our current-utterance SLU moduleê SHA-Parallel architecture: To accelerate the predicting process further, we can also operate two history-attention functions in parallel and merge their output. This architecture is called SHA-Parallel (SHA-P), as shown in Fig. 6.
In SHA-P, the Q matrix of both two History-attention sub-layers are generated from H . We calculate the contextual representation of the current utterance from both historical utterances and results in SHA-P by and merge H and H by where H is the merged contextual representation. The contextual representation of the current utterance H in SHA-P is calculated bỹ Generally, SHA-P infers faster than SHA, while SHA has a better performance. We will further analyze the advantage and weaknesses of these two architectures in section 4.
By our SHA module,ê contains current utterance information together with the salient information of dialogue history. Particularly, SHA is a portable module. When solving single-turn SLU tasks, we can directly input e into our current-utterance SLU module.

Transformer Encoder with Layer-Refined Mechanism
In this section, we will introduce the detail of our current-turn SLU module.
Basic model: Following previous non-autoregressive models [8,48], we employ an M-layer Transformer encoder to predict SLU results for current utterance, which is a non-autoregressive structure. To better capture the sequential information, we utilize the relative position representations [42] instead of the absolute position embedding used in the original Transformer.
Specifically, givenê = {ê ,ê 1 , ...,ê } as input, we first employ a position relative self-attention to address the dependencies of each tokens by where and W ∈ R × are fully connected matrices, = = is equal to , a and a indicates the relative position representations of token and . Following Shaw et al. [42], we obtain a and a by where is the farthest relative position, w = {w − , ..., w ∈ R }, w = {w − , ..., w ∈ R } are learnable vectors, which are initialized randomly and modified with the training process.
Notably, the -th Transformer Encoder layer, we replaceê with the output of the ( − 1)-th (2 ≤ ≤ ) Transformer Encoder layer as the input of the self-attention to calculate Z.
Subsequently, we employ a feed forward network (FFN) to attain the output of the -th Trans- The Transformer Encoder finally outputs H = {h , h 1 , ..., h ∈ R } from the -th Transformer Encoder layer (H is actually H ). Then, we utilize H to calculate the prediction of ID and SF byŷ whereŷ ∈ R andŷ = {ŷ 1 , ...,ŷ ∈ R } denote the predicted ID and SF result distributions of the current utterance, and are the the categories of the intent label and slot labels, W ∈ R × and W ∈ R ×2 are fully connected matrices, b ∈ R and b ∈ R are bias vectors, and ⊕ denotes the concatenation operation.
The objective of our basic model can be formulated as: And the joint loss function of SLU is defined as: Note that and are the ground truth labels (scalar), whileŷ andŷ are the predicted result distributions whose dimensions are equal to the categories of their tasks. Besides, during inference, we obtain the label of ID and SF by where and resS = { 1 , ..., } donate the predicted ID label and SF labels of the current utterance. We will record them and further utilize them in our SHA module as part of the historical results in the following turn (i.e., + 1 turn) of dialogue.
Layer-Refined Mechanism (LRM): Previous work [9,35] has widely proved that ID and SF are closely related. Taking advantage of the correlation between these two tasks, especially utilizing one task's results in the other, can effectively enhance the overall performance. However, since the non-autoregressive approach predicts the results of ID and SF simultaneously, we can not directly employ these results in a one-pass prediction process like Stack-Propagation [35] do. Although we can utilize a two-pass mechanism [48] to generate the first-pass results and guide the second-pass prediction via them, it is a trade-off between autoregression and non-autoregression, which costs much time.
Considering that the Transformer contains a multi-layer architecture, and each encoder layer has the same structure. We design the LRM, which works between two Transformer layers and utilizes the middle states of the Transformer to guide the final prediction.
Since the SF returns a sequence of results, we further calculate the weighted average of their correponding embedding vectors via the attention mechanism by where is the weight of e obtained by e 0 ∈ R is the utterance-level SF result embedding vector, and V ∈ R is a trainable vector. By the above calculation, we obtain result embedding vectors e , e and e 0 , which contain semantic information from the preliminary results of ID and SF. Subsequently, we merge these result embeddings with the former output and obtain a new hidden states sequence We use H ′ as input of the ( + 1)-th Transformer layer. LRM can incorporate the bidirectional semantic information from one task to the other by propagating the combination of former output and preliminary results so that ID and SF become more accurate. The complete Markov chain process can be simplified as follow: ( , | X, DH) = (ỹ | X, DH) · (ỹ | X, DH,ỹ ) · (y | X, DH,ỹ ,ỹ ) · (y | X, DH,ỹ ,ỹ ,ŷ ) LRM is a portable plugin for the Transformer, which can be utilized between two Transformer encoder layer intervals. Actually, using LRM once in the whole Transformer structure is enough because it already considers the interaction between ID and SF. On the contrary, overusing LRM will make the Transformer layers lose their own feature and negatively affect the final performance because we use the SLU classifier in LRM. Also, LRM is not suitable for the following autoregressive SLG task because it simultaneously uses all SLU prediction labels.
Besides, the network structure of LRM is mainly composed of a fully connected layer as the SLU classifier and an embedding layer for result embedding. Therefore, our LRM is very lightweight and consumes very little inference time.

Slot Label Generation
In this section, we introduce the Slot Label Generation (SLG) task in detail.
As we aforementioned, the SF task heavily depends on the strongness of the sequential dependency information among each slot chunk. Although we have tried to obtain dependency information via self-attention with relative position representations [42], it is not enough for the sequence labeling task like SF.
To obtain more sequential dependency information for the model, we design the SLG as auxiliary multitasking, jointly training the original sequence labeling-based SLU tasks and sharing their encoders.
Specifically, given the token sequence X = { , 1 , ..., } and previous slot labels { 1 , ..., −1 }, SLG generates the next slot label y by an autoregressive way. The objective of our SLG can be formulated as: where = { 1 , ..., } are the ground truth labels of SLG. In practice, we share the encoder of SLU (our current-turn SLU module) to SLG task and extend the encoder with a Transformer decoder to construct the model architecture of SLG. The complete framework of SLG works as a seq2seq model like machine translation. Since the Transformer-based seq2seq model has been widely used, we do not describe it in more detail. Notably, following existing work, we utilize the teacher forcing strategy during the training process.
Moreover, to enhance the prediction consistency between the SLU and SLG tasks, we further design a consistency loss function based on the cross-entropy as where is the cross-entropy function, is a hyper-parameter,ŷ = {ŷ 1 , ...,ŷ ∈ R } is the predicted distribution of SLG , andŷ is the predicted distribution of SF obtained by Eq.21.
By SLG, the shared encoder can learn more sequential dependency information from the decoder through the cross attention mechanism in Transformer and further improve sequence labeling-based SLU tasks.
The loss function of overall multi-task learning is defined as: where is a hyper-parameter.

EXPERIMENTS
In this section, we demonstrate the effectiveness of SHA-LRT. We first introduce datasets, the necessary hyper-parameters, and the baselines used in our experiments. Then, we compare the performance of our model with baselines and analyze the experiment results on multi-turn and single-turn SLU tasks, respectively. Subsequently, we analyze the error caused by the uncoordinated problem and carry out an ablation study to verify the effeteness of SLG, LRM, and SHA. Finally, we combine our model with the pre-trained model and analyze the effect.

Experimental Settings and Baselines
Dataset: To evaluate the efficiency of our proposed model, we conduct experiments on a multiturn dataset: KVRET [13]. Evaluation Metrics: Following previous work, we utilize accuracy for ID and the F1 score for SF as the performance evaluation. Besides, we utilize overall accuracy to indicate the proportion of utterance in the corpus whose slots and intent are both correctly predicted. Usually, a higher intent accuracy and F1 score also lead to higher overall accuracy. However, this does not always happen, e.g., when the prediction contains more mistakes, but most are from the same utterances.
Setup: We use AdamW [30] to optimize the parameters in our model and adopted the suggested learning rate of 5e-5. The batch size is set to 32 according to the size of training data.
When tuning hyperparameters, we repeat the model 5 times and select the parameters with the best average performance on the validation set as the optimal.
We first select the hyper-parameters used in our basic model. Specifically, we select in the range of {128, 256, 512, 768}, and layers of SHA in {2, 3, 6} by the grid search. We finally choose as 768, and as 3. The size of the inner-layer in the feed-forward network is set as four times of . For other hyper-parameters , following [45], we set both encoder and decoder layers as 6, the number of attention heads as 8, and the dropout ratio as 0.3. We utilize LRM once between the second and the third Transformer layer.
Then, we choose the hyper-parameters used in Eq.32 and used in Eq.33. Specifically, we first fix as 1 and select in the range of (0,0.5] with the step 0.05. Subsequently, with the chosen , we select in range of (0,1] with the step 0.25. We finally get the optimal when is 0.35 and is 0.75. We will introduce the influence of and in our ablation study.
Baselines: To evaluate the performance of our model, we select the following models as benchmark.
• RPFSLU [9]: An portable framework that utilizes a GRU-based module to obtain historical information from historical utterances and results to help the current turn SLU task. • RPFSLU+BiModel [47]: Utilizing BiModel as the basic model of RPFSLU, which achieves the best performance on ID and overall accuracy. • RPFSLU+Stack-Propagation [35]: Utilizing Stack-Propagation as the basic model of RPFSLU, which has the best performance on the SF task.
Non-Autoregressive Baselines: • Basic model: The encoder of Transformer framework with relative position representations [45], which ignores historical information. • CAT-All: Dealing with dialogue history by concatenating all historical utterances with the current utterance together. • Basic model+CAT-All: Dealing with dialogue history by CAT-All, using the Basic model.

Our models:
• LRT: Our model without the SHA module, i.e., our current-turn SLU module with the SLG module. • LRT+CAT-All: Dealing with dialogue history by CAT-All, using the LRT model. For Joint MemNet, SDEN, and SDEN + , we adopt the reported results from [9]. For RPFSLU + BiModel and RPFSLU + Stack-Propagation, we re-implement the model. Note that, for all reimplemented models and our model, we repeat the experiment 5 times and report the average as the final results.

Result and Analysis
In this section, we show the results of our experiments and do some analysis. overall accuracy when utilizing SHA. Even with SHA-P, the enhancement is still 15.9% on SF and 17.2% on overall. This indicates the evident effectiveness of our SHA-LRT on multi-turn SLU tasks. We attribute this enhancement to accurately catching the salient historical information by our SHA module. As a comparison, simply concatenating all historical utterances perform unsatisfyingly. As shown in Table 2, the Basic model + CAT-All and LRT + CAT-All performs even worse than the Basic model and LRT, respectively. We attribute this phenomenon to the loss of salient information, which directly indicates the importance of capturing salient information when dealing with dialogue history.
Moreover, the performance of our Basic model and only utilizing our current-turn SLU module (LRT) are unsatisfactory due to lacking historical information, which directly indicates the importance of considering dialogue history in multi-turn SLU tasks. With SHA, the performance of these models is enhanced by a large margin (nearly 18% on SF and Overall). We will further analyze the effect of SHA in more detail in section 4.5.

Model
KVRET  Table 3. Latency of SLU models on multi-turn SLU tasks. "Latency" is the average inference time without minibatching. "Speedup" is compared against the existing SOTA model RPFSLU +Stack-Propagation.
Speed Up: The inference time of SLU models is shown in Table 3. All the models in this experiment are conducted with a single TITAN Xp GPU.
As shown in Table 3, our proposed model achieves significant speedup against the autoregressive SOTA model RPFSLU+BiModel (11.31×) and RPFSLU+Stack-Propagation (14.58×). The main reason is that both the module for handling dialogue history and the module for processing single-turn SLU in our SHA-LRT work in a non-autoregressive way. Specifically, our SHA module deals with every turn simultaneously for dialogue history, while RPFSLU processes them turn by turn. Moreover, autoregressive models calculate each token in the current utterance serially while our model predicts SLU results for each token in parallel. Thus SHA works efficiently and much faster than RPFSLU.
As shown in Table 3, the latency cost of SHA-LRT is only 4.87ms longer than that of the LRT, which indicates that our SHA module spends very little time. By the parallel structure, SHA-P works even 2.7% faster than SHA. Considering the significant enhancement on SLU performance brought by SHA, the trade-off on latency is acceptable. We also find that utilizing LRM consumes only 5.7% extra inference time than without LRM, indicating that LRM is a lightweight approach with only a little time cost.

Single-turn Performance
As introduced in section 3, our SHA module is portable, which is also appropriate for single-turn SLU tasks. Therefore, we further conduct experiments to evaluate the performance of our model on single-turn SLU tasks. In this experiment, we use Adam [21] as the optimizer and adopted the suggested learning rate of 0.001. Besides, we set as 128. All other hyperparameters and the evaluation metrics are the same as we conduct in multi-turn SLU tasks. As the single-turn task contains no dialogue history, we only utilize the LRT part in our model, i.e., the current-turn SLU module with the SLG module. The embedding sequence of the current sequence is directly input to our current-turn SLU module.

Dataset:
We conduct experiments on two public datasets, i.e., ATIS (Airline Travel Information Systems [19]) and SNIPS (collected by Snips personal voice assistant [11]). Compared with ATIS, the SNIPS dataset is more complex due to its large vocabulary size, cross-domain intents, and more out-of-vocabulary words. The statistics of ATIS and SNIPS are shown in Table 4  Baselines: To evaluate the performance of our model on single-turn SLU tasks, we compare our model with the existing baselines.
• Attention-BiRNN [26]: A LSTM [20] based encoder-decoder model with an intent attention mechanism. • Slot-gated [15]: A LSTM based joint model together with a slot-gated mechanism as a special gate function. • Bi-model [47]: An RNN based encoder-decoder model considering the intent and slot filling cross-impact to each other. • SF-ID [16]: A LSTM based joint model with cross-impact calculating between two tasks.
• Stack-Propagation [35]: A LSTM based joint model with stack-propagation framework and token-level ID, which is the state-of-the-art of the autoregressive Single-Intent SLU model. • AG-IF [39]: A LSTM-based joint model with an adaptive interaction network, which is designed for Multi-Intent SLU tasks, and is also the state-of-the-art model on ATIS and SNIPS datasets. Non-autoregressive Baselines: • Basic model: The encoder of Transformer framework with relative position representations.
• GL-GIN [38]: A LSTM-based joint model with a Global-Locally Graph Interaction Network, which is designed for Multi-Intent SLU task. • SlotRefine [48]: A Transformer based non-autoregressive model with a two-pass refine mechanism. For Joint Seq, Attention BiRNN, Slot-gated, BiModel, SF-ID, and Stack-Propagation, we adopt the reported results from Qin et al. [35]. For SlotRefine, since the benchmark in their original paper is calculated nonstandardly according to their open-source code, we re-implemented the model (all hyper-parameters strictly identical as Wu et al. [48]) and obtained the results. Since AG-IF and GL-GIN are specially designed for Multi-Intent SLU tasks, we re-implemented the model with the suggested hyper-parameters in their original paper. Note that, for the re-implemented models, basic model, and our model, we repeat the experiment 5 times and report the average as the final results. SLU Performance: The experiment results on ATIS and SNIPS datasets are shown in Table 5. The results show that our model significantly outperforms all the baselines and achieves the best performance in all three metrics. Compared with the prior non-autoregressive model SlotRefine, our model enhances the performance by 1.1% (ID), 0.1% (SF), and 0.3% (Overall) on ATIS and 1.0% (ID), 1.3% (SF), and 4.0% (Overall) on SNIPS. Compared with the SOTA baseline AG-IF, LRT also achieve improvement by 1.1% (ID), 0.2% (SF), and 0.2% (Overall) on ATIS and 0.4% (ID), 0.5% (SF), and 1.1% (Overall) on SNIPS. This indicates the effectiveness of our model.
Notably, without SLG and LRM, our basic model performs worse than both SlotRefine and AG-IF, but LRT outperforms both of them with SLG and LRM. We attribute this enhancement to the fact that our SLG task effectively obtains the sequential dependency information, and LRM directly takes the explicit result information into consideration, which grasps the relationship between the intent and slots. We will conduct experiments for the ablation study in section 4.5 to further verify this idea.
Speed Up: We also conduct experiments to test the latency of SLU models on single-turn SLU tasks in the same environment as we do on multi-turn SLU tasks. From Table 6 Table 6. Latency of SLU models on single-turn SLU tasks. "Latency" is the average inference time without minibatching. "Speedup" is compared against the existing SOTA model AG-IF [39].
More importantly, compared with the existing non-autoregressive model, our model also achieves speedup against SlotRefine (×2.47 on ATIS; ×2.93 on SNIPS) and Gl-GIN (×1.44 on ATIS; ×1.31 on SNIPS). The reason for this phenomenon is that SlotRefine needs to run the whole model twice, and GL-GIN contains a LSTM-based Encoder in its structure. As a comparison, our model is lightweight and obtains the final SLU results with a one-period prediction.
From Table 6, we can also find that utilizing LRM consumes only 3% extra inference time compared to without LRM. This indicates that LRM is a lightweight approach, which generates negligible time cost.
Above all, our proposed model significantly improves performance while substantially accelerate the inference speed against the existing autoregressive models on both single-turn SLU tasks and multi-turn SLU task.

Error Analysis
In this section, we will analysis the error caused by uncoordinated slots.

Error Analysis on Multi-turn SLU Tasks.
We first analyze this kind of error in multi-turn SLU tasks. To analyze the error in SF tasks in detail, we show the statistics of slot error (i.e., incorrect slots) on the validation set of KVRET. Specifically, we define the errors caused by the uncoordinated slot problem as "Unc. error". The uncoordinated slots includes two cases, i.e., correct 'B-tag' followed wrong 'I-tag' and wrong 'B-tag' following correct 'I-tag'. We define the first case as "BI error" and the second case as "IB error. ", respectively.  Table 7. The statistics of slot error on the validation set of KVRET after training 100 epochs.
The statistics of slot error on the validation set of KVRET is shown in Table 7. From the Table, we can find that our model reduces the uncoordinated slot, including both BI and IB errors. Compared with the Basic model, our model reduces 49 (84.5%) uncoordinated slots in total, where 76.9% (30/39) BI error and 100% (19/19) IB error are solved by our model, respectively. This phenomenon indicates that our model effectively solves the uncoordinated slot problem in multi-turn SLU tasks.
As the Basic model lacks historical information, comparing the Basic model with our SHA-LRT is somehow unfair. Thus, we further count the uncoordinated slots for LRT to make a more direct comparison. Compared with the Basic model, our LRT solves 56.4% (22/39) BI error and 42.1% (8/19) IB error and reduces 30 (51.7%) uncoordinated slots. Meanwhile, we also combine the Basic model with our SHA module and compare it with our SHA-LRT. Even both have the SHA module, our SHA-LRT still reduces about 50% more Unc. errors than the Basic model.
Moreover, as shown in Table 7, concatenating all historical utterances even exacerbates the uncoordinated slot problem. The reason is that the overly long utterance sequence makes it more challenging to obtain the sequence dependency among tokens.
Note that, for the Basic model, the uncoordinated problem seems not very serious, as Unc. errors account for only 12% of all slot errors. However, this is mainly because the uncoordinated slot problem requires at least one B-tag or I-tag to be predicted correctly. When the model is not effective enough, it predicts a large number of slot labels which both B-tag and I-tag of it are incorrect. We do not count this kind of slot as the uncoordinated slot, even though the B-tag and I-tag are different, which is 'uncoordinated' in a sense.

Error Analysis on Single-turn SLU Tasks.
To avoid the influence of dialogue history and make a more accurate analysis of the uncoordinated slot problem, we further analyze this problem in single-turn SLU tasks. We first visualize the number decrease of uncoordinated slots in the training process. As shown in Fig 8, the number of uncoordinated slots drops slow and inefficient for the basic model. For our LRT, the number of uncoordinated slots drops significantly faster. Ten epochs training of our model is already better than 100 epochs training of the basic model on this problem. Compared with SlotRefine, our model also drops much faster and achieves better convergence after 5 epochs of training.  To further analyze the error in SF tasks in detail, we show the statistics of slot error on the validation set of SNIPS after training 100 epochs. The statistics show that the uncoordinated slot problem composes a big part of all slot errors. Without any approach to solving this problem, our basic model encounters 57 uncoordinated slots, composing 33.7% of all slot errors. In these uncoordinated slots, 31 are caused by the "BI error, " while the "IB error causes 26. " The proportion of the "BI error" and the "IB error" is almost close.
Compared with the basic model, our LRT reduces 44 uncoordinated slots. The proportion of uncoordinated slots in all incorrect slot labels drops from 33.7% to 11.1%. Moreover, our LRT efficiently reduces both the BI and IB errors simultaneously. The reducing proportion between the BI error and the IB error of our model is also close. This is mainly because our SLG effectively obtains sequential dependency while LRM considers both B-tag and I-tag. As a comparison, SlotRefine corrects most uncoordinated slots caused by the "BI error" but still suffers "IB error. " For SlotRefine, the "IB error" proportion is much higher than the "BI error. " We provide an example for the case study. In Fig 9, we notice that the basic model suffers a serious uncoordinated slot problem, including both two cases, i.e., "BI error" and "IB error," respectively. SlotRefine solves "BI error" but predicts incorrect slot labels in the "IB error" case due to the error propagation from the wrong B-tag label. Our model solves the problems of both cases and predicts a correct slot label sequence.
Above all, our LRT indeed remedies the problem of the uncoordinated slots, leading to better performance on SF.

Ablation Study
In this section, we do the ablation study to verify the effectiveness of SHA, LRM and SLG in detail.

Ablation Study on Multi-turn SLU Tasks.
We first make the ablation study on the multi-turn SLU dataset KVRET.  To begin with, we first analyze how performance varies based on the location of the utterance in the dialogue. As shown in Fig. 10, the performance of the LRT model drops heavily at the late part of the dialogue in both the ID and SF. This phenomenon is mainly because of two reasons. First, users usually offer more detailed descriptions in the early dialogue, especially for the initial utterance. Moreover, later utterances rely more on salient historical information than utterances in the early dialogue. Compared with LRT, the decline of SHA-LRT is significantly reduced, which initially verifies the significance of our SHA module. We will further verify the effectiveness of our SHA module in the following. Effect of SHA: From Table. 9, we find that the RNN-based framework RPFSLU is incompatible with the Transformer-based model. Employing RPFSLU for the Basic model or LRT even reduces prediction accuracy for SLU tasks. This phenomenon further emphasizes the positive meaning of our motivation to design SHA.

Model
In comparison, as mentioned in section 4.2, our SHA module significantly enhances the SLU performance for the Transformer-based model. As shown in Table. 9, compared with only utilize the Basic model, Basic model+SHA enhances the performance by 2.0%, 16.4% and 17.1% (1.9%, 16.3%, 17.1% for SHA-P) on ID,SF and Overall, respectively. Moreover, compared with LRT, SHA-LRT brings an enhancement of 1.2%, 17.8% and 18.0% (1.6%, 17.6%, 17.7% for SHA-P) on ID,SF and Overall, respectively. This enhancement is mainly originated from the semantic information contained in the historical utterances and results. SHA effectively incorporates the salient historical information into the model, resulting in a satisfying performance.
To further detect the origin of this enhancement, we design some variant architectures of SHA as follow: • SHA-LRT (Utterance-Only): Remove the History-Result-attention of SHA and direct input H into the feed-forward network. In this architecture, only historical utterances will be incorporated into the model. In this architecture, we incorporate historical results into the model, referring to the semantic similarity of historical utterances but ignoring the salient information of the corresponding utterances. By conducting experiments for these variant models, we find that the effect of only utilizing historical utterances is minimal (%0.9 on Overall). However, the model has a notable improvement (%7.6 on Overall) when only utilizing historical results. This result manifests that the enhancement made by SHA is mainly contributed by efficiently exploiting the historical results. Further, ResultAttention-Only enhances the Overall performance of Result-Only by 7.8%, which verifies the importance of referring to the semantic similarity of historical utterances when doing historyattention. The complete SHA module further enhances ResultAttention-Only by 1.3% on Overall, indicating historical utterances also contain salient information and need to be concerned.
We then compare the effect of SHA and SHA-P. When LRM and SLG are both utilized, SHA performs better on SF and overall accuracy, and SHA-P performs better on ID. In other cases, SHA performs better on all metrics.
Finally, we verify the compatibility of the SHA module with our other modules. As shown in the third group of Table 9, we make a permutation for our SLG, LRM, and SHA module. We can first find that both SLG and LRM are able to work non-conflicting with our SHA module and improve the prediction accuracy. Compared with only utilizing the SHA module (i.e., Basic model + SHA/SHA-p), LRM enhances the SF performance by 1.4%/1.7% and SLG by 1.5%/1.2%. These three modules can also combine as our complete model and further enhance the SLU performance. Therefore, all modules in our SHA-LRT have high compatibility.

Ablation Study on Single-turn SLU Tasks.
Subsequently, we do an ablation study for LRM and SLG. As we have verified the effectiveness of those modules on multi-turn tasks, we make further analysis for LRM and SLG on single-turn datasets in this section.
Effect of LRM: By comparing basic model + LRM with the basic model, we find that LRM enhances the performance by 0.9% (ID), 0.8% (SF), and 0.8% (Overall) on ATIS and 1.9% (ID), 0.8% (SF), and 2.8% (Overall) on SNIPS, which shows that utilizing LRM alone still enhances SLU performance on both two tasks. We attribute the improvement to the direct utilization of ID and SF preliminary results. The semantic information from both slot labels and ID labels enhances the performance when predicting.  Fig. 11. SF performance comparison of using LRM in different Transformer encoder intervals on SNIPS validation set. 1-2 represents the interval of the 1st and the 2nd Transformer encoder layer, and so on.
As we introduced in section 3.4, LRM can be utilized in each two Transformer encoder layer intervals. Motivated by finding the best place to operate LRM, we conduct the experiment to compare SF performance on the validation set of SNIPS when using LRM in different Transformer encoder intervals. The experiment result is shown in Fig. 11.  We further conduct experiments to evaluate SF performance for different usage counts of LRM. As shown in Fig. 12, the F1 score on the validation set achieves a satisfying level when employing LRM once or twice and drops significantly with increasing usage count. This phenomenon indicates that employing LRM one time is enough, and overuse LRM will negatively influence SLU prediction. We have introduced the reason in section 3.3, using LRM once in the whole Transformer structure already considers the interaction between ID and SF. Overusing LRM will make the Transformer layers lose their own feature and negatively affect the final performance due to the utilization of the SLU classifier in LRM.
Effect of SLG: As shown in Table 10, compared with the basic model, the basic model + SLG also enhances the performance. Specifically, SLG brings enhancement by 0.3% for ID, 1.1% for SF, and 1.0% for overall on ATIS, while 1.6% for ID, 1.4% for SF, and 4.4% for overall on SNIPS. We attribute this improvement to the autoregressive structure of SLG, which brings sequential solid dependency information, making prediction more accurate.
Moreover, comparing LRT with LRT* and LRT+SLG with LRT+SLG*, we find that utilizing consistency loss function in Eq.32 brings a slight enhancement. More importantly, the consistency loss effectively reduces the standard deviation, especially on SF and overall. This is mainly because utilizing consistency loss can enhance the prediction consistency between the SLU and SLG tasks, making the model more stable. Moreover, the standard deviation on ID tasks is much lower compared with SF. We consider this is because ID contains fewer categories and is easier to be predicted. We further conduct experiments to study the impact of on the SF performance. In this experiment, we keep the other settings and change the from 0 to 0.5. According to the results shown in Fig 13, we find that the F1 scores change little when increases from 0 to 0.35, but the stand deviation drops continuously. Then, when is larger than 0.4, both the F1 score and the stand deviation worsen rapidly. We consider the large weight of makes the wrong prediction in SLU excessively affect the SLG task and bring negative effect since the predicted labels of SLU are not the same as the ground-truth labels. Similar to , we also conduct experiments to study the impact of on the SF performance. As shown in Fig. 14, the F1 score enhances when changes from 0 to 0.25, which indicates the effect of our SLG module. Then, the F1 score continues to rise until reaches 0.75 and begins to drop with the increase of . This observation indicates that we should make a suitable balance between the SLU and SLG in our final framework.

Effect of Pretraining
In this section, we conduct experiments to evaluate our model's ability to combine the pre-trained model.
Recently, some work builds their models based on the large-scale pre-trained models, e.g., BERT [12], RoBERTa [29], XLNet [51], etc., which utilized billions of external corpus and tremendous model parameters. Since the number of parameters in these models is much more than ours, it is unfair to compare the performance of our model with them directly. Thus, we also combine our approaches with pre-trained models to highlight the effectiveness of SHA-LRT.
We first conduct experiments on the multi-turn SLU dataset KVRET. In practice, we combine pre-trained models with SHA-LRT and fine-tune the complete model on the SLU dataset. More specifically, we employ pre-trained models to obtain hidden states of the utterance. Then we replace the embedding sequence with the pre-trained hidden states as the input of our SHA module to predict SLU results.
For baselines, we first select BERT-SLU [6], then we replace BERT in BERT-SLU with RoBERTa and XLNet to construct RoBERTa-SLU and XLNet-SLU, respectively. The experiment results are shown in Table 11. From Table 11, we find that our model is benefited from all three pre-train models. More specifically, the Overall of SHA-LRT (SHA/SHA-P) is enhanced by 2.8%/2.7% with BERT, 3.8%/3.5% with RoBERTa, and 5.4%/4.9% with XLNet, respectively. These results intuitively indicate that the pre-trained model further improves the effectiveness of our model, which verifies the compatibility between our model and the pre-trained model.

Model
Besides, as shown in Table 11, the performance of BERT-SLU/RoBERTa-SLU/XLNet-SLU in multi-turn SLU tasks results is unsatisfied due to lacking historical information. Even dealing with dialogue history by CAT-All, the performance of these models is still suffering low accuracy. This phenomenon verifies the effectiveness of SHA once again.
Moreover, to further evaluate the ability of SHA-LRT to combine the pre-trained model, we conduct experiments in the single-turn SLU tasks. To better use the pre-trained results, we retain the weights of the original pre-trained model and do not change their inner structure. Therefore, we only combine our SLG module with the pre-trained model. In practice, we employ pre-trained models as the encoder and utilize the same Transformer decoder as we introduced in section 3.5. Then we fine-tune the complete model on the SLU dataset. As shown in Table 12, BERT+SLG outperforms all previous BERT-based models on all evaluation metrics. Compared with BERT-SLU, SLG brings an enhancement of 0.8% (ID), 0.1% (SF), and 0.5% (Overall) on ATIS dataset and 0.5% (ID), 0.1% (SF), and 0.3% (Overall) on SNIPS dataset. All of these improvements of our model are statistically significant with < 0.05 under t-test. Similarly, RoBERTa+SLG also performs better than all previous RoBERTa-based models. Note that XLNetbased models perform terribly compared with BERT and RoBERTa. This is mainly because the amount of parameters in XLNet is too big, and the model can not be well finetuned with these two datasets. However, SLG still enhances XLNet, and the combined model XLNet+SLG performs better than XLNet+SlotRefine. The results show that our SLG approach is still useful for pre-trained models, which indicates that our SLG module is portable and effective.

Model
From Table 12, we further find that the improvement of pre-trained models on the SNIPS dataset is much more obvious than that on the ATIS dataset. Considering that SNIPS contains much more Out-of-Vocabulary (OOV) tokens, we attribute the improvement of BERT to alleviating the OOV problem by the WordPiece encoding [41].
To compare the inference speed with existing SLU models, we also record the inference time for each SLU model when employing BERT. As shown in Table 13, our BERT+SLG also achieves significant speedup (×4.50 on ATIS; ×4.55 on SNIPS) against the autoregressive SOTA model Stack-Propagation. Meanwhile, compared with the existing non-autoregressive model SlotRefine, our model can reduce 50% inference time, as our model does not need to run BERT two times. It is worth noting that our model consumes almost the same time as BERT-SLU [6] Table 13. Latency of Bert-based SLU models. "Latency" is the average inference time without minibatching. "Speedup" is compared against the existing SOTA model BERT+Stack-Propagation.
Besides, although BERT can bring enhancement, the trade-off for this enhancement is 3.77 times of inference latency when comparing BERT+SLG with SHA-LRT in Table 13.
To sum up, our proposed model is well combined with the pre-trained models and outperforms all existing models on both multi-turn and single-turn SLU tasks.

CONCLUSION
In this paper, we propose a fast and accurate non-autoregressive model for multi-turn SLU task,i.e., SHA-LRT. To capture the salient information in the dialogue history, we design the Salient History Attention module, which significantly enhances the multi-turn-SLU performance by taking advantage of historical utterances and historical results with a History-attention approach. To solve the uncoordinated slot problem, we propose a Layer-Refined Mechanism and the Slot Generation task, respectively. LRM guides the final prediction via the interaction of intermediate predicted results between Transformer layers. By this process, LRM explicitly introduces the correlation between ID and SF into the model with a little cost of time. SLG addresses the sequential dependency information and effectively enhances the performance of SF through multi-task learning with no extra inference time. Experiments on KVRET datasets indicate that our model outperforms other previous multi-turn SLU methods by a large margin on both SLU performance and the inference speed. Moreover, we conduct experiments on two public single-turn SLU datasets and further find that our model is also effective on single-turn SLU tasks, which manifests the robustness of our model.