{'fraction_non_alphanumeric': 0.029080899952130205, 'fraction_numerical': 0.012386309238870272, 'mean_word_length': 4.83028083028083, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Diaformer: Automatic Diagnosis via Symptoms Sequence Generation

Automatic diagnosis has attracted increasing attention but remains challenging due to multi-step reasoning. Recent works usually address it by reinforcement learning methods. However, these methods show low efficiency and require taskspecific reward functions. Considering the conversation between doctor and patient allows doctors to probe for symptoms and make diagnoses, the diagnosis process can be naturally seen as the generation of a sequence including symptoms and diagnoses. Inspired by this, we reformulate automatic diagnosis as a symptoms Sequence Generation (SG) task and propose a simple but effective automatic Diagnosis model based on Transformer (Diaformer). We firstly design the symptom attention framework to learn the generation of symptom inquiry and the disease diagnosis. To alleviate the discrepancy between sequential generation and disorder of implicit symptoms, we further design three orderless training mechanisms. Experiments on three public datasets show that our model outperforms baselines on disease diagnosis by 1%, 6% and 11.5% with the highest training efficiency. Detailed analysis on symptom inquiry prediction demonstrates that the potential of applying symptoms sequence generation for automatic diagnosis.


Introduction
Automatic diagnosis has recently attracted increasing attention from researchers because of its potential in simplifying diagnostic procedures (Tang et al. 2016;Kao, Tang, and Chang 2018), helping make better and more effective diagnostic decisions (Shivade et al. 2014;Xia et al. 2020), and even helping build a diagnostic dialogue system as a dialogue management Wei et al. 2018;Xu et al. 2019;Teixeira, Maran, and Dragoni 2021). An automatic diagnosis system is built on conversations between the agent and the patient where allows the agent to probe for symptoms and make diagnoses. As an example is shown in Table 1, the agent interacts with users to inquiry about additional symptoms (i.e., implicit symptoms) beyond their self-reports (i.e., explicit symptoms) and make a disease diagnosis at the end. When inquiring about additional symptoms, the automatic diagnosis system can only obtain the Table 1: An example of automatic diagnosis data. value of symptoms in the "implicit symptoms" set or get a "not sure" answer for outside symptoms. Thus, the disease diagnosis task can be defined as inquiring the implicit symptoms step by step with limited interaction turns, and then diagnosing the disease based on explicit symptoms and the additional symptoms inquired. Note that different from the dialogue system of automatic diagnosis, automatic diagnosis we called here is symptom checking task in (Tang et al. 2016), which also serves as dialogue manager in taskoriented dialogue system of automatic diagnosis (Wei et al. 2018;Xu et al. 2019;Luo, Li, and Glass 2020;Xia et al. 2020;Teixeira, Maran, and Dragoni 2021).
Due to the existing of implicit symptoms, this task can be considered as a multi-step reasoning problem. The challenge of the task is how to capture the underlying dynamics and uncertainties of reasoning process, then inquiry about accurate symptoms under small labeled data and limited turns. Most previous methods usually address this problem as a sequential decision-making process, then formulate the process as Markov Decision Processes (MDPs) and employ Reinforcement Learning (RL) for policy learning (Tang et al. 2016;Peng et al. 2017;Liu et al. 2017;Ling et al. 2017;Kao, Tang, and Chang 2018;Wei et al. 2018;Xu et al. 2019;Liao et al. 2020;Xia et al. 2020;Hou et al. 2021;Teixeira, Maran, and Dragoni 2021). However, RL learns how to inquire about symptoms and make a disease diagnosis only with final accuracy rewards, which partly deviates from the actual doctor's diagnostic process. In real clinical diagnosis scenario, doctors carefully select relevant questions and ask patients with a medical diagnostic logic (Xia et al. 2020). The policy learning of RL tries to learn which symptom inquiry improves the rewards but not the doctor's diagnostic logic directly. As a result, RL relies on the random trials to learn how to improve the reward, but don't learn directly the correlation among symptoms and the standard diagnosis paradigm. It leads to low efficiency in learning how to make symptom inquiry decisions. Besides, there is still no explicit solution to find ideal reward functions, which may make the RL-based model hard to balance the decision learning between disease diagnosis and symptom inquiry.
Considering the diagnosis process can be naturally seen as the generation of a sequence, we reformulate automatic diagnosis as a Sequence Generation (SG) task in this work. Different from RL-based methods, the multi-step inquiry process is explicitly modeled at generating a sequence including symptoms and diagnoses. This can improve the efficiency and explainability of multi-step reasoning process. Moreover, the latent relationship among previous explicit symptoms and current symptom can be learned. Hence, the accurate inquiry of implicit symptoms would help to improve the accuracy of disease diagnosis, which is similar to doctors' diagnostic logic. As the example shown in Table 1, RL-based models tentatively learn which symptom inquiry helps to predict the children's bronchitis target using policy learning in large state/action spaces. By contrast, the SGbased model learn to inquire sore throat, fever and brash breath sequentially based on the explicit symptoms, so that the model can learn the latent relationship of symptoms inquiry and the diagnosis decision-making more efficiently.
As a step forward, we propose a simple but effective automatic Diagnosis model based on Transformer (Diaformer). It consists of a symptom attention framework and learns with three orderless training mechanisms. The symptom attention framework is introduced to model the automatic diagnosis using the Transformer architecture (Vaswani et al. 2017). The self-attention mechanism of Transformer can reduce the position dependence and learn multiple relationships among the multiple symptoms. To consider previous symptoms in current step, we also propose a attention mask mechanism. Each implicit symptom can attend to the given explicit symptoms and the previous implicit symptoms, while each explicit symptom can only see the explicit symptoms. As we learn the symptom inquiry by symptoms sequence generation, there is a bias caused by the discrepancy between the order of symptoms sequence learned and the disorder of golden implicit symptoms. To address this challenge, we further propose three orderless training mechanisms: sequence shuffle, synchronous learning, and repeated sequence. The main idea is to encourage the model to inquire symptoms in an orderless but accurate way, whereby improving the generalizability at inference time. Extensive experiments on MuZhi dataset (Wei et al. 2018), Dxy dataset (Xu et al. 2019) and Synthetic dataset (Liao et al. 2020) show that our proposed model (Diaformer) outperforms baselines on disease diagnosis by 1%, 6% and 11.5% with the highest training efficiency. Further analysis on symptom inquiry prediction demonstrates applying symptoms sequence generation is an plausible way to automatic diagnosis task.
Our contributions are summarized as follows: • To the best of our knowledge, we are the first to apply symptoms sequence generation for automatic diagnosis.
We further show that our method can be applied under few conversation turns scenarios.
• We propose three orderless training mechanisms to alleviate the discrepancy between the sequential generation and the disorder of given implicit symptoms sets. The ablation studies show that these orderless mechanisms can significantly alleviate this bias.

Preliminaries
In this section, we first formulate the automatic diagnosis task as a sequence generation (SG) task. Formally, an automatic diagnosis data includes an explicit symptom set S exp , an implicit symptom set S imp and a target disease Dis. As shown in Figure 1, S exp = {Sym 1 , Sym 2 } and S imp = {Sym 3 , Sym 4 , Sym 5 }. For the task, the automatic diagnosis system can only access the explicit symptoms S exp at the beginning. Then the system can inquire symptoms in limited turns to obtain the implicit symptoms in S imp . When the symptom inquires a symptom, the user simulator will take one of the three answer including True for the positive symptom, False for the negative symptom, and Not sure for the symptom that is not mentioned in user goal S exp ∪S imp . We denote S add which S add ⊆ S imp as the additional symptoms that had been inquired by the system. In the end, the system is asked to make a disease diagnosis based on the explicit symptoms and the addition symptoms. The task objective of learning is maximize the likelihood of disease diagnosis P (Dis | S exp ∪ S add )P (S add | S exp ).
Since the implicit symptoms contain important information that inquired by the doctors, the intuition is that the more implicit symptoms the model inquires, the higher diagnosis accuracy model can get. We transfer task learning objective to maximize P (S imp | S exp ) as well as: then we use a symptom attention framework, shown in Figure 1, to learn predicting the implicit symptoms sequentially. Thus, we need to change the orderless S imp into the ordered symptoms sequence T imp as the target generation sequence. According to the autoregressive generation, the probability of output token depends on all previous. The objective of model learning is transferred to maximize the likelihood of T imp generation: where the T i imp denote i-th symptom in T imp and T <i imp denote all the symptoms in front of T i imp . Hence we change the symptom inquire task to a sequence generation task. Since the discrepancy between the order of T imp and the disorder of S imp , the SG training objective Eq.(2) is unequal to the Eq.(1), which seriously hinder the performance in automatic diagnosis. We propose three training mechanisms to make training objective Eq.(2) approximate to Eq.(1).

Methodology
In this section, we introduce the Diaformer, which consists of symptom attention framework and the orderless training mechanisms. Then we elaborate the generative inference.

Symptom Attention Framework
The illustration of symptom attention framework is shown in Figure 1. In this framework, we adopt multiple stacked Transformer blocks to model the automatic diagnosis by SG. Each block contains a feed forward network and a multihead attention, in which all input tokens share the parameters via self-attention (Vaswani et al. 2017).
Input Representation As shown in Figure 1a, all the symptoms in S exp and S imp are converted to the specific token embedding. Different from Transformer, we remove the position embedding, and the symptom input representation is computed by summing the corresponding token embedding, symptom state embedding and symptom type embedding.  Figure 1b show the attention mask matrix M, which determines whether query and key can attend to each other in self-attention by modifying the attention (Vaswani et al. 2017). Specifically, M is assigned as:

Attention Masks
, where 0 and −∞ indicate red point and white point in Figure 1b. With it, we can prevent symptom prediction from seeing leaked information in self-attention layer to achieve autoregressive generation training of implicit symptoms.
Symptom Token Attention Before input the framework, we initialize a implicit symptoms sequence T imp by S imp . As shown in Figure 1c, in each multi-head attentions of Transformer block, each explicit symptom can merely see the explicit symptoms and each implicit symptom can see the explicit symptoms and the previous implicit symptoms.
To be specific, the representations of symptom tokens are update in multi-head attention (Vaswani et al. 2017) as: where Q, K, V denote the query, key and value in multihead attention, [.] denotes concatenation along the sequence dimension, T i(l) imp indicates the l-th Transformer block layer output of the i-th implicit symptom in T imp and S (l) exp denotes l-th layer output of the explicit symptoms.
Decoder As shown in Figure 1a, we insert a [S] sequence which length is (n+1), where n is the number of implicit symptoms. We used the [S] sequence to learn generating the implicit symptoms as sequential symptom inquiry in the automatic diagnosis. The i-th [S] in [S] sequence is set to predict the i-th symptom in T imp . And the last [S] in the [S] sequence is used to predict the action of ending inquiry, which is a END symbol in Figure 1a. The attention flow of [S] sequence is shown in Figure 1d. In the autoregressive symptoms sequence generation, each [S] cannot see the target symptom and the implicit symptoms after the target symptom in T imp , aimed to learn the target symptom without information leakage. Specifically, the representation of [S] token is update as: i denotes the l-th layer output of the i-th token in [S] sequence. For the last layer outputs of [S], we use a symptom classification layer that contains a liner layer weights W sym ∈ R H×Cinq , where H is the hidden size and C inq is the number of symptom inquiry types. For the symptom classification layer output z, we compute the cross entropy loss with softmax, i.e. −log(sof tmax(z)), as the symptom inquiry loss L sym . As for the disease classification, we insert a special token [D] to input sequence, shown in Figure 1a. As shown in Figure 1d, the token [D] can attend all the symptoms. The vector of [D] is updated as: Similar to [S], on the last layer output of [D], we adopt a disease classification layer weights W dis ∈ R H×C dis , where C dis is the number of disease types. We compute the cross entropy loss of it as the disease classification loss L dis . The final training loss is computed as L = L dis + L sym .

Orderless Training Mechanisms
To alleviate the bias caused by the discrepancy between the order of the sequential generation of T imp and the disorder of implicit symptoms S imp , We propose three orderless training mechanisms of SG, which is demonstrated in Figure 2 and detailed as follow.
Sequence Shuffle We randomly shuffle the implicit symptoms sequence T imp to obtain new implicit symptoms sequence T imp in different order before input the model in each training step, so that the model can learning different order of symptom inquiry after multiple training epoch. It can prevent the model to impose over-fitting on inquiring symptoms in a specific order and fail to inquire the correct implicit symptom in a slightly different context. With the increase of training epoch, the model will gradually fit the symptom disorder distribution, whereby the T imp sequence generation approximate to the S imp inference.
Synchronous Learning While the model predicts the next symptom inquiry, we expect all the implicit symptoms, which have not been inquired, have equal probability to be inquired. As shown in Figure 1a, the [S] 1 is trained to predict the Sym 3 . However, Sym 3 , Sym 4 and Sym 5 should have the same priority to be inquired in the orderless set of the implicit symptoms. Therefore, we design the synchronous learning objective to train the model to synchronously predict the rest implicit symptoms that it can't see. As shown in the Figure 2, each symptom prediction token [S] is trained to predict all the rest implicit symptoms synchronously. Therefore, we use a concurrent softmax ) replacing the original softmax to train [S] to predict multiple symptoms synchronously. We remove the concurrent rate in ) and only use the concurrent softmax as a training mechanism, as we still use softmax in inference. The concurrent softmax can enable the model to learn multiple symptoms synchronously and eliminate the discrepancy between training and inference. The reset implicit symptoms S imp −S add is set as the training objective, so the concurrent softmax label y is defined as: where y i denote the label of inquiry class i. As for the symptom classification layer ouput z, the cross entropy loss of concurrent softmax is presented as: where C inq denote the number of inquiry type. To balance label learning in [S] sequence, the loss of a single [S] is divided by the number of synchronous labels. With the synchronous learning, the training objective Eq.
(2) transfer to , which is more approximate to the Eq.(1). Moreover, the synchronous learning helps improve training efficiency.
Repeated Sequence Due to the autoregressive generation, the model is plagued with learning a specific order generation of implicit symptoms in each training step. This results in a training imbalance of symptoms generations in different orders. For example in Figure 1a, the model can learn to inquiry symptoms sequentially as Sym 3 → Sym 4 → Sym 5 , but fail to learn to inquiry symptoms as like Sym 5 → Sym 4 → Sym 3 , because of the unidirectional sequence generation. To reduce the impact of unidirectional generation, we constructed the repeated sequences concatenated with the input sequence to learn different orders of sequence generation. As shown in Figure 2, sequence is added to enable that generate implicit symptoms as Sym 5 → Sym 4 → Sym 3 . The repeated sequence consist of last (n-1) implicit symptoms, in a new order sequence randomly shuffled like sequence shuffle, and (n-1) [S] tokens, set to predict symptoms in the new order. The repeated sequences share the first [S] and END symbol prediction with original sequence. Benefit from the ability of long dependency and the parallel computing in Transformer, we can expand input sequence with the repeated sequences to alleviate the bias between Eq.
(2) and Eq. (1), and improve the training efficiency of orderless generation. Since the symptoms sequence is relatively short, we set the number of repeated sequences as 4. inquire Step 1 True True predict "END" or inform symptom with low confidence infer infer inquire If not sure, find the next symptom Step 2 Step 3 infer inquire infer

False
Step 4

diagnosis
Step 5 Figure 3: Schematic of inference process. User is the user simulator.

Generative Inference
During inference stage, we firstly inserted a symptom prediction token [S] behind the symptoms sequence to calculate the probability distribution of symptoms. Then we mask the symptoms have been inquired and inquire the rest highest probability symptom. Next, the user simulator would determine if the inquired symptom is in the implicit symptoms set. If the symptom is not an implicit symptom, the model will find the next highest probability symptom to inquire user. Once the inquiry symptom is a implicit symptom, the user simulator will reply True or False for the symptom, and then the model insert it into the symptoms sequence and predict new probabilities of inquiry symptom, detailed as Figure 3. When the model predict END symbol with probability greater than ρ e or infer the inquiry symptom with probability less than ρ p , the model will stop symptom inquiry to diagnose disease. In disease diagnosis, we insert the disease prediction token [D] into the symptoms sequence to predict the disease.  MuZhi  4  66  568  142  Dxy  5  41  423  104  Synthetic  90  266 24,000 6,000  Overall Performance According to the task definition in (Wei et al. 2018) and (Liao et al. 2020), we set the maximum inquiry turn as 20. We evaluate all the model by three metrics, which are diagnosis accuracy, average inquiry turns and recall of the implicit symptoms. The recall of the implicit symptoms is a significant metric for SG-based models, which aim to inquire the implicit symptoms out as much as possible and then diagnose the disease. Besides, we add a training time metric to evaluate the training efficiency of models. The results on three datasets are shown in Ta Table 3: Results on three datasets. DAcc is the accuracy of diagnosis; SRec is the recall of the implicit symptoms; ATurn is the average of symptom inquiry turn; Ttime indicates the training time to get the best diagnosis result running on a 1080Ti GPU; Ttime's unit "m" indicate minute; ‡ marks the results reported by the original papers.
other models in diagnosis accuracy with highest training efficiency. Especially on Dxy dataset and Synthetic dataset, our model outperforms the current state-of-the-art model by 6% and 11.5% in diagnosis accuracy. Note that SG-based models have considerably high recall of implicit symptoms and perform much better on bigger dataset, i.e. Synthetic dataset. The upsurge on bigger dataset demonstrates the high training efficiency of SG-based models, which only need to be trained in 2 minutes on Dxy dataset and MuZhi dataset. It indicates that the symptoms sequence generation has considerable potential in automatic diagnosis. Additionally, Diaformer surpass the other SG-based mdoel, which demonstrates the symptom attention framework and the orderless mechanisms can further improve the performance and training efficiency in SG-based models. Moreover, we observe that SG-based models request more inquiry turns, due to the higher recall of symptoms lead to more inquiry turns. For simulate the process of doctor diagnosis, SG-based models tend to generate more implicit symptoms. For this limitation, we conduct the experiments of smaller limited turns.
Diagnosis with Smaller Limited Turns Considering higher recall of implicit symptoms requesting more inquiry turns that is unfair to the less inquiry turns model, we conduct the experiments with smaller maximum turns, including 5, 10, 15. Due to the limitation of not being open source for some models, we conduct this experiment on two RLbased model and all SG-based models. Table 4 shows the result of disease diagnosis with smaller limited turns. Overall, Diaformer still outperform other models in the limit of smaller turns in terms of recall of implicit symptoms and diagnosis accuracy. Note that in the limit of smaller turns, Diaformer always recall more implicit symptoms than the other SG-based models with almost equal turns, which indicates the higher performance of Diaformer. On the Synthetic dataset, SG-based models outperform the PPO model with smaller average turns. In the limit of 5 turns, Diaformer obtains competitive results of diagnosis accuracy with comparably few inquiry turns. The results of smaller limited turn indicate that SG-based models can still perform automatic diagnosis well with the smaller inquiry turns.

Ablation Study
We perform an ablation study to understand the importance of three training mechanisms of orderless generation. In Table 5, we compare three Diaformer variants (rows 2 -4) without one of the three orderless training mechanisms. As shown in Table 5, we can see each mechanism contribute to improve the performance. Without sequence shuffle, the model obtain lower recall of symptoms with higher inquiry turns. Without synchronous learning or repeated sequence, the accuracy of diagnosis and the recall of symptoms both reduce. Specifically, Figure 4 show the results of on Synthetic dataset in the series of training epoch as the same parameter initialization, in which Diaformer obviously performs best on diagnosis accuracy and recall of implicit symptoms. It indicates that orderless training mechanism help to improve the performance of symptoms sequence generation in automatic diagnosis.
learning to simultaneously conduct symptom inquiries and diagnose. Based on the work of (Tang et al. 2016), Kao, Tang, and Chang (2018)  Sequence Generation Sequence generation task aims to generate a target sequence condition on a source input. It covers many areas with a lot of tasks ). Among them, the natural language generation (NLG) have achieved great success with the development of neural networks. Recently, the Transformer-based models have obtained superior performance in NLG, such as (Radford et al. 2019;Yang et al. 2019;Song et al. 2019;Brown et al. 2020;Bao et al. 2020;Xiao et al. 2020;Sun et al. 2021). Most of them train as auto-regressive (AR), in which the probability of an output token depends on all previous tokens. Based on AR training objective, the sequence generation model can learn all the target tokens in parallel. Besides, some of them use the additional artificial symbol sequence (Xiao et al. 2020;Brown et al. 2020) or combine Masked Language model (Devlin et al. 2018) as auto-encoding (AE) (Dong et al. 2019;Bao et al. 2020) to enhance the model. With more and more relevant models being proposed, Transformer has shown great potential in sequence generation.

Conclusion
In this work, we reformulate the automatic diagnosis problem as a sequence generation task and propose a symptom attention framework for automatic diagnosis with symptoms sequence generation. Besides, we propose three orderless training mechanisms to alleviate the bias of the discrepancy between the sequential generation and the disorder of symptoms. Experimental results show that our model outperforms other models on three datasets of automatic diagnosis and demonstrates the potential of symptoms sequence generation in automatic diagnosis. Future work includes incorporating Diaformer into task-oriented dialogue system of diagnosis and effectively lessen the inquiry turns.

Ethics Statement
The problem of facilitating diagnosis is important in artificial intelligence applications. Our method shows promising accuracy and noticeable efficiency on automatic diagnosis, which is a multi-step reasoning problem as well as a symptom checking task in (Tang et al. 2016). Besides strong performance, the automatic diagnosis models learned from the insufficient and incomplete dataset has considerable risk of predicting error that may cause seriously harm. Under the ethical considerations, we suggest users regard it as an auxiliary tool that can help doctors make diagnose or help to give patients some advice. In fact, the proposed Diaformer is not limited to the automatic diagnosis problem. It can extend to use in some decision-making problems or RL problems through slight change. As for the usage on other problems, we suggest users design more intermediate-state tokens along with the decision tokens to form a decisions sequence and adjust the type embedding and attention mask mechanism for specific problems. Different from Decision Transformer  in the typical RL problems, our model tends to learn the relationship among the decisions directly and focus on alleviating the order bias for orderless or non-sequential RL problems. Note that all the decision-making models run the risk of biased prediction in real-life application scenarios, and be careful to use them.

Appendices A Why Use Concurrent Softmax
In the synchronous learning, we attend to train the output of [S] to predict multiple symptoms, which can be regarded as a multi-label task. Normally, the Bernoulli parameter for each symptom inquiry label is considered as a good choice to the multi-label task. However, there are three reason that we choose the concurrent softmax: (1) the synchronous labels are sparse, due to the number of implicit symptoms is much less than the number of symptom inquiry types. Training as Bernoulli parameter will perform inefficiently. (2) The synchronous learning is a training mechanism in the training stage, while we still use softmax to compute the probability distribution of symptom inquiry. As a result, if we use the Bernoulli parameter, it will lead to a big discrepancy between training and inference. In contrast, the concurrent softmax is almost same as softmax, used to learn the probability distribution of symptom inquiry. (3) Due to the imbalance of the symptom inquiry label, the independent learning in the Bernoulli parameter give rise to a serious overfitting problem. In our pilot experiments, the concurrent softmax indeed perform much better than the Bernoulli parameter.

B Dataset Details
MuZhi Dataset The MuZhi dataset is constructed by (Wei et al. 2018). The data is collected from the pediatric department of a Chinese online medical website, which is a popular website for users to consult doctors online. Usually, a patient will provide a self-report to indicate his or her basic information.Then the doctor will initiate a conversation to collect more information and make a diagnosis based on parent's self-report and conversations. The doctor can obtain additional symptoms not mentioned in the self-report during the conversation. For each patient, the doctor will give the final diagnosis as the label. In the dataset, symptoms extracted from self-report are regarded as explicit symptoms and the ones extracted from conversation are implicit symptoms. More detailed dataset statistics are shown in Table 6.

Dxy Dataset
The Dxy dataset is constructed by (Xu et al. 2019) and the data is collected from a Chinese online healthcare community, where users asking doctors for medical diagnosis or professional medical advice. The dataset contains five types of diseases, including allergic rhinitis, upper respiratory infection, pneumonia, children hand-foot-mouth disease, and pediatric diarrhea. And (Xu et al. 2019) extract the symptoms that appear in self-reports and conversation and normalize them into 41 symptoms. More detailed dataset statistics are shown in Table 6. Similar to the MuZhi dataset, symptoms appearing in self-reports are regarded as symptoms while the others are implicit symptoms.

Synthetic Dataset
The Synthetic dataset is constructed by (Liao et al. 2020) following (Kao, Tang, and Chang 2018), which is based on symptom-disease database called Sym-Cat. There are 801 diseases in the database and (Liao et al. 2020) classify them into 21 departments (groups) according to International Classification of Diseases (ICD-10-CM) and choose 9 representative departments from the database.
Each department contains top 10 diseases according to the occurrence rate in the Centers for Disease Control and Prevention (CDC) database. More detailed dataset statistics are shown in Table 6.

Minimum Baseline
• SVM-exp: To indicate the significance of symptom inquiry and give the minimum boundaries in automatic diagnosis, we use SVM model learned to distinguish the disease on explicit symptoms directly without any symptom inquiry. The SVM we used is based on the linear kernel and adopt one-vs-one scheme to support multiclass.

RL-based Models
• Flat-DQN: This is the model from (Wei et al. 2018), which has one layer policy and an action space including both symptoms and diseases. • HRL: This is the model from (Liao et al. 2020), which integrates a hierarchical policy of two levels into the dialogue policy learning. The high level policy consists of a model named master that is responsible for triggering a model in low level, and the low level policy consists of several symptom checkers and a disease classifier. • KR-DS: This is the model from (Xu et al. 2019), which seamlessly incorporates rich medical knowledge graph into the topic transition in dialogue management. The model use a novel Knowledge-routed Deep Qnetwork (KR-DQN) ,which integrates a relational refinement branch for encoding relations among different symptoms and symptom-disease pairs, and a knowledgerouted graph branch for topic decision-making. • GAMP: This is the model from (Xia et al. 2020). GAMP is a generative adversarial regularized Mutual information Policy gradient framework (GAMP) for automatic diagnosis, that based on the Generative Adversarial Network (GAN) to optimize the RL model for automatic diagnosis. • PPO: This is the model from (Teixeira, Maran, and Dragoni 2021), which is proposed to leverage the models learned from the dialogue data to customize the settings of the reinforcement learning for more efficient action space exploration.

D Model Implementation Details
Because it is small scale of automatic diagnosis datasets, Diaformer use a small transformer network (L=5, H=512, A=6), where L denotes the number of layers, H denotes the hidden size, A denotes the number of attention heads. The learning rate is set to 5e − 5 and the batch size is 16. For the inference step, we set ρ e =0.9 and set ρ p as 0.009 for MuZhi dataset, 0.012 for Dxy dataset and 0.01 for synthetic dataset. The hyper parameters of Diaformer GP T 2 and Diaformer U niLM are the same as Diaformer.

E Standard Errors of Result
Due to space limitation, we give the standard error of our models in each experiment, which is carried by 5 times. The standard errors are given in Table 7.

F Examples of Result
To help readers to comprehend the models output on the datasets, we give a success example and a failure example on Synthetic dataset, that shown in Table 8. In the task, models are asked to sequentially inquire the implicit symptoms based on explicit symptoms and then diagnose the disease by the explicit symptoms and the inquired implicit symptoms. Besides, in the outputs of Diaformer, we find that model tend to inquire more symptoms in the failure examples, while inquire less symptoms in success example, e.g. the examples in Table 8. It indicates the model has learned the diagnosis paradigm to some extent.  Table 7: Results with the standard error of our models. DAcc and SRec are the same as Table 3. Note that 20 maximum turn is the original setting of the task.