{'fraction_non_alphanumeric': 0.03269082498072475, 'fraction_numerical': 0.006939090208172706, 'mean_word_length': 4.462099943851769, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Improving Skin Condition Classification with a Visual Symptom Checker Trained using Reinforcement Learning

We present a visual symptom checker that combines a pre-trained Convolutional Neural Network (CNN) with a Reinforcement Learning (RL) agent as a Question Answering (QA) model. This method increases the classification confidence and accuracy of the visual symptom checker, and decreases the average number of questions asked to narrow down the differential diagnosis. A Deep Q-Network (DQN)-based RL agent learns how to ask the patient about the presence of symptoms in order to maximize the probability of correctly identifying the underlying condition. The RL agent uses the visual information provided by CNN in addition to the answers to the asked questions to guide the QA system. We demonstrate that the RL-based approach increases the accuracy more than 20% compared to the CNN-only approach, which only uses the visual information to predict the condition. Moreover, the increased accuracy is up to 10% compared to the approach that uses the visual information provided by CNN along with a conventional decision tree-based QA system. We finally show that the RL-based approach not only outperforms the decision tree-based approach, but also narrows down the diagnosis faster in terms of the average number of asked questions.


Introduction
Doctors often ask their patients a series of questions in order to narrow down the set of plausible conditions matching the observed symptoms. By asking relevant questions, they can diagnose their patients more efficiently. We are interested in designing an automatic system that diagnoses patients based on an image of the present symptom and a sequence of questions and their corresponding answers.
Given an initial set of symptoms, a QA system such as a symptom checker enables the emulation of this conventional approach by asking the relevant questions in order to refine the differential diagnosis. Some work has been done to formulate the symptom checker using Bayesian networks [10] and recurrent neural networks [3]. All these symptom checkers, however, rely only on clinical descriptions without leveraging the visual information usually available in several domains such as dermatology and radiology.
The recent work of Akrout et al. [1] makes use of the visual information by combining a CNN and a decision tree QA model to improve the skin condition classification task. Their proposed decision tree-based approach picks the best symptom to ask by maximizing the information gain IG(S, C) between symptoms S and conditions C. Since decision trees are learned by heuristic methods such as greedy search, they only consider immediate information gain at the current splitting node and often result in sub-optimal solutions in a constrained search space. The RL framework overcomes this problem by searching for splitting strategies in the global search space based on the evaluation of long-term payoff. We note that designing QA systems has already been successfully formulated as an RL task for Query Reformulation [5] and search engine querying [2] and more recently to feature extraction [11] using DQN [4]. To the best of our knowledge, no prior RL-based QA model has been applied for differential diagnoses problems.
In this work, we extend the study in [1] by formulating the symptom checking problem as a Markov Decision Process (MDP). We show that the RL agent learns using DQN to ask the best symptom and outperforms the decision tree approach while asking fewer questions. This paper makes the following contributions: -We formulate the visual symptom checking problem as an MDP problem, and propose an RL-based approach to solve it. -We show a significant accuracy improvement compared with the decision tree approach. -We illustrate how the RL approach not only significantly improves the accuracy of the skin disease classification compared to the decision tree approach but also decreases the average number of asked questions.

A Visual Symptom Checker as an MDP
A visual symptom checker interacts with the patient by asking symptoms and receiving the patient's answer. In this section, we show how the visual symptom checking problem can be formulated as an MDP [6,8].

MDP Definition
An MDP is a tuple (X , A, P, R, γ) where X is the set of states, A is a finite set of actions from which the agent can choose, P : X × A × X → [0, 1] is a transition probability in which P(x, a, x ) defines the probability of observing state x after executing action a in the state x, R : X × A → R is the expected reward after being in state x and taking action a, and γ ∈ [0, 1) is the discount factor. An RL agent continually makes value judgments in order to select the right action. The action selection mechanism of an RL agent is called its policy, which is a mapping π : X → A from the state space to the action space. Given a policy π, the action-value function Q π : The action-value function Q π at state-action pair (x, a) is the expected discounted reward that the RL agent receives if it starts from state x at time t, chooses action a, and afterwards selects actions according to policy π, i.e., A t+k = π(X t+k ) for k = 1, 2, . . . . The goal of an RL agent is to find a policy π that maximizes this value for all state-action pairs. Such a policy is called the optimal policy π * , and its corresponding action-value function is called the optimal action-value function Q * . If the agent has access to the optimal action-value function, the optimal policy π * can be computed as π * (x) ← arg max a∈A Q * (x, a). Many RL algorithms, including Deep Q-Network (DQN) [4] that we briefly describe in Section 2.3, try to estimate Q * . For more information, refer to [8].
We show how the MDP framework can be applied to the symptom checking problem in two steps: we first describe in Section 2.2 how X , A, P and R are designed for the symptom checker. Afterwards, we describe in Section 2.3 the agent's algorithm that improves the skin condition classification.

MDP Formulation
Let S be the symptom space. Some examples are rash, redness and pigmented lesion. Let C be the condition space. Some examples are cellulitis, psoriasis and melanoma, to name a few. We denote any symptom by s i and any condition by c j where 1 ≤ i ≤ |S|, 1 ≤ j ≤ |C|, and | · | refers to the cardinality operator. Let S a ⊂ S be the set of asked symptoms and s a any asked symptom among S a .    answers is set to 0, (b) the environment state gets updated by putting +1 in the "itching" position after answering "yes" to the asked symptom "itching".
A symptom checker agent learns which symptom s ∈ S to ask for an environment of a simulated patient having a condition c * ∈ C. Figure 1 summarizes pictorially how a symptom checker can be viewed as an RL agent finding an optimal policy for an MDP. We define the MDP parameters as follows: • Action space A: It corresponds to the symptom space S where each action a ∈ A is a possible symptom s to ask. • State space X : Each state x ∈ X corresponds to a patient state. A state x is the concatenation of the patient's history of all answers and the pretrained CNN's output probabilities of the patient's image (see Fig 2). • Transition probability P: For an asked question, +1 or -1 are put at the question's position if the answer is respectively "yes" or "no" (see Fig 3). For each new random patient simulation, its condition c i = c * is known. The simulated patient answers the asked symptom s a by referring to the health knowledge matrix M [7] computed from electronic medical records of 273,174 patients. It represents a matrix of condition-symptom relationships where each cell (i, j) represents the conditional probability p(s j |c i ) of a symptom s j given a condition c i . The simulated environment answers s a by generating a random number k from a uniform distribution between 0 and 1. If k ≤ p(s a |c * ), the answer is "yes", and "no" otherwise.
• Reward R: This function is equal to p(c * |S a ), the probability of correct condition c * given the asked questions S a . This design of the reward function is mainly based on the intuition that we want to keep the probability of the true condition of the simulated patient as high as possible. At each interaction, when the simulated patient answers s a , we update the conditional probability p(c i |S a ) ∀i using the Bayes rule as described in the Appendix A.
We believe other reasonable reward functions can be defined.

Deep Q-Learning
The visual symptom checker agent learns an estimate of the optimal actionvalue function Q * . Mnih et al. [4] introduced the Deep Q-Network (DQN), a CNN approximating the action-value function Q(x, a; ω) ≈ Q * (x, a), where ω represents the network's parameters. The DQN squared error loss function L(ω) is defined as: where x is the current state, x is the next-state, and γ is the discount factor. DQN uses Q target (ω − ), a fixed version of Q net (ω) with parameters ω − that are periodically updated in order to stabilize rapid policy changes, due to the quick variations in Q-values. Another trick used in [4] to avoid divergence because of successive data sampling is the experience replay buffer. It stores transitions of (x, a, r, x ) and is randomly sampled to create the mini-batches used for training.

Classification of Skin Conditions
In this section, we describe both the training and inference setting of the visual symptom checker. In our setting, we consider |S| = 330, |C| = 9, and the discount factor γ = 0.99.

The Training Step
We initially train an Inception-v3 network [9] on a dataset consisting of 5,841 images that are equally divided into 9 skin conditions: atopic dermatitis, lupus, shingles, cellulitis, chickenpox, hives, psoriasis, gout and melanoma. For a given patient image, we run inference on the pretrained CNN to get the CNN's output probabilities which, concatenated with the history of answers initialized at zero, form the environment's state of the simulated patient. The RL agent interacts with this environment until either a predefined maximum number of questions to ask is reached or one of the 9 condition probabilities p(c i |S a ) exceeds a predefined threshold.

The Inference
Step We run inference on our visual symptom checker using annotated test images (i.e. a labeled image with a paragraph describing the patient's symptoms). The image description, commonly called a vignette, does not include the true condition of the patient, but rather their symptoms. We answer the asked questions of the trained RL agent with "yes" if the symptom is present in the vignette and with "no" otherwise. The Appendix B provides 9 examples of vignettes, one for each condition.

Experiments and Results
All the experiments have been performed within the simulated patient environment described in Section 2.2. We fix the maximum number of questions to ask at 10 and the confidence threshold to 95% respectively.

Architecture of the Q-network (DQN)
The Q network has 5 layers: an input layer of 339 units (a vector of |S| = 330 elements representing the history of answers concatenated with a vector of |C| = 9 probabilities from the CNN output), 3 hidden fully connected layers consisting of 350 ReLu units and an output layer composed of |S| = 330 linear units.

Evaluation
We have evaluated our visual symptom checker using 600 annotated test images. The evaluation metric chosen to compare the system before and after the visual symptom checker is the top-K accuracy, which is well known in medical imaging scenarios to successfully assess the differential diagnosis cases. Additionally, we compare the average number of asked questions between the RL approach and the decision tree approach in [1]. Table 1 shows how the RL agent that learns from a simulated patient environment increases the rank and the probability of the correct condition compared to the classification performance of the CNN alone. Here  The results of the decision tree QA model are those of [1]. For the RL approach, 5 independent agents were trained on one of the 5 CNNs corresponding to a specific cross-validation fold. Reported values are the mean and standard deviation of the evaluation accuracy across all the 5 environments. These results demonstrate that the proposed RL approach can significantly improve the classification performance across a multiclass classification task and outperform the decision tree approach used in [1].

Top-K accuracy
By sampling many times, the RL agent learns asymptotically, as the number of simulated patients increases, the conditional probabilities between symptoms and conditions. For instance, given two specific symptom s 0 and condition c 0 , if p(s 0 |c 0 ) = 60%, the environment will answer 60% of the times "yes". Therefore, 60% of the data collected by the agent from a simulated patient with a condition c 0 will have s 0 included. The fact that the training data is balanced proportionally to the symptoms' occurrences allows the agent to learn the probabilistic relationships which is one reason explaining why the RL approach can outperform greedy approaches like decision trees. Another reason is that RL sequentially backtracks rewards unlike decision trees which search through the space of possible branches with no backtracking.

Average Number of Asked Questions
While the reported number of Table 1 are obtained after a maximum number of 10 questions, we examined the evolution of the top-K accuracy after answering each asked question by both the proposed RL-based QA model and the decision tree model studied in [1]. Figure 4 shows the evolution of the top-K accuracy with the number of asked questions for both approaches. One can see that both top-K accuracy and top-K rate of the RL approach are higher compared to the decision tree approach.
Additionally, the RL agent achieves its highest performance with a fewer questions compared to the decision tree approach where top-K accuracy keeps increasing with a lower top-K accuracy rate as the number of questions increases. This suggests that the RL agent succeeds not only to narrow down the differential diagnosis with a better performance, but also quickly as compared with the decision tree approach.

Conclusion
The results of the visual symptom checker formulated as an RL problem demonstrate that the proposed methodology can effectively not only learn the probabilistic relationships between symptoms and conditions, but also find the most relevant questions to ask for improving the final predictions across the nine skin conditions considered. Future work remains as to extend the symptom checker to support more conditions and symptoms and to further look into ways to evaluate its performance against doctors and other diagnostic systems.

Bibliography
Appendix A Details of updating the posterior p(c i |S a ) ∀i Using the vector notation, we denote p(c i ) ∀i by p(C). As described in Section 2.2, the simulated patient uses a matrix M representing the conditional condition-symptom relationships p(S|C). Given the initial p(C) provided by the CNN's output probabilities, the updated p(C) after asking a set of symptoms S a is p(C|S a ) that can be computed using the Bayes rule: The likelihood p(S a |C) is a matrix computed using on both the M representing p(S|C) and the simulated patient's answers. If the simulated agents answers "yes" to the asked symptom s a , we append the column p(s a |C) to p(S a |C). If the answer is "no", we append 1 − p(s a |C) to p(S a |C).

Condition
Vignette Image Chicken pox A 3-year-old boy is noted by his parents to have two small red papules on his abdomen just before bedtime. The next morning he feels hot and has developed multiple lesions over his trunk, some of which look like blisters. Later that day he develops vomiting and diarrhoea and his parents take him to the accident and emergency department. The child has previously been well and attends nursery four days per week. No other family members are obviously affected. Examination The child feels hot and looks slightly miserable, he does not obviously look dehydrated. There are multiple red papules and vesicles over the childs abdomen, back and around his neck. Most of the vesicles are intact, however some have ruptured and are starting to crust. Mucous membranes are normal.

Melanoma
A 27-year-old woman with a pigmented skin lesion on her left anterior shoulder. She was originally seen four months earlier, and the lesion was noted to be lonely. She had been given a follow-up appointment to be reviewed in six months. She made the decision to present earlier because the lesion had changed in color. This change is concerning for her. There is diffuse warmth, mild erythema, and pitting edema over the dorsum of both hands. There is tenderness and limited hand grip bilaterally. There are multiple uric acid tophi located on the helix of his right ear.

Condition Vignette Image
Cellulitis A 24-year-old male with past medical history of staphylococcal and streptococcal skin infections and abscesses presented with an open lesion on his right posterior ankle accompanied by right leg erythema and edema. The patient reported that he had formed a blister in this same right heel 7 months prior. That blister healed, however the overlying skin did not fully return to normal. The patient said, there was always a dark spot there after that. Five days prior to this visit, his right foot and leg started swelling and becoming red, and the site of the prior blister opened up. The day prior to this visit, a black spot appeared adjacent to the open lesion, which then also broke open, forming a large wound. He denied fever or chills, and felt otherwise well. The patient reported that the area had been much more painful 23 days prior to this visit, at which time he could not put on a shoe due to the pain and swelling. The pain had improved by the time of visit. Vitals were normal. On exam, there was a 3.5 cm 2 cm weeping ulcer on the lateral Achilles area of his right heel, with surrounding blanchable erythema on his foot and ankle that extended to the mid lower leg. There was 1+ edema on the right lower leg.
Hives A 29-year-old female with no past medical history presented with a diffuse and itchy rash. The rash was most severe on her anterior and posterior lower extremities bilaterally, and less prominent on her lower abdomen and lower forearms. The rash started 2 days prior to her visit and got progressively worse over that time. She felt well otherwise. She reported no new lotions, detergents, or clothes. She got a flu shot 2 days prior (which she had received in the past without incident and she denied an egg allergy). The rash started later in the afternoon after she got her flu shot. The rash on her hands worsened with contact with hot water. The day before and the day of the visit, and the rash continued to progress and to become more itchy.
Atopic-dermatitis An 11-year-old girl presents to the pediatric dermatology clinic as a referral for a persistently itchy rash that involves the face, neck, trunk and extremities. Her mother states that the girl scratches rigorously at night and is unable to concentrate during school due to pruritus. She has a past medical history of moderate intermittent asthma and allergic rhinitis, and her mother reports a strong history of asthma on her side of the family without any dermatologic issues.