{"title": "Dialog-based Language Learning", "book": "Advances in Neural Information Processing Systems", "page_first": 829, "page_last": 837, "abstract": "A long-term goal of machine learning research is to build an intelligent dialog agent. Most research in natural language understanding has focused on learning from fixed training sets of labeled data, with supervision either at the word level (tagging, parsing tasks) or sentence level (question answering, machine translation). This kind of supervision is not realistic of how humans learn, where language is both learned by, and used for, communication. In this work, we study dialog-based language learning, where supervision is given naturally and implicitly in the response of the dialog partner during the conversation. We study this setup in two domains: the bAbI dataset of (Weston et al., 2015) and large-scale question answering from (Dodge et al., 2015). We evaluate a set of baseline learning strategies on these tasks, and show that a novel model incorporating predictive lookahead is a promising approach for learning from a teacher's response. In particular, a surprising result is that it can learn to answer questions correctly without any reward-based supervision at all.", "full_text": "Dialog-based Language Learning\n\nJason Weston\n\nFacebook AI Research,\n\nNew York.\n\njase@fb.com\n\nAbstract\n\nA long-term goal of machine learning research is to build an intelligent dialog\nagent. Most research in natural language understanding has focused on learning\nfrom \ufb01xed training sets of labeled data, with supervision either at the word level\n(tagging, parsing tasks) or sentence level (question answering, machine transla-\ntion). This kind of supervision is not realistic of how humans learn, where lan-\nguage is both learned by, and used for, communication. In this work, we study\ndialog-based language learning, where supervision is given naturally and implic-\nitly in the response of the dialog partner during the conversation. We study this\nsetup in two domains: the bAbI dataset of [23] and large-scale question answering\nfrom [3]. We evaluate a set of baseline learning strategies on these tasks, and show\nthat a novel model incorporating predictive lookahead is a promising approach for\nlearning from a teacher\u2019s response. In particular, a surprising result is that it can\nlearn to answer questions correctly without any reward-based supervision at all.\n\n1\n\nIntroduction\n\nMany of machine learning\u2019s successes have come from supervised learning, which typically involves\nemploying annotators to label large quantities of data per task. However, humans can learn by acting\nand learning from the consequences of (i.e, the feedback from) their actions. When humans act in\ndialogs (i.e., make speech utterances) the feedback is from other human\u2019s responses, which hence\ncontain very rich information. This is perhaps most pronounced in a student/teacher scenario where\nthe teacher provides positive feedback for successful communication and corrections for unsuccess-\nful ones [8, 22]. However, in general any reply from a dialog partner, teacher or not, is likely to\ncontain an informative training signal for learning how to use language in subsequent conversations.\nIn this paper we explore whether we can train machine learning models to learn from dialogs. The\nultimate goal is to be able to develop an intelligent dialog agent that can learn while conducting con-\nversations. To do that it needs to learn from feedback that is supplied as natural language. However,\nmost machine learning tasks in the natural language processing literature are not of this form: they\nare either hand labeled at the word level (part of speech tagging, named entity recognition), segment\n(chunking) or sentence level (question answering) by labelers. Subsequently, learning algorithms\nhave been developed to learn from that kind of supervision. We therefore need to develop evaluation\ndatasets for the dialog-based language learning setting, as well as developing models and algorithms\nable to learn in such a regime.\nThe contribution of the present work is thus:\n\nfeasibility of dialog-based language learning.\n\n\u2022 We introduce a set of tasks that model natural feedback from a teacher and hence assess the\n\u2022 We evaluate some baseline models on this data, comparing to standard supervised learning.\n\u2022 We introduce a novel forward prediction model, whereby the learner tries to predict the\nteacher\u2019s replies to its actions, yielding promising results, even with no reward signal at all.\n\n30th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain.\n\n\f2 Related Work\n\nIn human language learning the usefulness of social interaction and natural infant directed conversa-\ntions is emphasized, see e.g. the review paper [6], although the usefulness of feedback for learning\ngrammar is disputed [10]. Support for the usefulness of feedback is found however in second lan-\nguage learning [1] and learning by students [4, 8, 22].\nIn machine learning, one line of research has focused on supervised learning from dialogs using\nneural models [18, 3]. Question answering given either a database of knowledge [3] or short stories\n[23] can be considered as a simple case of dialog which is easy to evaluate. Those tasks typically\ndo not consider feedback. There is work on the the use of feedback and dialog for learning, notably\nfor collecting knowledge to answer questions [5, 14], the use of natural language instruction for\nlearning symbolic rules [7] and the use of binary feedback (rewards) for learning parsers [2].\nAnother setting which uses feedback is the setting of reinforcement learning, see e.g. [16] for a\nsummary of its use in dialog. However, those approaches often consider reward as the feedback\nmodel rather than exploiting the dialog feedback per se. Nevertheless, reinforcement learning ideas\nhave been used to good effect for other tasks as well, such as understanding text adventure games\n[12], machine translation and summarization [15]. Recently, [11] also proposed a reward-based\nlearning framework for learning how to learn.\nFinally, forward prediction models, which we make use of in this work, have been used for learn-\ning eye tracking [17], controlling robot arms [9] and vehicles [21], and action-conditional video\nprediction in atari games [13]. We are not aware of their use thus far for dialog.\n\n3 Dialog-Based Supervision Tasks\n\nDialog-based supervision comes in many forms. As far as we are aware it is a currently unsolved\nproblem which type of learning strategy will work in which setting. In this section we therefore\nidentify different modes of dialog-based supervision, and build a learning problem for each. The\ngoal is to then evaluate learners on each type of supervision.\nWe thus begin by selecting two existing datasets: (i) the single supporting fact problem from the\nbAbI datasets [23] which consists of short stories from a simulated world followed by questions;\nand (ii) the MovieQA dataset [3] which is a large-scale dataset (\u223c 100k questions over \u223c 75k\nentities) based on questions with answers in the open movie database (OMDb). For each dataset\nwe then consider ten modes of dialog-based supervision. The supervision modes are summarized\nin Fig. 1 using a snippet of the bAbI dataset as an example. The same setups are also used for\nMovieQA, some examples of which are given in Fig 2. We now describe the supervision setups.\nImitating an Expert Student\nIn Task 1 the dialogs take place between a teacher and an expert\nstudent who gives semantically coherent answers. Hence, the task is for the learner to imitate that\nexpert student, and become an expert themselves. For example, imagine the real-world scenario\nwhere a child observes their two parents talking to each other, it can learn but it is not actually\ntaking part in the conversation. Note that our main goal in this paper is to examine how a non-expert\ncan learn to improve its dialog skills while conversing. The rest of our tasks will hence concentrate\non that goal. This task can be seen as a natural baseline for the rest of our tasks given the same input\ndialogs and questions.\nPositive and Negative Feedback In Task 2, when the learner answers a question the teacher then\nreplies with either positive or negative feedback. In our experiments the subsequent responses are\nvariants of \u201cNo, that\u2019s incorrect\u201d or \u201cYes, that\u2019s right\u201d. In the datasets we build there are 6 templates\nfor positive feedback and 6 templates for negative feedback, e.g. \u201dSorry, that\u2019s not it.\u201d, \u201dWrong\u201d,\netc. To separate the notion of positive from negative (otherwise the signal is just words with no\nnotion that yes is better than no) we assume an additional external reward signal that is not part\nof the text. As shown in Fig. 1 Task 2, (+) denotes positive reward external to the dialog (e.g.\nfeedback provided by another medium, such as a nod of the head from the teacher). This is provided\nwith every positive response. Note the difference in supervision compared to Task 1: there every\nanswer is right and provides positive supervision. Here, only the answers the learner got correct\nhave positive supervision. This could clearly be a problem when the learner is unskilled: it will\nsupply incorrect answers and never (or hardly ever) receive positive responses.\n\n2\n\n\fFigure 1: Sample dialogs with differing supervision signals (tasks 1 to 10). In each case the same\nexample story is given for simplicity (shown \ufb01rst). Black text is spoken by the teacher, red text\ndenotes responses by the learner, blue text is provided by an expert student (which the learner can\nimitate), (+) denotes positive reward external to the dialog (e.g.\nfeedback provided by another\nmedium, such as a nod of the head from the teacher).\n\nStory Context (same for all examples below)\nMary went to the hallway.\nJohn moved to the bathroom.\nMary travelled to the kitchen.\n\nTask 1: Imitating an Expert Student\nWhere is Mary? A:kitchen\nWhere is John? A:bathroom\n\nTask 3: Answers Supplied by Teacher\nWhere is Mary? A:bedroom\nNo, the answer is kitchen.\nWhere is John? A:bathroom\nCorrect! (+)\nTask 5: Supporting Facts Supplied by Teacher\nWhere is Mary? A:kitchen\nYes, that\u2019s right! (+)\nWhere is John? A:hallway\nNo, because John moved to the bathroom.\nTask 7: No Feedback\nWhere is Mary? A:kitchen\nYes, that\u2019s right!\nWhere is John? A:bathroom\nYes, that\u2019s correct!\nTask 9: Asking For Corrections\nWhere is Mary? A:kitchen\nYes, that\u2019s right! (+)\nWhere is John? A:hallway\nNo, that\u2019s not right. A:Can you help me?\nBathroom.\n\nTask 2: Positive and Negative Feedback\nWhere is Mary? A:playground\nNo, that\u2019s incorrect.\nWhere is John? A:bathroom\nYes, that\u2019s right! (+)\nTask 4: Hints Supplied by Teacher\nWhere is Mary? A:bathroom\nNo, they are downstairs.\nWhere is John? A:kitchen\nNo, they are upstairs.\nTask 6: Partial Feedback\nWhere is Mary? A:kitchen\nYes, that\u2019s right!\nWhere is John? A:bathroom\nYes, that\u2019s correct! (+)\nTask 8: Imitation and Feedback Mixture\nWhere is Mary? A:kitchen\nWhere is John? A:bathroom\nThat\u2019s right! (+)\n\nTask 10: Asking For Supporting Facts\nWhere is Mary? A:kitchen\nYes, that\u2019s right! (+)\nWhere is John? A:hallway\nNo, that\u2019s not right. A:Can you help me?\nA relevant fact is John moved to the bathroom.\n\nFigure 2: Samples from the MovieQA dataset [3]. In our experiments we consider 10 different\nlanguage learning setups as described in Figure 1 and Sec. 3. The examples given here are for tasks\n2 and 3, questions are in black and answers in red, and (+) indicates receiving positive reward.\n\nTask 2: Positive and Negative Feedback\nWhat movies are about open source? Revolution OS\nThat\u2019s right! (+)\nWhat movies did Darren McGavin star in? Carmen\nSorry, that\u2019s not it.\nWho directed the \ufb01lm White Elephant? M. Curtiz\nNo, that is incorrect.\n\nTask 3: Answers Supplied by Teacher\nWhat \ufb01lms are about Hawaii? 50 First Dates\nCorrect! (+)\nWho acted in Licence to Kill? Billy Madison\nNo, the answer is Timothy Dalton.\nWhat genre is Saratoga Trunk in? Drama\nYes! (+)\n\nAnswers Supplied by Teacher\nIn Task 3 the teacher gives positive and negative feedback as in\nTask 2, however when the learner\u2019s answer is incorrect, the teacher also responds with the correction.\nFor example if \u201cwhere is Mary?\u201d is answered with the incorrect answer \u201cbedroom\u201d the teacher\nresponds \u201cNo, the answer is kitchen\u201d\u2019, see Fig. 1 Task 3. If the learner knows how to use this extra\ninformation, it effectively has as much supervision signal as with Task 1, and much more than for\nTask 2.\n\nHints Supplied by Teacher\nIn Task 4, the corrections provided by the teacher do not provide\nthe exact answer as in Task 3, but only a useful hint. This setting is meant to mimic the real life\noccurrence of being provided only partial information about what you did wrong. In our datasets\n\n3\n\n\fwe do this by providing the class of the correct answer, e.g. \u201cNo, they are downstairs\u201d if the answer\nshould be kitchen, or \u201cNo, it is a director\u201d for the question \u201cWho directed Monsters, Inc.?\u201d (using\nOMDB metadata). The supervision signal here is hence somewhere in between Task 2 and 3.\n\nSupporting Facts Supplied by Teacher\nIn Task 5, another way of providing partial supervision\nfor an incorrect answer is explored. Here, the teacher gives a reason (explanation) why the answer\nis wrong by referring to a known fact that supports the true answer that the incorrect answer may\ncontradict. For example \u201cNo, because John moved to the bathroom\u201d for an incorrect answer to\n\u201cWhere is John?\u201d, see Fig. 1 Task 5. This is related to what is termed strong supervision in [23]\nwhere supporting facts and answers are given for question answering tasks.\n\nPartial Feedback Task 6 considers the case where external rewards are only given some of (50%\nof) the time for correct answers, the setting is otherwise identical to Task 3. This attempts to mimic\nthe realistic situation of some learning being more closely supervised (a teacher rewarding you for\ngetting some answers right) whereas other dialogs have less supervision (no external rewards). The\ntask attempts to assess the impact of such partial supervision.\n\nNo Feedback In Task 7 external rewards are not given at all, only text, but is otherwise identical to\nTasks 3 and 6. This task explores whether it is actually possible to learn how to answer at all in such\na setting. We \ufb01nd in our experiments the answer is surprisingly yes, at least in some conditions.\n\nImitation and Feedback Mixture Task 8 combines Tasks 1 and 2. The goal is to see if a learner\ncan learn successfully from both forms of supervision at once. This mimics a child both observing\npairs of experts talking (Task 1) while also trying to talk (Task 2).\n\nAsking For Corrections Another natural way of collecting supervision is for the learner to ask\nquestions of the teacher about what it has done wrong. Task 9 tests one of the most simple instances,\nwhere asking \u201cCan you help me?\u201d when wrong obtains from the teacher the correct answer. This is\nthus related to the supervision in Task 3 except the learner must \ufb01rst ask for help in the dialog. This\nis potentially harder for a model as the relevant information is spread over a larger context.\n\nAsking for Supporting Facts Finally, in Task 10, a second less direct form of supervision for the\nlearner after asking for help is to receive a hint rather than the correct answer, such as \u201cA relevant\nfact is John moved to the bathroom\u201d when asking \u201cCan you help me?\u201d, see Fig. 1 Task 10. This is\nthus related to the supervision in Task 5 except the learner must request help.\n\nIn our experiments we constructed the ten supervision tasks for the two datasets which are all avail-\nable for download at http://fb.ai/babi. They were built in the following way: for each task we\nconsider a \ufb01xed policy1 for performing actions (answering questions) which gets questions correct\nwith probability \u03c0acc (i.e.\nthe chance of getting the red text correct in Figs. 1 and 2). We thus\ncan compare different learning algorithms for each task over different values of \u03c0acc (0.5, 0.1 and\n0.01). In all cases a training, validation and test set is provided. For the bAbI dataset this consists of\n1000, 100 and 1000 questions respectively per task, and for movieQA there are \u223c 96k, \u223c 10k and\n\u223c 10k respectively. MovieQA also includes a knowledge base (KB) of \u223c 85k facts from OMDB,\nthe memory network model we employ uses inverted index retrieval based on the question to form\nrelevant memories from this set, see [3] for more details. Note that because the policies are \ufb01xed the\nexperiments in this paper are not in a reinforcement learning setting.\n\n4 Learning Models\nOur main goal is to explore training strategies that can execute dialog-based language learning. To\nthis end we evaluate four possible strategies: imitation learning, reward-based imitation, forward\nprediction, and a combination of reward-based imitation and forward prediction. We will subse-\nquently describe each in turn.\nWe test all of these approaches with the same model architecture: an end-to-end memory network\n(MemN2N) [20]. Memory networks are a recently introduced model that have been shown to do\n1 Since the policy is \ufb01xed and actually does not depend on the model being learnt, one could also think of it\n\nas coming from another agent (or the same agent in the past) which in either case is an imperfect expert.\n\n4\n\n\fFigure 3: Architectures for (reward-based) imitation and forward prediction.\n\n(a) Model for (reward-based) imitation learning\n\n(b) Model for forward prediction\n\nwell on a number of text understanding tasks, including question answering, dialog [3] and language\nmodeling [20]. In particular, they outperform LSTMs and other baselines on the bAbI datasets [23]\nwhich we employ with dialog-based learning modi\ufb01cations in Sec. 3. They are hence a natural\nbaseline model for us to use in order to explore differing modes of learning in our setup. In the\nfollowing we will \ufb01rst review memory networks, detailing the explicit choices of architecture we\nmade, and then show how they can be modi\ufb01ed and applied to our setting of dialog-based language\nlearning.\n\nMemory Networks A high-level description of the memory network architecture we use is given\nin Fig. 3 (a). The input is the last utterance of the dialog, x, as well as a set of memories (context)\n(c1, . . . , cN ) which can encode both short-term memory, e.g. recent previous utterances and replies,\nand long-term memories, e.g. facts that could be useful for answering questions. The context inputs\nci are converted into vectors mi via embeddings and are stored in the memory. The goal is to\nproduce an output \u02c6a by processing the input x and using that to address and read from the memory,\nm, possibly multiple times, in order to form a coherent reply. In the \ufb01gure the memory is read twice,\nwhich is termed multiple \u201chops\u201d of attention.\nIn the \ufb01rst step, the input x is embedded using a matrix A of size d \u00d7 V where d is the embedding\ndimension and V is the size of the vocabulary, giving q = Ax, where the input x is as a bag-of-\nwords vector. Each memory ci is embedded using the same matrix, giving mi = Aci. The output of\naddressing and then reading from memory in the \ufb01rst hop is:\n\no1 =\n\np1\ni mi, p1\n\ni = Softmax(q(cid:62)mi).\n\n(cid:88)\n\ni\n\n(cid:88)\n\nHere, the match between the input and the memories is computed by taking the inner product fol-\nlowed by a softmax, yielding p1, giving a probability vector over the memories. The goal is to select\nmemories relevant to the last utterance x, i.e. the most relevant have large values of p1\ni . The output\nmemory representation o1 is then constructed using the weighted sum of memories, i.e. weighted\nby p1. The memory output is then added to the original input, u1 = R1(o1 + q), to form the new\nstate of the controller, where R1 is a d\u00d7 d rotation matrix2. The attention over the memory can then\nbe repeated using u1 instead of q as the addressing vector, yielding:\n\no2 =\n\np2\ni mi, p2\n\ni = Softmax(u(cid:62)\n\n1 mi),\n\ni\n\nThe controller state is updated again with u2 = R2(o2 + u1), where R2 is another d \u00d7 d matrix to\nbe learnt. In a two-hop model the \ufb01nal output is then de\ufb01ned as:\n2 Ay1, . . . , u(cid:62)\n\n\u02c6a = Softmax(u(cid:62)\n\n2 AyC)\n\n(1)\n\n2Optionally, different dictionaries can be used for inputs, memories and outputs instead of being shared.\n\n5\n\nMemory Module Controller module Input addressing read addressing read Internal state Vector (initially: query) Output\t\r \u00a0Memory vectors Supervision (direct or reward-based) mmqMemory Module Controller module Input Output Predict Response to Answer addressing read addressing read Internal state Vector (initially: query) addressing Candidate(Answers(read Memory vectors mmqqAnswer (action taken)\fwhere there are C candidate answers in y. In our experiments C is the set of actions that occur in\nthe training set for the bAbI tasks, and for MovieQA it is the set of words retrieved from the KB.\nHaving described the basic architecture, we now detail the possible training strategies we can employ\nfor our tasks.\n\nImitation Learning This approach involves simply imitating one of the speakers in observed di-\nalogs, which is essentially a supervised learning objective3. This is the setting that most existing di-\nalog learning, as well as question answer systems, employ for learning. Examples arrive as (x, c, a)\ntriples, where a is (assumed to be) a good response to the last utterance x given context c. In our\ncase, the whole memory network model de\ufb01ned above is trained using stochastic gradient descent\nby minimizing a standard cross-entropy loss between \u02c6a and the label a.\n\nReward-based Imitation If some actions are poor choices, then one does not want to repeat\nthem, that is we shouldn\u2019t treat them as a supervised objective. In our setting positive reward is\nonly obtained immediately after (some of) the correct actions, or else is zero. A simple strategy is\nthus to only apply imitation learning on the rewarded actions. The rest of the actions are simply\ndiscarded from the training set. This strategy is derived naturally as the degenerate case one obtains\nby applying policy gradient [24] in our setting where the policy is \ufb01xed (see end of Sec. 3). In more\ncomplex settings (i.e. where actions that are made lead to long-term changes in the environment and\ndelayed rewards) applying reinforcement learning algorithms would be necessary, e.g. one could\nstill use policy gradient to train the MemN2N but applied to the model\u2019s own policy.\n\nForward Prediction An alternative method of training is to perform forward prediction: the aim\nis, given an utterance x from speaker 1 and an answer a by speaker 2 (i.e., the learner), to predict\n\u00afx, the response to the answer from speaker 1. That is, in general to predict the changed state of the\nworld after action a, which in this case involves the new utterance \u00afx.\nTo learn from such data we propose the following modi\ufb01cation to memory networks, also shown\nin Fig. 3 (b): essentially we chop off the \ufb01nal output from the original network of Fig. 3 (a) and\nreplace it with some additional layers that compute the forward prediction. The \ufb01rst part of the\nnetwork remains exactly the same and only has access to input x and context c, just as before. The\ncomputation up to u2 = R2(o2 + u1) is thus exactly the same as before.\nAt this point we observe that the computation of the output in the original network, by scoring\ncandidate answers in eq.\n(1) looks similar to the addressing of memory. Our key idea is thus\nto perform another \u201chop\u201d of attention but over the candidate answers rather than the memories.\nCrucially, we also incorporate the information of which action (candidate) was actually selected in\nthe dialog (i.e. which one is a). After this \u201chop\u201d, the resulting state of the controller is then used to\ndo the forward prediction.\nConcretely, we compute:\n\no3 =\n\ni (Ayi + \u03b2\u2217[a = yi]),\np3\n\ni = Softmax(u(cid:62)\np3\n\n2 Ayi),\n\n(2)\n\nwhere \u03b2\u2217 is a d-dimensional vector, that is also learnt, that represents in the output o3 the action that\nwas actually selected. After obtaining o3, the forward prediction is then computed as:\n\n(cid:88)\n\ni\n\n\u02c6x = Softmax(u(cid:62)\n\n3 A\u00afx1, . . . , u(cid:62)\n\n3 A\u00afx \u00afC)\n\nwhere u3 = R3(o3 + u2). That is, it computes the scores of the possible responses to the answer a\nover \u00afC possible candidates. The mechanism in eq. (2) gives the model a way to compare the most\nlikely answers to x with the given answer a, which in terms of supervision we believe is critical. For\nexample in question answering if the given answer a is incorrect and the model can assign high pi to\nthe correct answer then the output o3 will contain a small amount of \u03b2\u2217; conversely, o3 has a large\namount of \u03b2\u2217 if a is correct. Thus, o3 informs the model of the likely response \u00afx from the teacher.\nTraining can then be performed using the cross-entropy loss between \u02c6x and the label \u00afx, similar to\nbefore. In the event of a large number of candidates \u00afC we subsample the negatives, always keeping\n\u00afx in the set. The set of answers y can also be similarly sampled, making the method highly scalable.\n\n3Imitation learning algorithms are not always strictly supervised algorithms, they can also depend on the\nagent\u2019s actions. That is not the setting we use here, where the task is to imitate one of the speakers in a dialog.\n\n6\n\n\fTable 1: Test accuracy (%) on the Single Supporting Fact bAbI dataset for various supervision\napproachess (training with 1000 examples on each) and different policies \u03c0acc. A task is successfully\npassed if \u2265 95% accuracy is obtained (shown in blue).\n\nMemN2N\nimitation\nlearning\n\nSupervision Type\n\n\u03c0acc =\n\n1 - Imitating an Expert Student\n2 - Positive and Negative Feedback\n3 - Answers Supplied by Teacher\n4 - Hints Supplied by Teacher\n5 - Supporting Facts Supplied by Teacher\n6 - Partial Feedback\n7 - No Feedback\n8 - Imitation + Feedback Mixture\n9 - Asking For Corrections\n10 - Asking For Supporting Facts\nNumber of completed tasks (\u2265 95%)\n\n0.5\n100\n79\n83\n85\n84\n90\n90\n90\n85\n86\n1\n\n0.1\n100\n28\n37\n23\n24\n22\n34\n89\n30\n25\n1\n\n0.01\n100\n21\n25\n22\n27\n22\n19\n82\n22\n26\n1\n\nMemN2N\n\nreward-based\nimitation (RBI)\n0.01\n0.5\n100\n100\n91\n99\n92\n99\n90\n99\n100\n83\n59\n98\n29\n20\n98\n99\n83\n99\n84\n99\n9\n2\n\n0.1\n100\n92\n96\n91\n96\n81\n22\n98\n89\n96\n5\n\nMemN2N\nforward\n\nprediction (FP)\n0.01\n0.5\n29\n23\n30\n93\n99\n99\n66\n97\n98\n100\n99\n100\n99\n100\n67\n28\n21\n23\n48\n23\n5\n4\n\n0.1\n30\n54\n96\n99\n99\n100\n98\n64\n15\n30\n5\n\nMemN2N\nRBI + FP\n\n0.1\n99\n92\n100\n100\n99\n100\n99\n98\n90\n95\n8\n\n0.01\n100\n96\n98\n100\n100\n99\n99\n97\n84\n91\n8\n\n0.5\n99\n99\n99\n99\n100\n99\n98\n99\n95\n97\n10\n\nA major bene\ufb01t of this particular architectural design for forward prediction is that after training\nwith the forward prediction criterion, at test time one can \u201cchop off\u201d the top again of the model to\nretrieve the original memory network model of Fig. 3 (a). One can thus use it to predict answers \u02c6a\ngiven only x and c. We can thus evaluate its performance directly for that goal as well.\nFinally, and importantly, if the answer to the response \u00afx carries pertinent supervision information\nfor choosing \u02c6a, as for example in many of the settings of Sec. 3 (and Fig. 1), then this will be\nbackpropagated through the model. This is simply not the case in the imitation, reward-shaping [19]\nor reward-based imitation learning strategies which concentrate on the x, a pairs.\n\nReward-based Imitation + Forward Prediction As our reward-based imitation learning uses the\narchitecture of Fig. 3 (a), and forward prediction uses the same architecture but with the additional\nlayers of Fig 3 (b), we can learn jointly with both strategies. One simply shares the weights across\nthe two networks, and performs gradient steps for both criteria, one of each type per action. The\nformer makes use of the reward signal \u2013 which when available is a very useful signal \u2013 but fails to\nuse potential supervision feedback in the subsequent utterances, as described above. It also effec-\ntively ignores dialogs carrying no reward. Forward prediction in contrast makes use of dialog-based\nfeedback and can train without any reward. On the other hand not using rewards when available is a\nserious handicap. Hence, the mixture of both strategies is a potentially powerful combination.\n\nTable 2: Test accuracy (%) on the MovieQA dataset dataset for various supervision approaches.\nNumbers in bold are the winners for that task and choice of \u03c0acc.\n\nMemN2N\nimitation\nlearning\n\nSupervision Type\n\n\u03c0acc =\n\n1 - Imitating an Expert Student\n2 - Positive and Negative Feedback\n3 - Answers Supplied by Teacher\n4 - Hints Supplied by Teacher\n5 - Supporting Facts Supplied by Teacher\n6 - Partial Feedback\n7 - No Feedback\n8 - Imitation + Feedback Mixture\n9 - Asking For Corrections\n10 - Asking For Supporting Facts\nMean Accuracy\n\n0.5\n80\n46\n48\n47\n47\n48\n51\n60\n48\n49\n52\n\n0.1\n80\n29\n29\n29\n28\n29\n29\n50\n29\n29\n36\n\n0.01\n80\n27\n26\n26\n26\n27\n27\n47\n27\n27\n34\n\nMemN2N\n\nreward-based\nimitation (RBI)\n0.01\n0.5\n80\n80\n52\n26\n27\n52\n28\n51\n26\n51\n24\n49\n22\n21\n51\n63\n26\n52\n27\n52\n52\n34\n\n0.1\n80\n32\n32\n32\n32\n32\n21\n53\n34\n34\n38\n\nMemN2N\nforward\n\nprediction (FP)\n0.01\n0.5\n24\n24\n48\n24\n58\n60\n42\n58\n33\n43\n58\n60\n58\n60\n23\n46\n44\n67\n35\n51\n52\n40\n\n0.1\n23\n34\n57\n58\n44\n58\n53\n31\n52\n44\n45\n\nMemN2N\nRBI + FP\n\n0.5\n77\n68\n69\n70\n66\n70\n61\n72\n68\n69\n69\n\n0.1\n77\n53\n65\n54\n53\n63\n56\n69\n52\n53\n60\n\n0.01\n77\n34\n62\n32\n40\n62\n50\n69\n39\n36\n50\n\n5 Experiments\n\nWe conducted experiments on the datasets described in Section 3. As described before, for each\ntask we consider a \ufb01xed policy for performing actions (answering questions) which gets questions\ncorrect with probability \u03c0acc. We can thus compare the different training strategies described in Sec.\n4 over each task for different values of \u03c0acc. Hyperparameters for all methods are optimized on the\nvalidation sets. A summary of the results is reported in Table 1 for the bAbI dataset and Table 2 for\nMovieQA. We observed the following results:\n\n7\n\n\fmore than just yes/no answers without extra information (Task 2).\n\nFP does not as it does not use this feedback.\n\nTable 1, but also degrades when they are too sparse e.g. for \u03c0acc = 0.01.\n\nanswers, e.g. for \u03c0acc < 0.5. For imitating an expert however (Task 1) it is hard to beat.\n\n\u2022 Imitation learning, ignoring rewards, is a poor learning strategy when imitating inaccurate\n\u2022 Reward-based imitation (RBI) performs better when rewards are available, particularly in\n\u2022 Forward prediction (FP) is more robust and has stable performance at different levels of\n\u03c0acc. However as it only predicts answers implicitly and does not make use of rewards\nit is outperformed by RBI on several tasks, notably Tasks 1 and 8 (because it cannot do\nsupervised learning) and Task 2 (because it does not take advantage of positive rewards).\n\u2022 FP makes use of dialog feedback in Tasks 3-5 whereas RBI does not. This explains why FP\ndoes better with useful feedback (Tasks 3-5) than without (Task 2), whereas RBI cannot.\n\u2022 Supplying full answers (Task 3) is more useful than hints (Task 4) but hints still help FP\n\u2022 When positive feedback is sometimes missing (Task 6) RBI suffers especially in Table 1.\n\u2022 One of the most surprising results of our experiments is that FP performs well overall,\ngiven that it does not use feedback, which we will attempt to explain subsequently. This is\nparticularly evident on Task 7 (no feedback) where RBI has no hope of succeeding as it has\nno positive examples. FP on the other hand learns adequately.\n\u2022 Tasks 9 and 10 are harder for FP as the question is not immediately before the feedback.\n\u2022 Combining RBI and FP ameliorates the failings of each, yielding the best overall results.\nOne of the most interesting aspects of our results is that FP works at all without any rewards. In\nTask 2 it does not even \u201cknow\u201d the difference between words like \u201cyes\u201d or \u201c\u2019correct\u201d vs. words\nlike \u201cwrong\u201d or \u201cincorrect\u201d, so why should it tend to predict actions that lead to a response like\n\u201cyes, that\u2019s right\u201d? This is because there is a natural coherence to predicting true answers that\nleads to greater accuracy in forward prediction. That is, you cannot predict a \u201cright\u201d or \u201cwrong\u201d\nresponse from the teacher if you don\u2019t know what the right answer is. In our experiments our policies\n\u03c0acc sample negative answers equally, which may make learning simpler. We thus conducted an\nexperiment on Task 2 (positive and negative feedback) of the bAbI dataset with a much more biased\npolicy: it is the same as \u03c0acc = 0.5 except when the policy predicts incorrectly there is probability\n0.5 of choosing a random guess as before, and 0.5 of choosing the \ufb01xed answer bathroom. In this\ncase the FP method obtains 68% accuracy showing the method still works in this regime, although\nnot as well as before.\n\n6 Conclusion\n\nWe have presented a set of evaluation datasets and models for dialog-based language learning. The\nultimate goal of this line of research is to move towards a learner capable of talking to humans, such\nthat humans are able to effectively teach it during dialog. We believe the dialog-based language\nlearning approach we described is a small step towards that goal.\nThis paper only studies some restricted types of feedback, namely positive feedback and corrections\nof various types. However, potentially any reply in a dialog can be seen as feedback, and should\nbe useful for learning.\nIt should be studied if forward prediction, and the other approaches we\ntried, work there too. Future work should also develop further evaluation methodologies to test\nhow the models we presented here, and new ones, work in those settings, e.g.\nin more complex\nsettings where actions that are made lead to long-term changes in the environment and delayed\nrewards, i.e. extending to the reinforcement learning setting, and to full language generation. Finally,\ndialog-based feedback could also be used as a medium to learn non-dialog based skills, e.g. natural\nlanguage dialog for completing visual or physical tasks.\n\nAcknowledgments\n\nWe thank Arthur Szlam, Y-Lan Boureau, Marc\u2019Aurelio Ranzato, Ronan Collobert, Michael Auli,\nDavid Grangier, Alexander Miller, Sumit Chopra, Antoine Bordes and Leon Bottou for helpful\ndiscussions and feedback, and the Facebook AI Research team in general for supporting this work.\n\n8\n\n\fReferences\n[1] M. A. Bassiri.\n\nInteractional feedback and the impact of attitude and motivation on noticing l2 form.\n\nEnglish Language and Literature Studies, 1(2):61, 2011.\n\n[2] J. Clarke, D. Goldwasser, M.-W. Chang, and D. Roth. Driving semantic parsing from the world\u2019s response.\n\nIn Proceedings of computational natural language learning, 2010.\n\n[3] J. Dodge, A. Gane, X. Zhang, A. Bordes, S. Chopra, A. Miller, A. Szlam, and J. Weston. Evaluating\n\nprerequisite qualities for learning end-to-end dialog systems. arXiv preprint arXiv:1511.06931, 2015.\n\n[4] R. Higgins, P. Hartley, and A. Skelton. The conscientious consumer: Reconsidering the role of assessment\n\nfeedback in student learning. Studies in higher education, 27(1):53\u201364, 2002.\n\n[5] B. Hixon, P. Clark, and H. Hajishirzi. Learning knowledge graphs for question answering through con-\n\nversational dialog. In ACL, 2015.\n\n[6] P. K. Kuhl. Early language acquisition: cracking the speech code. Nature reviews neuroscience, 5(11):\n\n831\u2013843, 2004.\n\n[7] G. Kuhlmann, P. Stone, R. Mooney, and J. Shavlik. Guiding a reinforcement learner with natural language\n\nadvice: Initial results in robocup soccer. In AAAI-2004 workshop on supervisory control, 2004.\n\n[8] A. S. Latham. Learning through feedback. Educational Leadership, 54(8):86\u201387, 1997.\n\n[9] I. Lenz, R. Knepper, and A. Saxena. Deepmpc: Learning deep latent features for model predictive control.\n\nIn Robotics Science and Systems (RSS), 2015.\n\n[10] G. F. Marcus. Negative evidence in language acquisition. Cognition, 46(1):53\u201385, 1993.\n\n[11] T. Mikolov, A. Joulin, and M. Baroni. A roadmap towards machine intelligence.\n\narXiv:1511.08130, 2015.\n\narXiv preprint\n\n[12] K. Narasimhan, T. Kulkarni, and R. Barzilay. Language understanding for text-based games using deep\n\nreinforcement learning. arXiv preprint arXiv:1506.08941, 2015.\n\n[13] J. Oh, X. Guo, H. Lee, R. L. Lewis, and S. Singh. Action-conditional video prediction using deep networks\n\nin atari games. In Advances in Neural Information Processing Systems, pages 2845\u20132853, 2015.\n\n[14] A. Pappu and A. Rudnicky. Predicting tasks in goal-oriented spoken dialog systems using semantic\n\nknowledge bases. In Proceedings of the SIGDIAL, pages 242\u2013250, 2013.\n\n[15] M. Ranzato, S. Chopra, M. Auli, and W. Zaremba. Sequence level training with recurrent neural networks.\n\narXiv preprint arXiv:1511.06732, 2015.\n\n[16] V. Rieser and O. Lemon. Reinforcement learning for adaptive dialogue systems. Springer Science &\n\nBusiness Media, 2011.\n\n[17] J. Schmidhuber and R. Huber. Learning to generate arti\ufb01cial fovea trajectories for target detection. Inter-\n\nnational Journal of Neural Systems, 2(01n02):125\u2013134, 1991.\n\n[18] A. Sordoni, M. Galley, M. Auli, C. Brockett, Y. Ji, M. Mitchell, J.-Y. Nie, J. Gao, and B. Dolan. A neural\n\nnetwork approach to context-sensitive generation of conversational responses. NAACL, 2015.\n\n[19] P.-H. Su, D. Vandyke, M. Gasic, N. Mrksic, T.-H. Wen, and S. Young. Reward shaping with recurrent\nneural networks for speeding up on-line policy learning in spoken dialogue systems. arXiv preprint\narXiv:1508.03391, 2015.\n\n[20] S. Sukhbaatar, J. Weston, R. Fergus, et al. End-to-end memory networks. In Advances in Neural Infor-\n\nmation Processing Systems, pages 2431\u20132439, 2015.\n\n[21] G. Wayne and L. Abbott. Hierarchical control using networks trained with higher-level forward models.\n\nNeural computation, 2014.\n\n[22] M. G. Werts, M. Wolery, A. Holcombe, and D. L. Gast. Instructive feedback: Review of parameters and\n\neffects. Journal of Behavioral Education, 5(1):55\u201375, 1995.\n\n[23] J. Weston, A. Bordes, S. Chopra, and T. Mikolov. Towards ai-complete question answering: a set of\n\nprerequisite toy tasks. arXiv preprint arXiv:1502.05698, 2015.\n\n[24] R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning.\n\nMachine learning, 8(3-4):229\u2013256, 1992.\n\n9\n\n\f", "award": [], "sourceid": 521, "authors": [{"given_name": "Jason", "family_name": "Weston", "institution": "Facebook AI Research"}]}