{'fraction_non_alphanumeric': 0.031747058062847534, 'fraction_numerical': 0.010830208198629251, 'mean_word_length': 4.985487616099071, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Joint Speaker-Listener-Reinforcer Model for Referring Expressions

Referring expressions are natural language constructions used to identify particular objects within a scene. In this paper, we propose a unified framework for the tasks of referring expression comprehension and generation. Our model is composed of three modules: speaker, listener, and reinforcer. The speaker generates referring expressions, the listener comprehends referring expressions, and the reinforcer introduces a reward function to guide sampling of more discriminative expressions. The listener-speaker modules are trained jointly in an end-to-end learning framework, allowing the modules to be aware of one another during learning while also benefiting from the discriminative reinforcer's feedback. We demonstrate that this unified framework and training achieves state-of-the-art results for both comprehension and generation on three referring expression datasets. Project and demo page: https://vision.cs.unc.edu/refer


Introduction
People often use referring expressions in their everyday discourse to unambiguously identify or indicate particular objects within their physical environment. For example, one might point out a person in the crowd by referring to them as "the man in the blue shirt" or you might ask someone to "pass me the red pen on the table." In both of these examples, we have a pragmatic interaction between two people (or between a person and an intelligent agent such as a robot). First, we have a speaker who must generate an expression given a target object and its surrounding world context. Second, we have a listener who must interpret and comprehend the expression and map it to an object in the environment. Therefore, in this paper we propose an endto-end trained listener-speaker framework that models these behaviors jointly.
In addition to the listener and speaker, we also introduce a new reinforcer module that learns a discriminative reward model to help generate less ambiguous expressions (expressions that apply to the target object but not to other objects in the image). This goal corresponds to the Gricean Maxim [7] of manner, where one tries to be as clear, brief, and orderly as possible while avoiding obscurity and ambiguity. Avoiding ambiguity is important because the generated expression should be easily and uniquely mapped to the target object. For example, if there were two pens on the table one "long and red" and the other "short and red", asking for the "red pen" would be ambiguous while asking for the "long pen" would be better. The reinforcer module is incorporated using reinforcement learning, inspired by behavioral psychology that says that agents operating in an environment should take actions that maximize their expected cumulative reward. In our case, the reward takes the form of a discriminative classifier trained to reward the speaker for generating less ambiguous expressions.
Within the realm of referring expressions, there are two tasks that can be computationally modeled, mimicking the listener and speaker roles. Referring Expression Generation (speaker) requires an algorithm to generate a referring expression for a given target object in a visual scene, as shown in Fig. 1. Referring Expression Comprehension (listener) requires an algorithm to localize the object/region in an image described by a given referring expression, as shown in Fig. 2.
The Referring Expression Generation (REG) task has been studied since the 1970s [28]. Many of the early works in this space focused on relatively limited datasets, using synthesized images of objects in artificial scenes or limited sets of real-world objects in simplified environments [19,6,14]. Recently, the research focus has shifted to more complex natural image datasets and has expanded to include the Referring Expression Comprehension task [12,18,29] as well as to real-world interactions with robotics [4,3]. One reason this has become feasible is that several large-scale REG datasets have been collected at a scale where deep learning models can be applied. Kazemzadeh et al [12] introduced the first large-scale REG dataset on 20k natural images from the ImageClef dataset [8] using an interactive game, later expanded to im-  ages from the MSCOCO dataset [16] in Yu et al [29]. In addition, Mao et al [18] collected Google's REG dataset, also based on MSCOCO images, but in a non-interactive setting, resulting in more complex lengthy expressions. In this paper, we focus our evaluations on the three recent datasets collected on MSCOCO images [29,18].
Recent neural approaches to the referring expression generation and comprehension tasks can be roughly split into two types. The first type uses a CNN-LSTM encoderdecoder generative model [24] to generate (decode) sentences given the encoded target object. With careful design of the visual representation of target object, this model can generate unambiguous expressions [18,29]. Here, the CNN-LSTM models P (r|o), where r is the referring expression and o is the target object, which can be easily converted to P (o|r) via Bayes' rule and used to address the comprehension task [9,18,29,20] by selecting the o with the largest posterior probability. The second type of approach uses a joint-embedding model that projects both a visual representation of the target object and a semantic representation of the expression into a common space and learns a distance metric. Generation and comprehension can be performed by embedding a target object (or expression) into the embedding space and retrieving the closest expression (or object) in this space. This type of approach typically achieves better comprehension performance than the CNN-LSTM model as in [22,25], but previously was only applied to the referring expression comprehension task. Recent work [1] has also used both an encoder-decoder model (speaker) and an embedding model (listener) for referring expression generation in abstract images, where the offline listener reranks the speaker's output.
In this paper, we propose a unified model that jointly learns both the CNN-LSTM speaker and embedding-based listener models, for both the generation and comprehension tasks. Additionally, we add a discriminative reward-based reinforcer to guide the sampling of more discriminative expressions and further improve our final system. Instead of working independently, we let the speaker, listener, and reinforcer interact with each other, resulting in improved performance on both generation and comprehension tasks. Results evaluated on three standard, large-scale datasets verify that our proposed listener-speaker-reinforcer model significantly outperforms the state-of-the-art on both the comprehension task (Tables 1 and 2) and the generation task (evaluated using human judgements in Table 4, and automatic metrics in Table 3).

Related work
Recent years have witnessed a rise in multimodal research related to vision and language. Given the individual success in each area, and the need for models with more advanced cognition capabilities, several tasks have emerged as evaluation applications, including image captioning, visual question answering, and referring expression generation/comprehension. Image Captioning: The aim of image captioning is to generate a sentence describing the general content of an image. Most recent approaches use deep learning to address this problem. Perhaps the most common architecture is a CNN-LSTM model [24], which generates a sentence conditioned on visual information from the image. One paper related to our work is gLSTM [10] which uses CCA semantics to guide the caption generation. A further step beyond image captioning is to locate the regions being described in captions [26,21,25]. The Visual Genome [15] collected captions for dense regions in an image that have been used for dense-captioning tasks [11]. While there have been many approaches achieving quite good results on captioning, one challenge has been that the caption that a system should output for an image is very task dependent. Therefore, there has been a movement toward more focused tasks, such as visual question answering [2], and referring expression generation and comprehension which involve specific regions/objects within an image (discussed below). Referring Expression Datasets: REG has been studied for many years [28,14,19] in linguistics and natural language processing, but mainly focused on small or artificial datasets. In 2014, Kazemzadeh et al [12] introduced the first large-scale dataset RefCLEF using 20,000 real-world natural images [8]. This dataset was collected in a two-player game, where the first player writes a referring expression given an indicated target object. The second player is shown only the image and expression and has to click on the correct object described by the expression. If the click lies within the target object region, both sides get points and their roles switch. Using the same game interace, the authors further collected RefCOCO and RefCOCO+ datasets on MSCOCO images [29]. The RefCOCO and RefCOCO+ datasets each contain 50,000 referred objects with 3 referring expressions on average. The main difference between RefCOCO and RefCOCO+ is that in RefCOCO+, players were forbidden from using absolute location words, e.g. left dog, therefore focusing the referring expression to purely appearance-based descriptions. In addition, Mao et al [18] also collected a referring expression dataset -RefCOCOg, using MSCOCO images, but in a non-interactive framework. These expressions are more similar to the MSCOCO captions in that they are longer and more complex as their was no time constraint in the non-interactive data collection setting. This dataset, RefCOCOg, has 96,654 objects with 1.3 expressions per object on average.

Referring Expression Comprehension and Generation:
Referring expressions are associated with two tasks, comprehension and generation. The comprehension task re-quires a system to select the region being described by a given referring expression. To address this problem, [18,29,9] model P (r|o) and looks for the object o maximizing the probability. People also try modeling P (o, r) directly using embedding model [22,25], which learns to minimize the distance between paired object and sentence in the embedding space. The generation task asks a system to compose an expression for a specified object within an image. While some previous work used rule-based approaches to generate expressions with fixed grammar pattern [19,5,12], recent work has followed the CNN-LSTM structure to generate expressions [18,29]. Most related work to us is [1] which uses a speaker model for referring expression generation in abstract images, then uses an offline listener to rerank the speaker's output. On contrary, our model jointly trains the speaker and listener modules and has an additional reinforcer module to encourage unambiguous speaker expression generation. Moreover, we use our models for both generation and comprehension tasks on three datasets.

Model
Our model is composed of three modules: speaker (Sec 3.1), listener (Sec 3.2), and reinforcer (Sec 3.3). During training, the speaker and listener are trained jointly so that they can benefit from each other and from the reinforcer. Because the reward function for the reinforcer is not differentiable, it is incorporated during training using a reinforcement learning policy gradient algorithm.

Speaker
For our speaker module, we follow the previous state-ofthe-art [18,29], and use a CNN-LSTM framework. Here, a pre-trained CNN model is used to define a visual representation for the target object and other visual context. Then, a Long-short term memory (LSTM) is used to generate the most likely expression given the visual representation.
Because of the improved quantitative performance over [18], we use the visual comparison model of [29] as our speaker (to encode the target object). Here, the visual representation includes the target object, context, location/size features, and two visual comparison features. Specifically, the target object representation o i is modeled as the fc7 features from a pre-trained VGG network [23]. Global context, g i , is modeled as features extracted from the VGG-fc7 layer for the entire image. The location/size representation, l i , for the target object is modeled as a 5dimensional vector, encoding the x and y locations of the top left and bottom right corners of the target object bounding box, as well as the bounding box size with respect to the image, i.e., l i = [ x tl W , y tl H , x br W , y br H , w·h W ·H ]. As referring expressions often relate an object to other objects of the same type within the image ("the red ball" vs "the blue ball" or "the larger elephant"), comparisons tend Listener is a joint-embedding model learned to minimize the distance between paired object and expression representations.
In addition, a Reinforcer module helps improve the speaker by sampling more discriminative (less ambiguous) expressions for training. The model is jointly trained with 3 loss functions -generation loss, embedding loss, and reward loss, thereby improving performance on both the comprehension and generation tasks.
to be quite important for differentiation. The comparison features are composed of two parts: a) appearance similarity where n is the number of objects chosen for comparisons, b) location and size similarity - hj wihi ]. The final visual representation for the target object is then a concatenation of the above features followed by a fully-connected layer fusing them together, This joint feature is then fed into the LSTM for referring expression generation. During training we minimize the negative log-likelihood: Note that the speaker can be modeled using any form of CNN-LSTM structure.
In [18], Mao proposed to add a Maximum Mutual Information (MMI) constraint encouraging the generated expression to describe the target object better than the other objects within the image (i.e., a ranking loss on objects). We generalize this idea to incorporate two triplet hinge losses composed of a positive match (r i , o i ) and two negative matches The first term is from [18], while the second term encourages that the target object be better described by the true expression compared to expressions describing other objects in the image (i.e., a ranking loss on expressions). Contrastive examples are sampled from the objects and expressions in the same image.

Listener
We use a joint-embedding model to mimick the listener's behaviour. The purpose of this embedding model is to encode the visual information from the target object and semantic information from the referring expression into a joint embedding space that embeds vectors that are visually or semantically related closer together in the space. Here for referring expression comprehension task, given a referring expression representation, the listener embeds it into the joint space, then selects the closest object in the embedding space for the predicted target object.
As illustrated in Fig. 3, for our listener joint-embedding model (outlined by a red dashline), we use an LSTM to encode the input referring expression and the same visual representation as the speaker to encode the target object (thus connecting the speaker to the listener). We then add two MLPs (multi-layer perceptions) and two L2 normalization layers following each view, the object and the expression. Each MLP is composed of two fully connected layers with ReLU nonlinearities between them, serving to transform the object view and the expression view into a common embedding space. In this space, we force the similarity on target object, referring expression pairs by applying a hinge loss over triplets. As a listener, the triplet loss function consists of a positive match and two negative matches: where the negative matches are randomly chosen from the other objects and expressions in the same image.
Note that the listener model is not limited to this particular triplet-based model. For example, [22] computes a similarity score between every object for given referring expression, and minimizes the cross entropy of the SoftMax knowing the target object, which could also be applied here.

Reinforcer
Besides using the ground-truth pairs of target object and referring expression for training the speaker, we also use reinforcement learning to guide the speaker toward generating less ambiguous expressions (expressions that apply to the target object but not to other objects). This reinforcer module is composed of a discriminative reward function and performs a non-differentiable policy gradient update to the speaker.
Specifically, given the softmax output of the speaker's LSTM, we sample words according to the categorical distribution at each time step, resulting in a complete expression after sampling the <END> token. This sampling operation is non-differentiable as we do not know whether an expression is ambiguous or not until we feed it into a reward function. Therefore, we use policy gradient reinforcement learning to update the speaker's parameters. Here, the goal is to maximize the reward expectation F (w 1:T ) under the distribution of p(w 1:T ; θ) parameterized by the speaker, i.e., J = E p(w 1:T ) [F ]. According to the policy gradient algorithm [27], we have Where log p(w t ) is defined by the softmax output. We then use this gradient to update our speaker model during training.
The only thing left is to choose a reward function that encourages the speaker to sample less ambiguous expressions. As illustrated in Fig. 3 (outlined in dashed orange), the reinforcer module learns a reward function using paired objects and expressions. We again use the same visual representation for the target object and use another LSTM to encode the expression representation. Rather than using two MLPs to encode each view as in the listener, here we concatenate the two views and feed them together into a MLP to learn a 1-d Logistic Regression score between 0 and 1. Trained with cross-entropy loss, the reward function computes a match score between an input object and expression. We use this score as the reward signal in Eqn. 4 for sampled expression and target object pairs. After training, the reward function is fixed to assist our joint speaker-listener system.

Joint Model
In this subsection, we describe some specifics of how our three modules (speaker, listener, reinforcer) are integrated into a joint framework (shown in Fig. 3). For the listener, we notice that the visual vector in the embedding space is learned to capture the neighbourhood vectors of referring expressions, thus making it aware of the listener's knowledge. Therefore, we take this MLP embedded vector as an additional input for the speaker LSTM model (concatenated with the standard CNN visual representation) to encode listener based information into the speaker (see 'concat' box in Fig. 3). During training, we sample the same triplets for both the speaker and listener. and make the word embedding of the speaker and listener shared to reduce the number of parameters. For the reinforcer module, we do sentence sampling using the speaker's LSTM as shown in the top right of Fig. 3. Within each mini-batch, the sampled expressions for the target objects are fed into the reward function to obtain reward values.
The overall loss function is formulated as a multi-task learning problem:

Comprehension and Generation
For the comprehension task, at test time, we could use either the speaker or listener to select the target object given an input expression. Using the listener, we would embed the input expression into the learned embedding space and select the closest object as the predicted target. Using the speaker, we would generate expressions for each object within the image and then select the object whose generated expression best matches the input expression. Therefore, we utilize both modules by ensembling the speaker and listener predictions together to pick the most probable object given a referring expression.
Surprisingly, using the speaker alone (setting λ to 0) already achieves state-of-art results due to the joint training. Adding the listener further improves performance to more than 4% over previous state-of-art results.
For the generation task, we first let the speaker generate multiple expressions per object via beam search. We then use the listener to rerank these expressions and select the least ambiguous expression, which is similar to [1]. To fully utilize the listener's power in generation, we propose to consider cross comprehension as well as the diversity of expressions by minimizing the potential: The first term and second term in unary potential measure how well the target object and generated expression match using the speaker and listener modules respectively, which was also used in [1]. The third term in unary potential measures the likelihood of the generated sentence for describing other objects in the same image. The pairwise potential penalize the same sentences being generated for different objects (encouraging diversity in generation). In this way, the expressions for every object in an image are jointly generated. Compared with the previous model that attempted to tie language generation of referring expressions together [29], the constraints in Eqn. 7 are more explicit and overall this works better to reduce ambiguity in the generated expressions.

Optimization
We optimize our model using Adam [13] with an initial learning rate of 0.0004, halved every 2,000 iterations, with a batch size of 32. The word embedding size and hidden state size of the LSTM are set to 512. We also set the output of each MLP and FC in Fig. 3 to be 512-dimensional. To avoid overfitting, we apply dropout with a ratio of 0.2 after each linear transformation in the MLP layers of the listener. We also regularize the word-embedding and output layers of the LSTM in the speaker using dropout with ratio of 0.5. For the constrastive pairs, we set λ l 1 = 1 and λ l 2 = 1 in the listener's objective function (Eqn. 3), and set λ s 1 = 1 and λ s 2 = 0.1 in the speaker's objective function (Eqn. 2).

Datasets
We perform experiments on three referring expression datasets: RefCOCO, RefCOCO+ and RefCOCOg (described in Sec 2). All three datasets are collected on MSCOCO images [16], but with several differences: 1) Re-fCOCO and RefCOCO+ were collected using an interactive game interface while RefCOCOg was collected in a noninteractive setting and contains longer expressions, 2) Ref-COCOg contains on average 1.63 objects of the same type per images, while RefCOCO and RefCOCO have 3.9 on average, 3) RefCOCO+ disallowed absolute location words in referring expressions. Overall, RefCOCO has 142,210 expressions for 50,000 objects in 19,994 images, RefCOCO+ has 141,565 expressions for 49,856 objects in 19,992 images, and RefCOCOg has 104,560 expressions for 54,822 objects in 26,711 images.
Additionally, each dataset is provided with dataset splits for evaluation. RefCOCO and RefCOCO+ provide person vs. object splits for evaluation. Images containing multiple people are in "TestA" while images containing multiple objects of other categories are in "TestB". For RefCOCOg, the authors divide their dataset by randomly partitioning objects into training and testing splits. Thus same image may appear in the two splits. As only training and validation splits have been released for this dataset, we use the hyperparamters cross-validated on RefCOCO to train models on RefCOCOg.

Comprehension Task
After training, we can either use the speaker or listener to perform the comprehension task. For the speaker that models P (r|o), we feed every ground-truth object region within the given image to the speaker and select the most probable region for the expression as the comprehension result, i.e., o * = argmax oi p(r|o i ). For the listener, we directly compute P (r, o) between the proposal/object and expression and pick the object with the highest probability. For evaluation, we compute the intersection-over-union (IoU) of the comprehended region with the ground-truth object. If the IoU score of the predicted region is greater than 0.5, we consider this a correct comprehension.
To demonstrate the benefits of each module, we run ablation studies in Lines 3-12 of Table 1 (for speaker as comprehender) and in Lines 3-8 of Table 2 (for listener as comprehender) on all three datasets. Each row shows the results after adding a module during training. For some models that have both speaker and listener, we highlight the module being used for comprehension in bold. For example, "speaker+listener" means we use the speaker module of the joint model to do the comprehension task, while "speaker+listener" means we use the listener module for this task. Note our speaker module is implemented using the "visdif" model in [29] as mentioned in Section 3.1 1 . Following previous work [18] [29][20], we show the results trained with MMI and those trained w/o MMI on speakers. We compare our models with the "baseline" model [18] (Line 3, 8 in Table 1), the pure listener model (Line 1), and   Table 2: Ablation study using listener or ensembled listener+speaker modules for the comprehension task (indicated in bold).
Top half shows performance given ground truth bounding boxes for objects, bottom half performance using automatic object detectors to select potential objects. We find that jointly training with the speaker improves listener's performance and that adding the reinforcer module in an ensembled speaker+listener prediction performs the best.
First, we evaluate the performance of the speaker on the comprehension task (Table 1). We observe all speaker models trained with MMI outperform w/o MMI. We also find Figure 4: Example comprehension results based on detection. Green box shows the ground-truth region, blue box shows correct comprehension using our "speaker+listener+reinforcer+MMI" model, and red box shows incorrect comprehension. We use top two rows to show some correct comprehensions and bottom two rows to show some incorrect ones.
that the speaker can be improved by joint training with the listener module and by incorporating the reinforcer module. With MMI ranking, the speaker learned with joint training (Line 12) is able to outperform the pure listener by around 2% on all three datasets, which already achieves state-of-art performance on the comprehension task. Second, we show evaluations using variations of the listener module or ensembled listener+speaker modules (indicated in bold) for the comprehension task in Table 2. We note that the listener generally works better than speaker for the comprehension task, indicating that the deterministic joint-embedding model is more suitable for this task than the speaker model -similar results were observed in [22]. While the reinforcer module seems not to be as effective as the speaker, we still find that the joint training always brings additional discriminative benefits to the listener module, resulting in improved performance (compare Line 3-8 with Line 1 in Table 2). Ensembling the speaker and listener together achieves the best results overall.
Both of the above experiments analyze comprehension performance given ground truth bounding boxes for potential comprehension objects, where the algorithm must select which of these objects is being described. This provides an analysis of comprehension performance independent of any particular object detection method. Additionally, we also show results using an object detector to automatically select regions for consideration during comprehension in the bottom half of each table (Tables 1 and 2). As our detection algorithm, we use current state of the art detector in effectiveness and speed, SSD [17], trained on a subset of the MS COCO train+val dataset, removing images that are in the test splits of RefCOCO or RefCOCO+ or in the validation split of RefCOCOg. We empirically select 0.35 as the confidence threshold for detection output. While performance drops somewhat due to the strong dependence of "visdif" model on detection [29], the overall improvements brought by each module are consistent with using ground-truth objects, showin the robustness of our joint model. Fig. 4 shows some comprehension results using our full model.

Generation Task
For the generation task, we evaluate variations on the speaker module. Evaluating the generation is not as simple as comprehension. In image captioning, BLEU, ROUGE,   METEOR and CIDEr are common automatic metrics and have been widely used as standard evaluations. We show automatic evaluation using the METEOR and CIDEr metrics for generation in Table 3 where "+rerank" denotes models incorporating the reranking mechanism and global optimization over all objects (Eqn. 7). To computer CIDEr robustly, we collect more expressions for objects in the test sets for RefCOCO and RefCOCO+, obtaining 10.1 and 9.4 expressions respectively on average per object. For Ref-COCOg we use the original expressions released with the dataset which may be limited, but we still show its performance for completeness. We choose the "speaker+tie" model in [29] as reference, which learns to tie the expression generation together. Generally we find that the speaker in jointly learned models achieves higher scores than the single speaker under both metrics across datasets. The full model with "+rerank" achieves the highest scores. In addition, since previous work [29] has found that these metrics do not always agree well with human judgments for referring expressions, we also run a human evaluation on the same set of objects as [29] for RefCOCO and Ref-COCO+. Here we ask Turkers to click on the referred object given a generated expression. These results are shown in Table. 4. Results indicate that the "+rerank" model achieves the best performance and we also observe the advantages of joint model training of listener+speaker.
In Fig. 5, we compare the generated expressions using the speaker module of different models. We then show the joint expression generation using our full model with "+rerank" in Fig. 6. As observed, the expressions of every target object are considered together, where each of them is meant to be relevant to the target object and irrelevant to the other objects.

Conclusion
We demonstrated the effectiveness of a unified framework for referring expression generation and comprehension. Our model consists of speaker and listener modules trained jointly to improve performance and a reinforcer module to help produce less ambiguous expressions. Experiments indicate that our model outperforms state of the art for both comprehension and generation on multiple datasets and evaluation metrics.