{'fraction_non_alphanumeric': 0.027914303717706365, 'fraction_numerical': 0.014114681789540012, 'mean_word_length': 4.878148148148148, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Learning language through pictures

We propose Imaginet, a model of learning visually grounded representations of language from coupled textual and visual input. The model consists of two Gated Recurrent Unit networks with shared word embeddings, and uses a multi-task objective by receiving a textual description of a scene and trying to concurrently predict its visual representation and the next word in the sentence. Mimicking an important aspect of human language learning, it acquires meaning representations for individual words from descriptions of visual scenes. Moreover, it learns to effectively use sequential structure in semantic interpretation of multi-word phrases.


Introduction
Vision is the most important sense for humans and visual sensory input plays an important role in language acquisition by grounding meanings of words and phrases in perception. Similarly, in practical applications processing multimodal data where text is accompanied by images or videos is increasingly important. In this paper we propose a novel model of learning visually-grounded representations of language from paired textual and visual input. The model learns language through comprehension and production, by receiving a textual description of a scene and trying to "imagine" a visual representation of it, while predicting the next word at the same time.
The full model, which we dub IMAGINET, consists of two Gated Recurrent Unit (GRU) networks coupled via shared word embeddings. IMAGINET uses a multi-task Caruana (1997) objective: both networks read the sentence word-by-word in parallel; one of them predicts the feature representation of the image depicting the described scene after reading the whole sentence, while the other one predicts the next word at each position in the word sequence. The importance of the visual and textual objectives can be traded off, and either of them can be switched off entirely, enabling us to investigate the impact of visual vs textual information on the learned language representations.
Our approach to modeling human language learning has connections to recent models of image captioning (see Section 2). Unlike in many of these models, in IMAGINET the image is the target to predict rather then the input, and the model can build a visually-grounded representation of a sentence independently of an image. We can directly compare the performance of IMAGINET against a simple multivariate linear regression model with bag-of-words features and thus quantify the contribution of the added expressive power of a recurrent neural network. We evaluate our model's knowledge of word meaning and sentence structure through simulating human judgments of word similarity, retrieving images corresponding to single words as well as full sentences, and retrieving paraphrases of image captions. In all these tasks the model outperforms the baseline; the model significantly correlates with human ratings of word similarity, and predicts appropriate visual interpretations of single and multi-word phrases. The acquired knowledge of sentence structure boosts the model's performance in both image and caption retrieval.

Related work
Several computational models have been proposed to study early language acquisition. The acquisition of word meaning has been mainly modeled using connectionist networks that learn to associate word forms with semantic or perceptual features (e.g., Li et al., 2004;Coventry et al., 2005;Regier, 2005), and rule-based or probabilistic implementations which use statistical reg-ularities observed in the input to detect associations between linguistic labels and visual features or concepts (e.g., Siskind, 1996;Yu, 2008;Fazly et al., 2010). These models either use toy languages as input (e.g., Siskind, 1996), or childdirected utterances from the CHILDES database (MacWhinney, 2014) paired with artificially generated semantic information. Some models have investigated the acquisition of terminology for visual concepts from simple videos (Fleischman and Roy, 2005;Skocaj et al., 2011). Lazaridou et al. (2015) adapt the skip-gram word-embedding model (Mikolov et al., 2013) for learning word representations via a multi-task objective similar to ours, learning from a dataset where some words are individually aligned with corresponding images. All these models ignore sentence structure and treat inputs as bags of words.
A few models have looked at the concurrent acquisition of words and some aspect of sentence structure, such as lexical categories (Alishahi and Chrupała, 2012) or syntactic properties (Howell et al., 2005;Kwiatkowski et al., 2012), from utterances paired with an artificially generated representation of their meaning. To our knowledge, no existing model has been proposed for concurrent learning of grounded word meanings and sentence structure from large scale data and realistic visual input.
Recently, the engineering task of generating captions for images has received a lot of attention (Karpathy and Fei-Fei, 2014;Mao et al., 2014;Kiros et al., 2014;Vinyals et al., 2014;Chen and Zitnick, 2014;Fang et al., 2014). From the point of view of modeling, the research most relevant to our interests is that of Chen and Zitnick (2014). They develop a model based on a contextdependent recurrent neural network (Mikolov and Zweig, 2012) which simultaneously processes textual and visual input and updates two parallel hidden states. Unlike theirs, our model receives the visual target only at the end of the sentence and is thus encouraged to store in the final hidden state of the visual pathway all aspects of the sentence needed to predict the image features successfully. Our setup is more suitable for the goal of learning representations of complete sentences.

Models
IMAGINET consists of two parallel recurrent path-  Chung et al. (2014). GRUs are related to the Long Short-Term Memory units (Hochreiter and Schmidhuber, 1997), but do not employ a separate memory cell. In a GRU, activation at time t is the linear combination of previous activation, and candidate activation: where � is elementwise multiplication. The update gate determines how much the activation is updated: The candidate activation is computed as: The reset gate is defined as: Our gated recurrent units use steep sigmoids for gate activations: and rectified linear units clipped between 0 and 5 for the unit activations: σ(z) = clip(0.5(z + abs(z)), 0, 5) Figure 1 illustrates the structure of the network. The word embeddings is a matrix of learned parameters W e with each column corresponding to a vector for a particular word. The input word symbol S t of sentence S at each step t indexes into the embeddings matrix and the vector x t forms input to both GRU networks: This input is mapped into two parallel hidden states, h V t along the visual pathway, and h T t along the textual pathway: The final hidden state along the visual pathway h V τ is then mapped to the predicted target image representationî by the fully connected layer with parameters V and the clipped rectifier activation: Each hidden state along the textual pathway h T t is used to predict the next symbol in the sentence S via a softmax layer with parameters L: The loss function whose gradient is backpropagated through time to the GRUs and the embeddings is a composite objective with terms penalizing error on the visual and the textual targets simultaneously: where θ is the set of all IMAGINET parameters. L T is the cross entropy function: while L V is the mean squared error: By setting α to 0 we can switch the whole textual pathway off and obtain the VISUAL model variant. Analogously, setting α to 1 gives the TEX-TUAL model. Intermediate values of α (in the experiments below we use 0.1) give the full MUL-TITASK version. Finally, as baseline for some of the tasks we use a simple linear regression model LINREG with a bag-of-words representation of the sentence:î = Ax + b whereî is the vector of the predicted image features, x is the vector of word counts for the input sentence and (A, b) the parameters of the linear model estimated via L 2 -penalized sum-ofsquared-errors loss.  (Simonyan and Zisserman, 2014). We used 1024 dimensions for the embeddings and for the hidden states of each of the GRU networks. We ran 8 iterations of training, and we report either full learning curves, or the results for each model after iteration 7 (where they performed best for the image retrieval task). For training we use the standard MS-COCO training data. For validation and test, we take a sample of 5000 images each from the validation data.

Word representations
We assess the quality of the learned embeddings for single words via two tasks: (i) we measure similarity between embeddings of word pairs and compare them to elicited human ratings; (ii) we examine how well the model learns visual representations of words by projecting word embeddings into the visual space, and retrieving images of single concepts from ImageNet.
Word similarity judgment For similarity judgment correlations, we selected two existing benchmarks that have the largest vocabulary overlap with our data: MEN 3K (Bruni et al., 2014) and SimLex-999 (Hill et al., 2014). We measure the similarity between word pairs by computing the cosine similarity between their embeddings from three versions of our model, VISUAL, MULTI-TASK and TEXTUAL, and the baseline LINREG.  tures human similarity judgments better than VI-SUAL include antonyms (dusk, dawn), collocations (sexy, smile), or related but not visually similar words (college, exhibition).
Single-word image retrieval In order to visualize the acquired meaning for individual words, we use images from the ILSVRC2012 subset of Im-ageNet (Russakovsky et al., 2014) as benchmark. Labels of the images in ImageNet are synsets from WordNet, which identify a single concept in the image rather than providing descriptions of its full content. Since the synset labels in ImageNet are much more precise than the descriptions provided in the captions in our training data (e.g., elkhound), we use synset hypernyms from Word-Net as substitute labels when the original labels are not in our vocabulary. We extracted the features from the 50,000 images of the ImageNet validation set. The labels in this set result in 393 distinct (original or hypernym) words from our vocabulary. Each word was projected to the visual space by feeding it through the model as a one-word sentence. We ranked the vectors corresponding to all 50,000 images based on their similarity to the predicted vector, and measured the accuracy of retrieving an image with the correct label among the top 5 ranked images (Accuracy@5). Table 2 summarizes the results: VISUAL and MULTITASK learn more accurate word meaning representations than LINREG.

Sentence structure
In the following experiments, we examine the knowledge of sentence structure learned by IMAG-INET, and its impact on the model performance on image and paraphrase retrieval.
Image retrieval We retrieve images based on the similarity of their vectors with those predicted by IMAGINET in two conditions: sentences are fed to the model in their original order, or scrambled. Figure 2 (left) shows the proportion of sentences for which the correct image was in the top 5 highest ranked images for each model, as a function of the number of training iterations: both models out- perform the baseline. MULTITASK is initially better in retrieving the correct image, but eventually the gap disappears. Both models perform substantially better when tested on the original captions compared to the scrambled ones, indicating that models learn to exploit aspects of sentence structure. This ability is to be expected for MULTI-TASK, but the VISUAL model shows a similar effect to some extent. In the case of VISUAL, this sensitivity to structural aspects of sentence meaning is entirely driven by how they are reflected in the image, as this models only receives the visual supervision signal.
Qualitative analysis of the role of sequential structure suggests that the models are sensitive to the fact that periods terminate a sentence, that sentences tend not to start with conjunctions, that topics appear in sentence-initial position, and that words have different importance as modifiers versus heads. Figure 3 shows an example; see supplementary material for more.
IMAGINET vs captioning systems While it is not our goal to engineer a state-of-the-art image retrieval system, we want to situate IMAGINET's performance within the landscape of image retrieval results on captioned images. As most of these are on Flickr30K (Young et al., 2014), we ran MULTITASK on it and got an accuracy@5 of 32%, within the range of numbers reported in previous work: 29.8% (Socher et al., 2014), 31.2% (Mao et al., 2014), 34% (Kiros et al., 2014) and 37.7% (Karpathy and Fei-Fei, 2014). Karpathy and Fei-Fei (2014) report 29.6% on MS-COCO, but with additional training data.
Original a couple of horses UNK their head over a rock pile rank 1 two brown horses hold their heads above a rocky wall . rank 2 two horses looking over a short stone wall . Scrambled rock couple their head pile a a UNK over of horses rank 1 an image of a man on a couple of horses rank 2 looking in to a straw lined pen of cows Original a cute baby playing with a cell phone rank 1 small baby smiling at camera and talking on phone . rank 2 a smiling baby holding a cell phone up to ear . Scrambled phone playing cute cell a with baby a rank 1 someone is using their phone to send a text or play a game . rank 2 a camera is placed next to a cellular phone . " a variety of kitchen utensils hanging from a UNK board ." "kitchen of from hanging UNK variety a board utensils a ." Figure 3: For the original caption MULTITASK understands kitchen as a modifier of headword utensils, which is the topic. For the scrambled sentence, the model thinks kitchen is the topic.
Paraphrase retrieval In our dataset each image is paired with five different captions, which can be seen as paraphrases. This affords us the opportunity to test IMAGINET's sentence representations on a non-visual task. Although all models receive one caption-image pair at a time, the co-occurrence with the same image can lead the model to learn structural similarities between captions that are different on the surface. We feed the whole set of validation captions through the trained model and record the final hidden visual state h V τ . For each caption we rank all others according to cosine similarity and measure the proportion of the ones associated with the same image among the top four highest ranked. For the scrambled condition, we rank original captions against a scrambled one. Figure 2 (right) summarizes the results: both models outperform the baseline on ordered captions, but not on scrambled ones. As expected, MULTITASK is more affected by manipulating word order, because it is more sensitive to structure. Table 3 shows concrete examples of the effect of scrambling words in what sentences are retrieved.

Discussion
IMAGINET is a novel model of grounded language acquisition which simultaneously learns word meaning representations and knowledge of sentence structure from captioned images. It acquires meaning representations for individual words from descriptions of visual scenes, mimicking an important aspect of human language learning, and can effectively use sentence structure in semantic interpretation of multi-word phrases. In future we plan to upgrade the current wordprediction pathway to a sentence reconstruction and/or sentence paraphrasing task in order to encourage the formation of representations of full sentences. We also want to explore the acquired structure further, especially for generalizing the grounded meanings to those words for which visual data is not available.