{'fraction_non_alphanumeric': 0.0307551574033184, 'fraction_numerical': 0.012585956721973377, 'mean_word_length': 4.851421188630491, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Understanding Infographics through Textual and Visual Tag Prediction

We introduce the problem of visual hashtag discovery for infographics: extracting visual elements from an infographic that are diagnostic of its topic. Given an infographic as input, our computational approach automatically outputs textual and visual elements predicted to be representative of the infographic content. Concretely, from a curated dataset of 29K large infographic images sampled across 26 categories and 391 tags, we present an automated two step approach. First, we extract the text from an infographic and use it to predict text tags indicative of the infographic content. And second, we use these predicted text tags as a supervisory signal to localize the most diagnostic visual elements from within the infographic i.e. visual hashtags. We report performances on a categorization and multi-label tag prediction problem and compare our proposed visual hashtags to human annotations.


Introduction
If a hashtag can be worth 140 characters, how much is a visual hashtag worth? While text can be used to clearly convey a short message, a meaningful icon conveys the gist of a webpage or poster right away, grabbing attention while helping store the message in memory [2]. Identifying these visual regions requires an understanding of both the textual and visual content of the infographic. In this paper, we introduce a system that identifies these "visual hashtags", iconic image regions that represent key topics of an infographic. For instance, given an infographic with topics "economy" and "environment", relevant visual hashtags could be crops showing a coin (for economy) or the earth (for environment).
Infographics are visual encodings of visual and textual media, including graphs, visualizations, and graphic designs. They are specifically designed to provide an effective * Indicates equal contribution. Figure 1. Our proposed approach for generating visual hashtags: the text in an infographic predicts the tags, the visual model fires on the patches that most activate for this tag, and a segmentation pipeline is run to extract the representative visual elements from the highly activating regions of the infographic. The result is automatically generated by our model. visual digest with the intent of delivering a message. Tags can serve as key words describing this message to facilitate data organization, retrieval from large databases, and sharing on social media.
Analogously, we propose an effective visual digest of infographics via visual hashtags. Instead of providing visual summaries or thumbnails of the whole infographic, visual hashtags correspond to specific visual concepts or top-ics inside the infographic's rich visual space. We introduce a computational system that, given an infographic as input, produces discriminative textual and visual hashtags. Just as YouTube videos use representative frames as thumbnails, we identify relevant crops of an infographic as a "preview" of its content. Such thumbnails may aid in retrieval applications (e.g. organizing and visualizing large infographic collections from a webpage or file system). We evaluate the quality of visual hashtags by comparing the system's output to the image regions humans box as relevant to a particular textual tag on a given image.
Unlike most natural images, infographics often contain embedded text that provides meaningful context for the visual content. We leverage this text to first make category (topic) and tag (sub-topic) predictions. We then use these predictions to constrain and disambiguate the automatically extracted visual features.
This disambiguation is a key step in identifying the most diagnostic regions of an infographic. For instance, in Fig.  2 which contains diverse visual elements, if a predicted text tag is Environment then the system can condition visual object proposals on this topic and focus on related regions like the water droplet and spray bottle. On the other hand, if the predicted text tag is Education, the system can condition proposals on regions like the book. Thus, we can use the predicted text tags as a kind of supervisory signal for the visual model, to identify visual regions indicative of the different topics in the infographic.
Approach: We present our tagging application on a dataset of 29K infographics scraped from Visually (http: //visual.ly/view). Each infographic comes with a designer-assigned category label, multiple tags, and other meta-data (Sec. 3). We achieve prediction accuracy of 46% when predicting the top category out of 26 categories. For text tags, we achieve 48.2% top-1 average precision at predicting at least one of the possible few tags for an image out of 391 possible tags. These predictions are driven by text that we automatically extracted from the infographics and post-processed with a single-hidden-layer neural network (Sec. 4.1). Separately, we train category and tag prediction from image patches using a deep multiple-instance learning framework (Sec. 4.2). At test time, we run our patch-based visual network densely over an infographic, constrained to the tags predicted by the text network, to generate visual region proposals associated to the text tags. These proposals are then fed to a deep mask segmentation pipeline to generate the final visual hashtags (Fig. 1).
Contributions: We introduce the problem of visual hashtag discovery, which consists of extracting diagnostic visual regions for particular topics. We demonstrate the utility of a patch-based, deep multiple instance approach for the processing of intractably large (up to 8000 pixels/side) and visually rich images. Unlike approaches that use text out-side of an image for visual recognition tasks, we show the power of extracting text from within the image itself for facilitating visual recognition. On a novel curated dataset of 29K infographics, we report performances on a categorization problem and a multi-label tag prediction problem, and show results of our automatically extracted visual hashtags.

Related work
Conventionally, computer vision research has focused mostly on understanding natural images and scenes, while very little work has been done on digitally born media. Some work has been present in [18] where the authors use computer vision techniques for geometry diagrams, and more recently in [10] where the authors use graph structures to syntactically parse diagrams. In a similar vein, [23] show that simpler, abstract digital images can be used in place of natural images to understand the semantic relationship between visual media and their natural language representation. However, to the best of our knowledge there is no work on automated understanding of infographics using computer vision techniques.
Our task of text tag prediction for images is similar to that presented in [3], however we attempt it on infographic images as opposed to natural images. Also, unlike [3], where the authors trained a joint embedding of visual and text features, we solve the problem using just the visual features of an image. To work around the large size of the infographic images, we use a variant of multiple-instance learning approach [5].
We also predict text tags using the text extracted from within these images, which has not been tried before to the best of our knowledge. To obtain a distributed representation for the extracted text, we used the mean word2vec [12] representation, as suggested by [20]. We also tried other representations like the glove embedding [15], and tweet2vec [4].
In this paper, we present a method to extract visual hashtags from infographics using only image-level tags. This weakly suppervised learning scheme is similar to [21], where the category labels are used to estimate the location of the elements in the image. However, unlike [21], we combine this weakly supervised model with a tag classifier based in the extracted text to improve the final prediction.

Infographics dataset
We scraped 63,885 static infographic images from the Visually website, a community platform for hand curated visual content. Each infographic is hand categorized, tagged, and described by the designer, making it a rich source of annotated images. Despite the difference in visual content, compared to other scene text datasets such as ICDAR 03 [11], ICDAR 15 [9], COCO-Text [19] and VGG SynthText in the wild [7], the Visually dataset is similar in size and richness of text annotations, with metadata including labels for 26 categories (available for 90.21% of the images), 19K tags (for 76.81% of the images), titles (99.98%) and descriptions (93.82%).
We curated a subset of this 63K dataset to obtain a representative subset of 28,973 images (Table 1). Uploaded tags are free text, so many of the original tags are either semantically redundant or have too few instances. Redundant tags were merged using WordNet [13] and manually, and only the 391 tags with at least 50 image instances each were retained. To produce the final 29K dataset, we further filtered images to contain a category annotation, at least one of the 391 tags. 99.6% of these images had visual aspect ratio between 1:5 and 5:1. Of this dataset, 10% was held out as our test set, and the remaining 26K images were used for training our text and visual models. For 330 of the test images, we collected additional crowdsourced annotations in order to have ground truth visual hashtags for evaluation.

Approach
Given an infographic as input, our goal is to predict one or more text tags and visual hashtags that are diagnostic of the topics depicted in the infographic. We split this problem into two steps: (1) predicting the text tags for an infographic, and (2) using the predicted text tags to localize the most representative visual regions.
Infographics are composed of a mix of text and visual elements, which combine to generate the message of the infographic. Given that the text is a very strong cue for the topic, we use it to provide context -a sort of supervisory signal -for the visual hashtag predictions. We use the text features to infer the category and tags for the infographic, and given these labels, we ask the visual model to predict the most confident visual regions indicative of these labels. Learning a mapping directly from visual features to labels is a more ambiguous problem: not all topics are represented visually, and not all visual elements are relevant to the topic of the infographic (Sec. 5.2). Textual features help to disambiguate the mapping between visual regions and topics. Importantly, the text we use for prediction is extracted from within the image using optical character recognition.

Text to labels
Given an infographic encoded as a bitmap as input, we detected and extracted (i.e., optical character recognition) the text, and then used the text to predict labels for the whole infographic. These labels come in two forms: either a single category per infographic (1 of 26), or multiple tags per infographic (out of a possible 391 tags).
Automatic text extraction: We used the stand-alone text spotting system of Gupta et al. [7] to discover text regions in our infographics. We automatically cleaned the text using spell checking and dictionary constraints in addition to the ones already in [7] to further improve results. On average, we extracted 95 words per infographic (capturing the title, paragraphs, annotations, and other text).
Feature learning with text: For each extracted word, we computed a 300-dimensional word2vec representation [12]. The mean word2vec of the bag of extracted words was used as the distributed representation for the extracted text of the whole image (the global feature vector of the text). This mean word2vec representation was fed into two single-hidden-layer neural networks for predicting the category and tags of each infographic. Category prediction was set up as a multi-class problem, where each infographic belongs to 1 of 26 categories. Tag prediction was set up as a multi-label problem with 391 tags, where each infographic Figure 3. Our proposed training approach separately samples and processes visual and text regions from an infographic to predict labels automatically. Bags of patches are sampled in a multiple instance learning formulation, and their predictions are averaged to produce the final classification. Text regions are automatically localized, extracted, and converted into word2vec representations. The average word2vec representation is then fed into a single hidden layer neural network to produce the final classification.
could have multiple tags ( Table 1). The network architecture is the same for both tasks and is depicted in the red box in Fig. 3, where the label is either one category or multiple tags. We used 26K labeled infographics for training and the rest for testing.

Patches to labels
Separately from the text, we trained a deep neural network model to learn an association between just the visual features and category and tag labels.
Working with large images: Since we have categories and tags for all the images in the training data, a first attempt might be to directly learn to predict the category or tag from the whole image. However, the infographics are large images often measuring beyond 1000x1000 pixels. Resizing the images reduces the resolution of visual elements which might not be perceivable at small scales. In particular, relative to the full size of the infographic, many of the pictographs take up very little real-estate but could otherwise contribute to the label prediction. A fully convolutional approach with a batch of such large images was infeasible in terms of memory use. Our approach was to use a bag of sampled patches to represent the image. To sample the patches, we tried both random crops and object proposals from Alexe et al. [1].
Multiple instance learning (MIL) prediction: Given a category or tag label, we expect that specific parts of the infographic may be particularly revealing of that label, even though the whole infographic may contain many diverse visual elements. In MIL, the idea is that we may have a bag of samples (in this case, image patches) to which a label corresponds. The only constraint is that at least one of the samples correspond to the label; the other samples may or may not be relevant.
We used the deep MIL formulation from Wu et al. [20] for learning deep visual representations. We passed each sampled patch from an infographic through the same convolutional neural network architecture, and aggregated the hidden representations to predict a label for the whole bag of patches (depicted in the blue box in Fig. 3). For aggregating the representations, we tried both element-wise mean and max at the last hidden layer before the softmax transformation, but found mean worked better. As with the text model, we trained separate models for multi-class category prediction and multi-label tag prediction.
Feature learning with patches: We sampled 5 patches from each infographic and resized each to 224x224 pixels for input into our convolutional neural network. For feature learning, we used ResNet-50 [8], a residual neural network architecture with 50 layers, initialized by pretraining on Im-ageNet [17]. We retrained all layers of this network on 26K infographics with ground truth labels.

Labels to visual hashtags
The text in an infographic is often the strongest predictor of the topic matter, achieving significantly better accuracies at predicting the category and tags of infographics than the visual features alone (Sec. 5). Driven by these results, we make our initial label (category and tag) predictions using the text features. The predictions in turn constrain the visual network to produce activations for the target label.
At inference time, we sample 3500 random crops per infographic and compute the confidence, under the visual classifier, of the target label. We assign this confidence score to all the pixels within the patch, and aggregate perpixel scores for the whole infographic. After normalizing these values by the number of sampled patches each pixel occurred in, we obtain a heatmap of activations for the target label. We use this activation map both to visualize the most highly activated regions in an infographic for a given label, and to extract visual hashtags from these regions.
For automatically extracting visual hashtags, we thresh-  old the activation heatmap for each predicted tag, and identify connected components as proposals for regions potentially containing visual hashtags. These are cropped and passed to the SharpMask segmentation network [16]. Finally, visual hashtags corresponding to the predicted textual tags for an input infographic are obtained by cropping tight bounding boxes around SharpMask's proposals from the original images (Fig. 4).

Technical details
Text model: For category prediction, the mean word2vec representation of an infographic was fed through a 300-dimensional fully-connected linear layer, followed by a ReLu, and a 27-dimensional (including a background class) fully-connected output layer. The feature vectors of all 29K training images fit in memory and could be trained in a single batch, with a softmax cross-entropy loss. For tag prediction, the output layer was 391-dimensional and was passed through a sigmoid layer. Given the multi-label setting, this network was trained with binary cross-entropy (BCE) loss and one-hot encoded target vectors. Both networks were trained for 20K iterations with a learning rate of 1e − 3.
Visual model: We used bags of 5 patches for aggregating visual information from infographics. We tried bags of random patches and bags of objectness proposals [1]. Rather than the raw objectness proposals with varied aspect ratios, we took a tight-fitting square patch around each objectness proposal. We found this improved results.
As in the text model, we trained category classification with a softmax cross-entropy loss with 27-dimensional target vectors, and tag prediction with a BCE loss applied to 391-dimensional sigmoid outputs. We used a momentum of 0.9 and weight decay of 1e − 4. Our learning rate was initialized at 1e − 2. For category prediction, we updated the learning rate every epoch, and stopped training after 5 epochs. For tag prediction, we updated the learning rate every 50 epochs for 500 epochs. Tags were more specific and also much more unbalanced than category labels, so the model needed to train for significantly longer to see enough patch samples for different tags.
Activation maps: To discover maximally activated image regions for a given label, 3500 multi-scale crops were used. To generate each crop, we sampled a random coordinate value for the top left corner of the crop, and a side length equal to 10-40% of the minimum image dimension.

Results
We evaluate the ability of our full system to (1) predict category and tag labels for infographics and (2) to extract visual hashtags from images: visual regions or icons relevant to the visualization topic. Predicting the category is a high-level prediction task about the overall topic of the infographic. Predicting the multiple tags for an infographic is a finer-grained task of discovering sub-topics. We solve both tasks, and present results of our text and visual models.
Given the text model's tag predictions, the visual model that learned to associate visual concepts with tags is used for finding the relevant visual areas, and to extract visual hashtag proposals (Fig. 7). To evaluate these proposals, we collected human ground truth. For a total of 650 imagetag pairs, participants boxed image regions corresponding to the provided tag (Fig. 8). We compare our model's visual hashtag proposals to these ground truth bounding boxes.

Category prediction
Evaluation: For each infographic, we measured the accuracy of predicting the correct ground truth category out of 26, within the top 1, 3, and 5 most confident predictions.
Quantitative results: Chance level for our distribution of infographics across categories was 15.4%. We achieved 46% top-1 accuracy at predicting the category using our text model ( Table 4).
The purely vision-driven predictions are provided as a comparison point, although the final label predictions are performed using the text features. The text tends to contain a lot more information, while not all concepts can be communicated visually. The best performing visual model used a bag of random patches in a MIL framework (as in Fig. 3). Mean aggregation outperformed max aggregation for category prediction (Vis-rand-mean better than Visrand-max). Random crops outperformed objectness proposals (Vis-rand-mean better than Vis-obj-mean). We hypothesize this to be the case because each time we sampled random crops from an image, our model was exposed Figure 6. Some of the highest activating patches per category. The visual network trained to predict category labels for whole images assigned high confidence (under the respective categories) to these randomly-sampled patches from unique infographics.
to new visual regions, whereas the number of objectness proposals was a limited sample of patches from an image. In other words, our model received more diverse training data in the random crops case. The patch-based predictions were similar to, or better than, the full visualization resized (Vis-resized). A patch-based approach is naturally better suited for sampling regions for visual hashtag extraction. We also tried to combine text and visual features directly during training but did not achieve gains in performance above the text model alone, indicating that it is a sufficiently rich source of information in most cases.
Top activations per category: To validate that our visual network trained to predict categories learned meaningful features, we visualize the top patches that received the highest confidence under a few different categories (Fig. 6). These patches were obtained by sampling 100 random patches from each image, storing the single patch that maximally activated for each category per image, and outputting the top patches across all images.

Tag prediction
Evaluation: Each infographic in our 29K dataset comes with an average of 1-9 tags. At prediction time, we generate 1, 3, and 5 tags, and measure precision and recall of these predicted tags at capturing all ground truth tags for an image, for a variable number of ground truth tags.
Quantitative results: We achieved 48.2% top-1 average precision at predicting at least one of the tags for each of our infographics, since all the infographics in our dataset contain an average of 2 tags (Table 5). Since tags are finergrained than category labels, it is often the case that some word in the infographic itself maps directly to a tag. Using this insight, we add a simple automatic check: if any of the extracted words exactly match any of the 391 tags, we snap the prediction to the matching tags (Word2Vec-snap). Without this additional step, predicting top-1 tag achieves an average prediction of 30.1% using text features.
Text modeling baselines: We computed several other representations of the extracted text (Table 6). We used a voting scheme (Word2Vec-voting) by voting for the closest text tag, in word2vec embedding space, for each word in the extracted text, and predicting the top-voted tags. We also computed the Tweet2Vec [4] representation of the extracted text, as well as the mean of the Glove representations [15] of all the words (Glove-mean). Using the mean word2vec as the text features (Word2Vec-mean) gave the best results for tag prediction.
Text can disambiguate visual predictions: In some infographics, visual cues for particular tags or topics may be missing (e.g., for abstract concepts), they may be misleading (as visual metaphors), or they may be too numerous (in which case the most representative must be chosen). In these cases, label predictions driven by text are key, as in Fig. 5a, where visual features might seem to indicate that the infographic is about icebergs, or ocean, or travel; in this case, however, iceberg is used as a metaphor to discuss microblogging and social media. Our text model is able to pick up on this, and direct the visual features to activate in the relevant regions.

Visual hashtag proposals
Collecting ground truth: The Visually data comes with image-level categories and tags. Because a goal of this paper is to discover visual hashtags -individual elements within infographics that correspond to the different labelswe wanted to measure how humans complete this task. We designed an interface in which participants are given an infographic and a text tag, and are asked to mark bounding boxes around all non text-regions (e.g., pictographs) that contain a depiction of the tag (Fig. 8). If an image had multiple tags, it would be shown multiple times but to different users, with unique image-tag pairings. We collected a total of 3655 bounding boxes (ground truth visual hashtags) for the 330 images from 43 undergraduate students. Each image was seen by an average of 3 participants and we obtained an average of 4 boxes per image.
Evaluation: On average, infographics had 2 ground truth tags, with a total of 650 unique image-tag pairs for which participants annotated visual hashtags. Of these 650 pairings, participants indicated that 119 (18%) did not have corresponding visual features. In these cases, the hashtag had no visual counterpart and could perhaps only be in- Figure 7. Examples of automatic text and visual hashtag generation. (a) Text features predict half the ground truth tags correctly, and the visual model discovers associated visual regions in the infographic. Unique visual hashtags are automatically retrieved for each tag. (b) Text features predict the ground truth tags correctly, and visual features discover visual hashtags. In this specific example, there is not a one-to-one mapping between textual and visual hashtags. Similar visual areas are activated for these text tags.
(a) (b) ferred from the text of the infographic.
We evaluated the remaining 531 image-tag pairs with participant annotations (ground truth hashtags). We fed each of these image-tag pairs to our pipeline to obtain predicted visual hashtags (Sec. 4.3) and computed the intersection-over-union (IOU) of each of our predicted hashtags with participant annotations. We report only the single highest-confidence prediction for each image-tag pair ( Table 2). The confidence of our proposals is measured as the mean activation value of our visual model within the hashtag bounding box. See Fig. 9 for examples of our predicted hashtags overlaid with participant annotations.
Our pipeline was constructed for high-precision as opposed to high-recall, because our goal is to produce a reasonable visual hashtag for an image-tag pair, rather than all possible hashtags. Therefore, our evaluation measures the percent of predictions that overlap with at least one of the human annotations. We report precision as the percent of predicted hashtags that have an IOU > 0.5 with at least one ground truth hashtag (in an image-tag pair). This threshold was chosen because it is most commonly used in the object detection literature [6].
To contrast with precision, we also report the total percent of image-tag pairs for which a successful proposal with IOU > 0.5 was generated (Acc.). In this case, for any image-tag pair for which a proposal was not generated, IOU is set to be 0.
Object proposal baselines: Our average precision of 15.2% and accuracy of 9.4% beat other approaches on the task of outputting a visual hashtag proposal for a given image-tag pair ( Table 2). We took the highest-confidence object proposals from Alexe et al. [1] (Objectness) and Pinheiro et al. [16] (SharpMask). We also used a topperforming neural network model of saliency [14] (SalNet) in place of our visual model's activation map, and ran it through the same post-processing pipeline as outlined in Sec. 4.3 to obtain visual hashtag proposals. The benefits our activation map has over saliency is that saliency is tagagnostic and will always output the same map for an image. Our visual model is conditioned on a particular tag label and activates in regions of a design that are most predictive of the label. For a comparison to another weakly-supervised approach, we adjust our network to have an average pooling layer at the end, as in CAM [21]. As a chance baseline we report the performance of random crops (Random).
Increasing accuracy: When we take into account all image-tag pairs, the average percent of instances for which the predicted hashtag overlaps the ground truth with an IOU > 0.5 drops to 9.4% (from a precision of 15.2%). Our approach fails to output proposals for 38% of the image-tag pairs. Most of the filtering happens at the SharpMask stage, where region proposals from the visual activation map are passed to SharpMask for refinement. If SharpMask does not find an object candidate in an image region, that region is discarded. As a stand-alone method, SharpMask fails to output proposals for 34% of image-tag pairs. SharpMask is also used as a post-processing step for the SalNet model. In comparison, Objectness generates a candidate for all images. We can increase the percent of proposals returned by adding a fallback option to our method (Ours-fallback): even if SharpMask discards all candidates, return the most confident candidate. This allows us to guarantee proposals for all images, at the cost of lower precision.

Conclusion
To this point, the space of complex visual information beyond natural images has received limited attention in computer vision in the domain of classification and detec-  [14] 10.9% 7.0% Objectness [1] 9.0% 9.0% SharpMask [16] 8.6% 5.6% CAM [21] 5.4% 2.8% Random 0.9% 0.01% Figure 8. We collected ground truth visual hashtags by asking dozens of participants to box any visual regions corresponding to a provided tag. For evaluating our automatic model, ground truth was collected for a total of 650 image-tag pairings. tion (notable exceptions include: [10,22]). We present a novel direction based on a dataset of infographics, containing rich visual media, with a mix of visual and textual features. In this paper, we showed how textual and visual elements can be used to jointly reason about the high-level top- ics (categories) of infographics, as well as the finer-grained sub-topics (tags). We demonstrated the power of text features in disambiguating and providing context for visual features. We presented a system whereby aside from predicting text labels, we can automatically extract iconic representative elements, what we call "visual hashtags". Despite never being trained to explicitly recognize objects in images, our model is able to localize a subset of the ground truth (human-annotated) visual hashtags.
Infographics are specifically designed with a human viewer in mind, characterized by higher-level semantics, such as a story or a message. Beyond simply detecting the objects contained within them, an understanding of these infographics would involve the parsing and understanding of the included text, the layout and spatial relationships between the elements, and the intent of the designer. Human designers are experts at piecing together elements that are cognitively salient (or memorable) and maximize the utility of information. This new space of multi-media data gives computer vision researchers the opportunity to model and understand the higher-level properties of textual and visual elements of the story being told.