{'fraction_non_alphanumeric': 0.03415118560321651, 'fraction_numerical': 0.01135950783539613, 'mean_word_length': 4.876458867065186, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Explain Me the Painting: Multi-Topic Knowledgeable Art Description Generation

Have you ever looked at a painting and wondered what is the story behind it? This work presents a framework to bring art closer to people by generating comprehensive descriptions of fine-art paintings. Generating informative descriptions for artworks, however, is extremely challenging, as it requires to 1) describe multiple aspects of the image such as its style, content, or composition, and 2) provide background and contextual knowledge about the artist, their influences, or the historical period. To address these challenges, we introduce a multi-topic and knowledgeable art description framework, which modules the generated sentences according to three artistic topics and, additionally, enhances each description with external knowledge. The framework is validated through an exhaustive analysis, both quantitative and qualitative, as well as a comparative human evaluation, demonstrating outstanding results in terms of both topic diversity and information veracity.


Introduction
For the general public, art tends to be considered as a mysterious and remote discipline that requires a lot of study to be fully appreciated. In the last few years, many efforts have been made to apply artificial intelligence technologies to the domain of art to make it more accessible [16]. Thanks to the large-scale digitisation of artworks from collections all over the world [22,69,52,64], computer vision techniques have been widely adopted to address different artrelated problems [31,47,11,20,46,36,8,62,18,72].
Most of the existing work in the field is focused on the automatic analysis of paintings, addressing problems such as attribute prediction [47,62], content analysis [11,21] or style identification [30,57,31]. However, there is still an absence of research that conveys in-depth and comprehensive information of artworks for the general public. In other words, most previous work only allows people to under- Figure 1. Art description generation. Comparison between standard image captioning (top) and art image description generation (bottom). Standard image captioning usually generates a single factual natural sentence to describe the content of a real-world image, while our art image description contains multiple sentences to explain an artwork from different artistic perspectives. stand a unique aspect of an artwork by providing a single tag, usually associated with either its style or its content.
However, a real understanding of art is much more complex than being able to successfully categorize each piece into a set of pre-defined tags. The intricate relationships between the content, form, and context of each artwork are not at hand in a simple categorization process. In this work, we take a step forward on the field of art understanding and accessibility by proposing to automatically generate rich descriptions of paintings from multiple artistic perspectives. We propose a multi-topic and knowledgeable art description generation framework that produces detailed explanations about different aspects of paintings. Such a service would facilitate a deeper interaction between the general public and artworks, as well as potentially ease the work of art curators by automatically generating comments of paintings.
Art description generation may be seen as a problem similar to image captioning [67,12,1], which aims to naturally and factually describe the content of the scene in the image. Compared to conventional image captioning, however, generating descriptions for artworks faces two extra challenges. First, a comprehensive explanation of an artwork requires not only the factual description of its content, but also background knowledge, such as details about its author, the context of the creation process, and so on. This information is rarely contained in the artwork image itself. Second, according to art historians [2], an informative description of an artwork should address three main topics: content, form, and context. While in standard image captioning only the content is considered, the distinction among different artistic topics requires to handle a more complex language modeling scheme.
To address these challenges, we introduce a multi-topic and knowledgeable art description generation framework that: 1) introduces external knowledge into the description generation process, and 2) proposes a multi-topic language model to describe the different aspects of the painting. The main idea is illustrated in Fig. 1, where an artwork image is used to both generate a description and retrieve related knowledge from an external source. Our framework follows three steps. Firstly, by training a language model, we generate a masked sentence with fillable slots for the concepts that require external information to be known, such as artist, date of creation, location, etc. Moreover, the language model incorporates information about different artistic topics so that the masked sentences are generated according to each topic. Secondly, a knowledge retrieval module is employed to retrieve external information related to the painting from open access databases (e.g. Wikipedia 1 ). Finally, we design a knowledge-filling module that extracts candidate words from the retrieved knowledge and selects the appropriate concepts for each slot.
In our exhaustive experimental section, including quantitative comparisons, qualitative analysis, and human evaluation, we show that our framework generates satisfactory art descriptions more accurately and informatively than others. Overall, our main contributions are: • We propose the first framework for art description generation that creates multi-topic long descriptions of fine-art paintings. So far, art description generation has been tackled as an image captioning task by only generating short factual sentences about artworks. • We design a multi-topic language modeling module to generate multi-topic descriptions. Additionally, we annotate an art description dataset with sentence-topic labels based on art historians protocols [2], which we share publicly 2 to inspire future work not only in art description but also in general art understanding. • We leverage a knowledge retriever and train a knowledge filling module as a fill-in-the-blank task to incorporate art information relevant to each painting. This method can be easily applied to other domains.

Related Work
Artwork Analysis Computer vision techniques have been widely adopted to address art-related problems [28,47,70,60]. A fundamental task in the field is to extract representative features that can capture the insights of the style [31,20,72,27] or the content of a painting [11,10,56,21] and use them for the automatic analysis of artworks, in tasks such as classification [44,5,63,17], style identification [30,57,31,72], object recognition [11,9,21], or image retrieval [5,11,9]. Although art categorization is challenging due to the inherent diversity and abstraction in art, it only studies a single aspect of the artwork. However, paintings are complex images full of symbolism. A single label cannot totally represent the elaborated relationship between the depicted elements, the painter's motivations, and the historical context of the production. For a full comprehension of paintings, we propose to generate coherent language representations in the form of artistic descriptions. Until now, only a few studies [59,18,19] have applied multimodal vision and language techniques to the domain of art. While in [18] a system to find paintings given textual descriptions is proposed, other methods [59,19] predicted answers to questions about artworks. However, generating comprehensive descriptions for fine-art paintings is still rarely studied.
Image Captioning Encoder-decoder image captioning models for natural images are data-driven methods using deep neural networks [67,73,1,51,76,38,77,43,68]. The classic scheme [67] combines a convolutional neural network (CNN) as image encoder and a recurrent neural network (RNN) as caption decoder. Several variations have emerged, such as adding attention [73] or using detected objects instead of plain pixels [1]. Although these models obtain good results for natural images, they do not transfer well to cultural images [58]. To generate descriptions for artworks, previous work introduced an ontology [74] and a hierarchical model [75], leveraging low-level features, e.g., image texture and meta-data of cultural images, which heavily rely on feature engineering. Moreover, they could only generate a single and factual sentence about the image content. In contrast, we generate multi-topic descriptions, relying on external sources to improve the information quality. External knowledge has been used in image captioning [71,76,38,77,42,3], mostly by relying on available tags [42] or texts [3] associated to the target image. Differently, our external knowledge is retrieved by only using the image.   Figure 2. Proposed framework. It consists of three parts: masked sentence generation, knowledge retrieval, and knowledge filling.

Method
Our framework contains three main parts: 1) masked sentence generation, 2) knowledge retrieval, and 3) knowledge extraction and filling. As shown in Fig. 2, we first extract D-dimensional visual features from L spatial locations from the image [73] using a pre-trained ResNet [24], 3 Then, in the masked sentence generation part, we input V into the topic decoder to generate multi-topic masked sentences that describe the painting from multiple aspects. These masked sentences have blanked concepts to be filled with knowledge at a later stage. In the knowledge retrieval part, we take the averagepooling vectorv = i v i /L as the global visual feature for multi-attribute prediction. We also employ an object detector to detect visual concepts. The predicted attributes and the detected objects are used to retrieve relevant knowledge from an external source with DrQA [7]. Finally, in the knowledge extraction and filling part, given the generated multi-topic masked sentences and the retrieved knowledge text, we extract candidate knowledge concepts, and use a BERT-based model [14] to get the final description.

Masked Sentence Generation
Traditional image captioning datasets, such as MSCOCO [40], provide high-quality general captions. Image captioning decoders trained on these corpora predict a probability distribution over a closed vocabulary to generate text. However, these decoders have difficulties in generating specific entities that occur sparsely in the vocabulary. For example, in a vocabulary for art description, the artist name, location, or timeframe may all be in a low frequency. Moreover, the desired art descriptions should contain external knowledge not directly present in the image. We address these issues by only relying on the decoder to generate masked sentences, which are later completed by the knowledge extraction and filling module.

Data Preprocessing
Given set D of descriptions about paintings, we obtain a training corpus for the masked sentence generation part by performing Named-Entity-Recognition (NER). Specifically, we apply Stanford CoreNLP name tagger [45] to the descriptions to extract entities of the following types: person, location, organization, ordinal, number, date, and misc. Then, we replace the found entities in the description with their corresponding entity type, e.g.: An account of Vasari says that Signorelli wanted to represent in the figure of the naked Christ his own son, who died of plague in 1502.

is transformed into
An account of [person] says that [person] wanted to represent in the figure of the naked [person] his own son, who died of plague in [date].

Topic Decoder
We envisage a decoder that can handle multi-topic description generation: given a visual feature V and a desired topic d, the decoder should generate corresponding topic-related masked sentences. In this section, we first introduce a baseline decoder that generates topic-agnostic masked sentences. Then, we explore two variants to address the multitopic challenge. Finally, we explain how the multi-topic masked description is generated. Figures illustrating each decoder can be found in Appendix A.
Baseline Decoder Following Xu et al. [73], we employ a long short-term memory (LSTM) [25]-based decoder to decode image visual features into masked sentences. The decoder generates one word y t (in the one-hot vector representation) at each time step t based on the attention-based visual context vector z t , the previous hidden state h t−1 , and the previously generated word y t−1 . Formally: where [·] denotes concatenation, E is an embedding matrix, f att is a trainable function for predicting the attention weights, for which we use a multilayer perceptron, and α t = {α t1 , . . . , α tL } are attention scores that sum to one. The visual context vector z t is a dynamic representation of the relevant part of the input image at time t. Based on the LSTM state h t and visual context vector z t , we calculate the output word probability with a fully-connected layer as: where W y and b y are the parameters in the fully-connected layer. Given the ground truth masked sequence y * 1:T , the weights of the decoder are optimized by minimizing the negative log-likelihood in training as: Topic Parallel Decoder Due to the linguistic distinctions that different artistic topics may present, we propose to use different decoders to generate masked sentences for different topics independently, i.e., using different baseline decoders as sub-decoders for each topic, respectively. This parallel setting is intuitive as it divides topic-related sentences into different decoding branches, enabling the decoders to not disturb each other. Formally, the parallel decoder can be formulated as: where d is the topic label, which also serves as a selector for the sub-decoders. Within each sub-decoder, the computation is the same as in the baseline decoder: Equations (2)-(4) can be written equivalently. The different sub-decoders are optimized separately during training.

Topic Conditional Decoder
To improve the computational efficiency as well as to leverage common knowledge among the different topics, we also propose a single-model solution for multi-topic description generation. Inspired by stylized captioning [23], we explore a topic conditional decoder. The conditional decoder injects a topic conditional vector into the decoding process, formulated as: Specifically, the topic label d is transformed into a N topicdimensional one-hot vector d to represent N topic topics, where each element represents the corresponding topic. Then, we feed d into a topic embedding layer and concatenate the resulting vector with the standard inputs of the baseline decoder as: where E topic is the topic embedding matrix. To ensure that the generated masked sentences correctly contain the target topic, we employ a topic classifier T C for constrain, i.e., The topic classifier is implemented as in TextCNN [32] and is jointly optimized with the topic conditional decoder with a classical cross-entropy loss L ce . Overall, the objective L cond of topic conditional decoder is: Multi-Topic Masked Description Given N topic topics, we generate N topic masked sentences, one per topic, with either the parallel or the conditional topic decoder. The full multi-topic masked description is then the concatenation of the N topic independently generated masked sentences.

Knowledge Retrieval
To address the challenge of generating informative descriptions, we rely on external knowledge bases, such as Wikipedia. We use DrQA [7], an efficient document retriever, to find relevant information. Formally, a given query text q and all articles c j ∈ C in the knowledge base are tokenized and encoded as TF-IDF vectors, denoted asq and c j , respectively. Then, a similarity score is computed as: Unlike previous work that uses off-the-shelf tags [42] or text [3] as query to the knowledge base, we automatically constitute a query from the image by extracting: 1) Artistic attributes using a multi-task attribute prediction model [17]. Specifically, for each painting we predict its artist, type, timeframe, and school; and 2) Visual concepts using an object detection model [50] pre-trained in Visual Genome [35]. Concepts such as person, apple, etc. are extracted to describe the general content of the image, removing visual concepts that are unlikely to appear in paintings, such as cell phone. The words from the two sources are appended together to constitute our q. As an output of the knowledge retrieval module, we return the top-5 article c j 's with the highest score s j 's. To improve the ranking accuracy, we pre-process each c j as 1) stop word removal and word stemming, and 2) bigram TF-IDF.

Knowledge Extraction and Filling
In this module, we fill the masked concepts in the generated multi-topic masked descriptions with one or several knowledge words. Given the top-5 articles from the knowledge retrieval part, we further narrow down the knowledge space by extracting named entities, again using Stanford CoreNLP. We use the extracted named entities and the artistic attributes from Sec. 3.2 to compose a set of candidate words G. Then, we train a BERT-based model [14] as a sequence to sequence task to find appropriate words from G to fill the blanks in the generated multi-topic masked descriptions. Specifically, we generate an input sequence as: where k is a sequence with the concatenation of all the words in G and y denotes the sequence of words in the multi-topic masked description. The output description is: where BERT(·) is trained by minimizing the cross-entropy loss to produce the original image descriptions in D.

Experiments
Here we describe the experiments and their results. Implementation details can be found in Appendix B.
Art Dataset We use the SemArt dataset [18], which consists on 21, 384 painting images. Each image is associated with an artistic comment and seven attributes, such as artist, title, or date. The dataset is split into 19, 244 images for training, 1, 069 for validation, and 1, 069 for test.
Artistic Topic Annotation To investigate multi-topic description generation, we annotate the original comments in SemArt with their correspondent artistic topic. Following art historian protocols [2] we use three topics: 1) content, which describes what the artwork is about, i.e. the message; 2) form, which describes how the work looks, i.e., the constituent elements of the work independent of their meaning; and 3) context, which describes in what circumstances the work is or was. We rely on Amazon Mechanical Turk 4 (AMT). We split the original comments on the train and test set into individual sentences and ask workers to annotate each sentence with one of the three topics. Workers are exposed to the image, the original full comment, the title, the artist name, and the year of creation. In total, 17, 249 images along with 33, 543 sentences are annotated. 5 Knowledge Base As external source of information, we use the 2016-12-21 dump 6 of English Wikipedia. For each page, only the plain text is extracted. All structured and non-text data sections such as lists and figures are stripped. After discarding internal disambiguation, list, index, and outline pages, we retain 5, 075, 182 articles.

Human Evaluation
The evaluation of generated text is a challenging task [6], due to the complexity of automatically measuring not only grammatical correctness but also veracity, informativeness, and diversity. Automatic metrics designed to evaluate factual tasks such as machine translation (e.g., BLEU [48]) or image captioning (e.g., CIDEr [66]) do not work well on more creative tasks such as ours. Following previous work [15,55,65], we based our evaluation on how well humans perceive the text generate by our models.
We conduct a human evaluation on AMT on 100 randomly selected validation paintings. For each painting, we show a generated description to 3 annotators, together with the image, the original SemArt comment, the title, the artist, and the creation year. We ask annotators to rate each description according to the metrics below (higher is better): • Understandable: integer from 1 to 4 measuring if the description can be understood by a human. • Relevance: integer from 1 to 4 measuring if the description is relevant to the given painting.  Table 1. The baseline model (SAT [73]) do not use topic modeling or external knowledge. The other three models (denoted as Ours) use the parallel decoder, the knowledge retrieval, and knowledge extraction and filling modules. The main difference between them lies in the source of external knowledge used at inference. Specifically, in Ours (Wikipedia), we use Wikipedia as knowledge This painting depicts a river landscape with a village on a frozen river and a <unk>.
This painting is one of a pair of winter landscapes by Jan van de <unk> and Jan van <unk>.
The painting depicts a river landscape with figures on a road.
The composition of the <unk> of the Virgins is based on a painting by the young Titian. It shows the influence of <unk>.
The picture shows a view of the winter with the city in background. It is one of the most important studies of the artist's mature style of light shadow in the reminiscent of Hendrick Avercamp's composition. This is one of the most important works of Barend Avercamp's late period painting.
The painting depicts a river landscape with skaters and a rowing boat in the foreground. This painting is a typical example of Beyeren's landscapes that he had to be seen in his own lifetime and he was a good example of his contemporaries. This painting is one of the earliest known works by Beyeren.
This painting depicts a wooded landscape with a horsedrawn cart in the foreground. This painting is one of Constable's most ambitious landscapes. He had learned from Fromde and the composition is built up with a a trees in. This painting is one of the earliest known works by the artist. This painting is one of the most important of the series of paintings of the English landscape.
It is assumed that the painting represents the virgin and child in the center of the panel. This painting is a fine example of the artist ' s mature style in which he seems to have learned his skill in rendering of light and shade the composition not only a few meters. This painting was executed early by Leonardo. source. In Ours (SemArt), we build a knowledge base with the comments in SemArt and find the most relevant one with knowledge retrieval module. Finally, in Ours (Oracle), we use the original associated comments, assuming a perfect accuracy in the knowledge retrieval module.
Results show that the baseline achieves the highest score in the Understandable metric. This is because the text generated is simpler and shorter than the text generated with our models: e.g., the average number of words of SAT output is 30.9, while in Ours (Oracle) it is 71.8. The shorter the sentence, the less prone to contain grammatical errors. Nevertheless, from all the other metrics, which are more related to the informative aspect of the description, we observe that our framework outperforms the baseline by a large margin.
When comparing our three different settings, Ours (Oracle) achieves the best performance in all the metrics, with a large margin in Relevance and Veracity. This is natural, as the ground truth comments are used as the source of knowledge. In the existence metrics (i.e. Content, Form, and Context), the performance of the three models is very close, all of them outperforming the baseline by a large margin. This shows the effectiveness of the topic decoder, which is able to produce different types of sentences. Ours (Wikipedia) performs the worst among the three settings, as it uses the most challenging source of knowledge and also suffers from a domain-gap between the source of training (SemArt comments) and the source of testing (Wikipedia articles).

Qualitative Analysis and Examples
We further explore the results of our framework with a qualitative analysis in Fig. 3. We compare three methods. In the first row, we use SAT [73] trained on MSCOCO, namely SAT-Transfer. The generated sentences are very similar to the standard captions generated for natural images. This model suffers from: 1) not containing art-specific information, and 2) a visual domain-gap when transfering from natural images to paintings, with miss-detected concepts, such as 'dog' and 'person', in Madonna and Child and The Hay-Wain, respectively. In the second row, SAT is trained on the SemArt dataset, i.e. SAT-Baseline. The results show that it can generate some short and understandable sentences as in The Hay-Wain. The main problem, however, is that it does not contain background knowledge, leaving some specific knowledge words as unk in the output. Finally, the last row

Semi-supervised
In the centre of the triptych the central <unk> represents the Adoration of the Magi and the <unk>. The composition is based on a drawing by PERSON_ in LOCATION_. This is NUMBER_ of the most important works of MISC_ art.  These results also reveal some of our method limitations. Missdetection of the visual content (a common mistake in traditional image captioning) is shown in River Landscape, where the boats are confused by 'skaters'. In The Hay-Wain, there is a syntax error ('a a') in the form sentence, which is produced in the knowledge filling part, based on BERT, i.e. it does not appear in the masked-sentence decoder based on LSTM. We hypothesized that it is because the size of training set is relatively small for BERT to learn the language structure of our task. This can be solved by applying language augmentation techniques. Finally, in Madonna and Child, there is an imperfect-knowledge mistake, in which the painting is assigned to the wrong artist.

Modules Analysis
Exploring Artistic Topic Our topic decoder relies on the availability of a corpus with topic labels for training. For a more general approach, we explore un/semi-supervised topic models to automatically predict the topic label for each sentence. For the unsupervised setting, we use Latent Dirichlet Allocation (LDA) [4], which assumes each topic is a mixture over an underlying set of words. While for the semi-supervised setting, we use Guided-LDA [29], which incorporates lexical priors, manually set as a list of seed words, to LDA. In our case, the seed words are obtained from a subset of sentences with topic labels, from which we select the top-10 words with the highest frequency. The larger the subset, the more accurate the seed words obtained. Here we use 3245 samples. The artistic comments are split into individual sentences and pre-processed to optimize each topic model method. After training, the topic model predicts a pseudo topic label for each sentence and used to train our parallel topic decoder.
LDA achieves 43.3% accuracy when predicting topic labels on the test set, while Guided-LDA achieves 51.6%. To reduce the influence of the knowledge filling module, in Fig. 4 we compare the generated masked sentences for the different topic models. In the unsupervised setting, the content and context sentences are generated correctly, while the form sentence is confusing. In the semi-supervised setting, the output looks better, even close to the supervised approach, revealing that only a small amount of topic annotations may be necessary for art description generation. Knowledge Retrieval The impact of the knowledge retrieval module and how it affects the overall framework has been already discussed in Sec. 4.1. According to Table 1, the easiest to retrieve the knowledge, the highest the scores. In Appendix D, we provide accuracy rates for the knowledge retrieval module on its own with different settings. It can be seen that the performance of the knowledge retrieval module is a crucial bottleneck on our system.

Comparative Evaluation
Although automatic metrics may not correlate well with the evaluation of the knowledge and creativity required to describe art, we include a standard automatic evaluation for completeness. We compare our proposed model against: Classic image captioning (1) NIC [67], LSTM-based encoder-decoder model without attention; (2) SAT [73], which incorporates soft-attention (note that this corresponds to the baseline decoder in Sec. 3.1.2); and (3) Att2in [51], similar to SAT but in where the attention-derived context visual feature is only input to the cell node of the LSTM.
Stylized image captioning Our conditional topic decoder can be seen as a use case of stylized image captioning. We compare our framework with (4) MScap [23]. In our reimplementation, we regard topic as style.
Transformer-based image captioning We evaluate the state-of-the-art Transformer (5) OSCAR [37], which is one of the latest multi-modal approaches in vision-language task. We use its original pre-trained weights and finetune it on the SemArt dataset.
Text-summarization As a different perspective, we generate painting descriptions with text-based summarization methods. That is, given the knowledge articles retrieved in Table 2. Comparative evaluation. We compare our models against multiple alternatives. All the methods are trained on the Se-mArt dataset under equivalent conditions. Methods using external knowledge (LSA and Ours) use Wikipedia as knowledge source. GM:GreedyMatching, S-T: Skip-Thought, EA: EmbeddingAverage. Sec. 3.2, we summarize their content with (6) LSA [61], a language-independent algebraic method.
Ours We compare our method when using the (7) Topic Parallel Decoder and the (8) Topic Conditional Decoder. Table 2 shows the comparison results. We adopt a wide range of natural language evaluation metrics, including BLEU-4 [48], CIDEr [66], METEOR [13], ROUGE-L [39], GreedyMatching [53], Skip-Thought [34], and Em-beddingAverage [41]. We notice that (1) the scores are much lower than in traditional image captioning: e.g., the BLEU-4 score in MSCOCO is around 30 [67,73], while in SemArt it is less than 10; (2) the scores among different metrics shows very large variation. On one hand, this phenomenon shows that the proposed task is very challenging. On the other hand, these automatic metrics may not be appropriate to evaluate the richness and diversity of art description generation. Thus we also include the qualitative comparison results in Appendix E.
Firstly, our parallel decoder performs better than the conditional decoder. It is natural as the parallel decoder has larger capacity in network and can fit different topics in a specific manner. When comparing against the classic image captioning (NIC, SAT, and Att2in), our parallel decoder method outperforms them in 5 out of 7 metrics. The performance gain is mainly caused by the masked-sentencegeneration-filling schema that reduces the burden of the decoder to handle low frequency words and named-entities. In MScap, we directly let the model to conditionally learn the topic, without the help of masked-sentence-generationfilling, the performance even getting worse. With respect to the Transformer-based approach OSCAR, it performs very poorly in most of the metrics. We deduce this is caused by: 1) the scale of the training set, and 2) the use of Bottom-Up-Top-Down [1] regional features as image input, which are extracted by an object detector pre-trained in natural images and do not consider the visual domain-gap with paintings.
Finally, the text-summarization method LSA performs very poor in most of the metrics except in Skip-Thought, demonstrating that relying only on external knowledge without "looking" into the image is not the best approach to generate a description for a painting. This is because with summarization methods: 1) it is not possible to control the generation process for the different topics, 2) they heavily rely on the accuracy of the knowledge retrieval module, and 3) descriptions cannot be generated when the external knowledge of a painting does not exist. According to a random subset of 150 annotated images, 7 80.7% of the samples do not have a specific article on Wikipedia. However, 71.3% of the them do have artist article, which is useful to find knowledge but not enough for producing a relevant painting description on its own. Moreover, even when retrieving a non-relevant article, our method can generate correct sentences for the content and the form topics. The ratio of slots per topic on the SemArt dataset, i.e., average number of slots in content/form/context sentences, are 0.98/0.91/2.12, respectively, which shows that content and form require less external data than context. Finally, the artistic attributes predicted from the image are also used to fill the slots (Sec. 3.3), which may yield to meaningful information even when the retrieved knowledge fails.

Conclusion
We proposed the first multi-topic knowledgeable framework for art description generation. To generate multi-topic descriptions, we annotated an art description dataset with sentence-level topic labels. We explored this problem from multiple views, including proposing two types of topic decoder and experimenting with un/semi-supervised as well as supervised settings. Besides, we introduced the use of external knowledge to enhance the background information in the description of the painting. Comprehensive evaluation and comparison showed the effectiveness of our method, which we hope contributes to guide future research.

Appendix A. Illustration of Different Decoders
We present the details of different decoders in Fig. 5. The formalization is provided in the main paper. In the baseline decoder, the context visual feature is calculated by weighting feature vectors from each region of the image and fed into the LSTM in each time step. For simplification, we omit the hidden state initialisation and the attention-based context visual feature calculation in the figure. In the parallel decoder, N topic baseline decoders are employed as subdecoders. Apart from the visual feature, it uses the topic label as an additional input, which is responsible for selecting the corresponding decoder for the given topic. Finally, for the conditional decoder, the input topic label is first embedded into a topic embedding. Then, the topic embedding is concatenated to the visual feature and the previous hidden state for each time step. We omit the visual feature concatenation operation in the conditional decoder figure for simplification. After the whole decoding process, the generated masked sentence is fed to a topic classifier to ensure that the sentence belong to the correct topic.

Appendix B. Implementation Details
We implement all our models with PyTorch [49]. We optimize the topic decoder with the Adam [33] with a learning rate of 5 × 10 −4 , which decays at a rate of 0.8 every 10 epochs. The batch size is set to 32. We extract L = 14 × 14 with D = 2, 048 feature maps from the layer before the last pooling layer of a pre-trained ResNet101 [24]. For predicting artistic attributes, we use a four-branch attribute pre- dictor model [17]. The dimensions of the LSTM-based decoder's hidden states and word embeddings are fixed to 512 for all of the models discussed herein. In the topic conditional decoder, the dimensionality of the topic embedding is set to 20. DrQA and BERT hyperparameters are set as in [7] and [14], respectively. At test time, we employ the beam search for generating text, where a beam size of 5 is empirically selected for all the topic decoder variants.

Appendix C. Training Details
For the image encoder, we use a pre-trained ResNet [24] that does not need to be trained. For the decoders, the baseline decoder is trained as the standard captioning model [67], where the whole description is used as ground truth caption for an image. While during training the topic decoder, the ground truth description for an image is split into N topic parts. Sentences with the same topic label are appended together as a topic-specific description. In the parallel decoder, each sub-decoder is trained independently with its topic-specific description. In training the conditional decoder, the topic-specific description are selected according to the topic label input to the decoder. In the topic classifier part, we employ the continuous approximation technique proposed by Hu et al. [26] to avoid sampling words from a probability distribution, so that the decoder and classifier can be trained in an end-to-end manner. Not all the comments contain the three topics. During training, if a comment does not span the e.g., form topic, the form decoder is not trained with that image.
In the knowledge retrieval part, both attributes prediction model and object detection model are pre-trained. While the DrQA [7] knowledge retriever adapts a non-machinelearning method. Thus, no optimization is needed in this part. In the knowledge extraction and filling part, BERT is trained with art descriptions. The input is a masked sentence and a list of candidate words, where the masks are generated by replacing the named-entities with their entity type, and candidate words are the named-entities that being replaced. The ground truth is the original sentence before masking. Note that to avoid trivial solutions, the candidate words are extracted from the whole paragraph of description while the input sentence is one short sentence.

MScap
This painting depicts a rive landscape with skaters in the foreground. This painting is one of a series of views of the <unk>. The painting is signed and dated lower right.

Ours
The painting depicts a river landscape with skaters and a rowing boat in the foreground. This painting is a typical example of Beyeren's landscapes that he had to be seen in his own lifetime and he was a good example of his contemporaries. This painting is one of the earliest known works by Beyeren. The painting depicts a young woman in a white dress with a <unk> and a woman holding a sword. This painting is a fine example of the artist's late style and he was influenced by Salomon. This is one of the most important works by Eriksen in Russia.
The painting depicts a Dutch landscape with a shepherd and a horsedrawn cart in the foreground. This painting is a typical example of Martszen's work in the use of the composition and the use of light as in the foreground. This painting belongs to a series of two representing the ten paintings of the Spanish infantry.

Appendix D. Knowledge Retrieval Module Evaluation
For evaluating the knowledge retrieval module, we annotate a small number of paintings (150) with their correspondent Wikipedia article. Not all the images possess exact associated Wikipedia article. However, articles related to the painting's author or theme can also provide useful information. Considering these factors, we first prepare several candidate Wikipedia articles for each painting and annotate each article with one label out of the following five labels: • Correct the article is about the exact painting.
• Theme the article is related to the content of the painting, e.g. myth, person, event, concept, etc. • Author the article is about the author.
• Ambiguation the article is about a painting with the same name but not the exact one, i.e. created by another author. • Incorrect unrelated article. Among them, articles with Correct, Theme or Author labels are regarded as positive articles that can provide useful information, while Ambiguation and Incorrect correspond to negative articles. In total, we have annotated 450 articles for 150 paintings (3 articles for each painting).
We evaluate the accuracy of the knowledge retrieval module by comparing the sorted list of articles from our retriever with the annotated Wikipedia articles, and find the position in which the annotated article is returned. In this way, we measure recall at k (R@k) metric with different values of k (e.g., k = 1, 5, 10). R@k represents the percentage of samples whose annotated article is returned within the top k positions by our retriever. As we have different labels for the annotated articles, we calculate the metrics for the different type of articles. Table 3 shows the evaluation results using attributes and objects words as query, as in the main paper. We can observe that the useful articles from our retriever mostly come from the author articles. We have also explored to incorporate the generated masked sentences into the query, whose results are shown in Table 4. Comparing the two tables, we find that the incorporation of masked sentences has a negative impact in the knowledge retriever, as these sentences occupy a large proportion in the query but do not contain much specific information.

Appendix E. More Qualitative Results
Here we show the generated sentences by all the methods evaluated in the main paper and provide more qualitative results of our proposed method. Figure 6 shows, the qualitative comparison of different methods in Section 4.2. In Figure 7, three more examples of descriptions generated by our method are shown.