{'fraction_non_alphanumeric': 0.04446510610369722, 'fraction_numerical': 0.012032378035440822, 'mean_word_length': 4.422597864768683, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
The NTT DCASE2020 Challenge Task 6 system: Automated Audio Captioning with Keywords and Sentence Length Estimation

This technical report describes the system participating to the Detection and Classification of Acoustic Scenes and Events (DCASE) 2020 Challenge, Task 6: automated audio captioning. Our submission focuses on solving two indeterminacy problems in automated audio captioning: word selection indeterminacy and sentence length indeterminacy. We simultaneously solve the main caption generation and sub indeterminacy problems by estimating keywords and sentence length through multi-task learning. We tested a simplified model of our submission using the development-testing dataset. Our model achieved 20.7 SPIDEr score where that of the baseline system was 5.4.


INTRODUCTION
This technical report describes the system participating to the Detection and Classification of Acoustic Scenes and Events (DCASE) 2020 Challenge, Task 6: automated audio captioning [1]. Automated audio captioning (AAC) is an intermodal translation task when translating an input audio into its description using natural language [2][3][4][5][6]. In contrast to automatic speech recognition (ASR), which converts a speech to a text, AAC converts environmental sounds to a text. This task potentially raises the level of automatic understanding of sound environment from merely tagging events [7,8] (e.g. alarm), scenes [9] (e.g. kitchen) and condition [10] (e.g. normal/anomaly) to higher contextual information, for example, "a digital alarm in the kitchen has gone off three times." Our submission focuses on solving the indeterminacy problems in AAC which were tackled in our previous studies [3,11]. This indeterminacy can be broadly divided into the indeterminacy in (i) word selection [11] and (ii) sentence length [3]. The first problem is caused by that one acoustic event/scene can be described with several words, such as {car, automobile, vehicle, wheels} and {road, roadway, intersection, street} [11]. The second one is caused by that a sound can be explained in either short or long sentences, such as "noisy car sounds," or "a lot of cars are driving on the roadway and there are very loud engine noises" [11]. Such indeterminacy leads to a combinatorial explosion of possible answers, making it almost impossible to estimate the ground-truth and difficulty in training an AAC system.
Our strategy for solving these problems is to simultaneously estimate keywords and sentence length through multi-task learning framework. Figure 1 shows the overview of our system. The preprocessing stage involves rule-based keywords and sentence length extraction from the caption and metadata. The captioning DNN has  keyword estimation and a sentence length estimation branches, and estimates the ground-truth caption by integrating these results.

SYSTEM DESCRIPTION
This section describes the detail of our system. Since this paper is a technical report, we focus on describing the detailed implementation of the system. Effectiveness of each modules will be discussed in the workshop paper through ablation studies.

Pre-processing
Audio pre-processing: As acoustic feature, we used three logmel-spectrograms calculated from the time-domain input audio x. The first one was the log-mel-spectrogram of the input audio S ∈ R F ×Ts , where F and Ts are the number of mel-filterbanks and time-frames. The second and third ones were that of the harmonicpercussive source separation (HPSS) outputs, H ∈ R F ×Ts and P ∈ R F ×Ts . These three spectrograms were concatenated on the channel dimension X ∈ R 3×F ×Ts .
The hyper-parameters of the audio pre-processing are as follows. All audio samples were down-sampled at 22.05 kHz. The window-and hop-size of short-time Fourier transform (STFT) were 4096 and 2048 points, respectively. The number of mel-filterbank was F = 64. The hyper-parameters of the HPSS were default one of librosa.decompose.hpss [12].
Caption pre-processing: All captions were tokenized using the word tokenizer of the natural language toolkit (NLTK) [13] while removing punctuation. All tokens in the development dataset were then counted, and words that appeared more than five times were appended in the word vocabulary. The vocabulary size was C cap = 2144, which includes BOS, EOS, PAD, and UNK tokens. In addition, the sentence length L of each caption was counted. Also, was extracted using the keyword vocabulary which is discussed below. Meta pre-processing: Meta keywords were extracted from the file_name and keyword provided in the metadata csv file, using a keyword vocabulary which was manually created beforehand. The procedure of creating the keyword vocabulary is as follows. First, file_name and keyword were split at places at space and punctuation. Next, words that seem to be nouns, verbs, adjectives, and adverbs were converted to its lemma. Finally, all lemmas were counted, and lemmas that appeared more than ten times were appended in the keyword vocabulary, which is a hash table that maps the original word to its lemma. The vocabulary size was C key = 421. The keyword vocabulary was used to extract meta keyword m = {m k } Km k=1 and caption keyword c. Note that the procedure for creating the keyword vocabulary can be automated by using the part-of-speech (POS)-tagger and the WordNet Lemmatizer of the NLTK, however, we did this manually because their use is prohibited in this task.

Data augmentation
TF-IDF-based sample selection and data augmentation: Since the target metrics of this challenge is SPIDEr, we need to accurately predict captions which include low frequent words and topics. To deal with word and topic bias in the training dataset, we adopted two tricks for training sample selection based on inverse document frequency (IDF), and one trick for data augmentation based on term frequency (TF)-IDF [15].
The first trick is for selecting an audio sample x from the training dataset. First, we concatenated the five ground-truth captions corresponding to each x in the training dataset, and used as a "sentence". Then, we calculated IDFs for all words in all sentences, and calculated the average IDF of each sentence. Finally, each average IDF was normalized by the sum of the average IDF. We regarded the normalized IDF as the parameter of the Categorical distribution, and selected x based on this probability.
The second trick is for selecting a ground-truth caption w from five captions corresponding to the selected x. The basic strategy was the same as the first trick. First, we calculated IDFs of all words in the five captions. Here, note that the document was the five captions in contrast to the first trick. Then, we calculated the normalized IDF and used as the parameter of Categorical distribution, and selected the target caption w based on this probability.
Finally, we adopted the third trick which is the TF-IDF based word replacement [15] to augment text data. Random data cropping: To train our captioning DNN using minibatches, we adjusted the input length of audio sequence and text sequence using random cropping and padding. We set the input length of audio to 20 seconds (T = 216), and the number of words is N = 20. Thus, the inputs of the captioning DNN were X ∈ R 3×F ×T and w = (w1, ..., wN ). For X whose Ts was greater than T , a random crop was performed so that the time-length was T , and shorter ones were applied zero-padding. Similarly, if the sentence length was greater than N , words after the N -th word were cropped, and PAD tokens were added for shorter ones. Mix-up: After adjusting input length T and N , we used the mixup data augmentation. First, we drew a mixing parameter β from a beta distribution as β ∼ Beta(0.4, 0.4) where ∼ is sample drawing from the right-hand distribution. Then, two audio samples were mixed by multiplying β and (1 − β), respectively. Since text inputs  a set of class labels, direct mixing of w is not suitable. Thus, we mixed the embedded word tokens by multiplying the mixing parameters. Figure 2 shows the network architecture of the captioning DNN. The pink area in Fig. 2 is a basic sequence-to-sequence (Sec2Sec)based captioning model [14] using bidirectional long short-term memory (BLSTM)-LSTM. The encoder BLSTMs outputted the initial hidden state h ∈ R D and cell states c ∈ R D of the decoder LSTM, where D = 120 was the hidden dimension of the whole network. Then, the decoder LSTM estimated posterior probability of n-th word given the audio signal x and 1st to (n − 1)-th words p(wn|x, w1,...,n−1) by using embedded word tokens. In our submission, to solve the indeterminacy problems in AAC, we additionally used sub-blocks for keyword and sentence length estimation.

Model description
The following describes these sub-locks in detail.
Audio embedding block A: The input audio X was first passed to this block. This block embeded X into a feature space as A = A(X) ∈ R D×T a . As shown in Fig. 2 (c), this block consisted of three convolutional neural network (CNN)-blocks and two fully connected (FC)-blocks. The kernel size, stride, padding, and number of output channels of CNN were 3, 1, 1, and 64 for all CNN layers, respectively. The kernel size and stride of the 2D maxpooling were 2 and 2, respectively. Then, the output of CNN-blocks 64 × Fa × Ta was reshaped in to 64Fa × Ta, where Fa = F 2 3 = 8 and Ta = T 2 3 = 27, respectively. The reshaped output was passed to the first FC layer which converts R 64Fa×Ta to R D×Ta . Finally, the second FC-layer outputted A ∈ R D×T a .
Caption keyword estimation block C: This block estimated caption keyword probabilities of each keyword p cap ∈ [0, 1] C key from A as p cap = C(A). We expected that this block guides the audio embedding block so that its output includes information of the keywords of the ground-truth caption. As shown in Fig. 2 (d), this block consisted of a muti-head self-attention (MHSA) layer and a FC layer. The number of heads of MHSA was 4. The output shape of MHSA and FC layer were D × Ta and C key × Ta, respectively. Since the caption keyword has no time-labels, we aggregated the output by taking a maximum value in the time direction and outputted caption keyword probabilities of each keyword p cap .
Meta keyword estimation block M: This block estimated meta keyword probabilities p meta ∈ [0, 1] C key and its embedding M ∈ R D×Km from A as {p meta , M } = M(A). As shown in Fig. 2 (e), the base architecture was the same as the caption keyword estimation block, which consisted of MHSA and FC layers. The base architecture outputted meta keyword probabilities of each keyword p meta . To embed the estimated meta keywords into the feature space, first, we used the argsort function which returns the index on which p meta sorts in descending order. Then the top Km = 15 indices were selected as the estimated meta keyword {m k ∈ N} Km k=1 . Finally, these indexes were passed to the embedding layer to obtain the estimated meta keyword embedding M . After this block, A and M were concatenated as (M , A, M ), and it was passed to BLSTMs.
Sentence length estimation block L: This block estimated the sentence length probability p len ∈ [0, 1] L max and its embedding l ∈ R D l as {p len , l} = L(h, c), where L max = 20 is the maximum sentence length that we assumed. First, h and c were concatenated, and the first FC layer estemited p len from the concatenated feature. Finally, p len was passed to the second FC layer, and outputted l. After this block, l was concatenated to h and c, and used as the initial hidden and cell state of the decoder LSTM.
Attention block: Before calculating p(wn|x, w1,...,n−1) using the final FC layer, this block integrated the output of the LSTM H ∈ R (D+D l )×N and M and outputs M ∈ R (D+D l )×N by using three FC-layers, like an MHSA with a single head. Then the tanh activation was applied to M . Finally, it is added to H as H +tanh(M ) and passed to the final FC-layer to estimate p(wn|x, w1,...,n−1).

Loss functions
In order to train encoder/decoder and sub-blocks simultaneously, we designed loss function as a sum of multiple losses functions. In addition, since the input audio and text were augmented by the mix-up, the cost function was also calculated using the mix-up; each loss was calculated for each of the two original label data and mixed using mixing-parameters β and (1 − β). The following describes these loss functions in detail.
Word estimation loss: For word prediction, we used the crossentropy loss between wn and p(wn|x, w1,...,n−1). To avoid overfitting, we used label smoothing where smoothing factor was 0.1.
Caption/meta keyword estimation loss: The weighted binarycross entropy was used as the loss function for both caption/meta keyword estimation block as (1) Note that for all variables, we omitted the superscripts cap and meta which indicate whether the variable belongs to caption keyword or meta keyword. Here, the meanings of each variable are followings: zi is 1 when ground-truth keyword set includes i-th word and 0 otherwise, pi is the i-th value of the estimated posterior vector p, and λi and γi are the weight for i-th keyword as λi = (p(zi)) −1 and γi = (1 − p(zi)) −1 , respectively, where p(zi) is the prior probability of the i-th keyword calculated by p(zi) = # of c-th keyword in training samples # of training samples .
To balance this loss and other losses, we multiplied a weight (1 − 10 −4 ) s to this loss, where s is the number of training steps. Sentence length estimation loss: We used the softmax cross entropy between L and p len as the loss for the sentence length estimation block. To balance this loss and other losses, we multiplied a weight 10 −2 to this loss. Keyword co-occurrence loss: In order to prevent the decoder outputs the words which are obviously not related to the meta keywords, we used the keyword co-occurrence loss between words in a caption and its meta keywords. For example, when meta keywords are {car, sing, bird}, words not related to the keywords such as {people, children, talking, talk, speak} may not be included in the correct caption. To prevent the decoder outputs such words, we adopted a penalty based on the decoder outputs p(wn|x, w1,...,n−1). Before training, we created a hash-table of the co-occurrence lists; the keys of the hash-table are all keywords in the keyword vocabulary, and the element of each key is a list of the words that have co-occurred with the keyword in the training dataset. For example, in the case of meta keywords are {car, sing, bird} and ground-truth captions are {Cars are driving and birds are singing, A car passes by while birds are chirping and singing}, {cars, are, driving, and, birds, singing, a, passes, by, while, chirping} are added to the cooccurrence lists of car, sing, and bird. Then, in the training step, we added penalties of the decoder outputs to the whole loss value as where bi ∈ {0, 1} a binary mask where bi = 1 when none of all co-occurrence lists of the ground-truth meta keywords includes the i-th word, and otherwise bi = 0.

Beam search and test time augmentation
We used the beam search decoding for the word decision process from p(wn|x, w1,...,n−1). The beam size was 5, and n-gram blocking size was 2, i.e. a hypothesis in a beam was discarded if there was a bi-gram that appeared more than once within it. In addition, we used test time augmentation (TTA) for audio input. This is because the audio input was randomly cropped for limiting the time-length as T = 216 in training phase. If the length of audio input is changed in testing phase, it may have a bad influence on the batch normalization layers. Therefore, in testing-phase, we also randomly cropped and zero-padded the audio input so as to T = 216. We generated five input audios by this process, and took the average of five outputs of the decoder.

Training hyper-parameters
We used the AdamW [17] optimier with a constant learning rate 10 −4 . The minibatch-size was 48. We randomly splitted 2893 + 1045 samples in the development dataset into 3842 training samples and 96 validation samples. We used a DNN whose validation score was the best while 300 epochs training.

Submitted systems
We used a model ensemble to output the final results; each model in the ensemble outputted ln p(wn|x, w1,...,n−1), and we took the average of all log-probabilities in the beamsearch phase. The four submitted results were four types of different combinations of following models. where single means we did not used the HPSS (i.e. X = S), and param2 means two additional modification: (i) before adding the meta keyword estimation loss, we multiplied 0.8 to it as a loss weight. (ii) we did not used the second trick in minibatch sample selection, i.e. the target caption was selected with equal probability from the five ground-truth caption of an audio.

EVALUATION ON DEV-TEST DATASET
To give a sense of the accuracy of the submitted system, we tested a simplified Submission 1 on the development-test dataset of the Challenge. First, we conducted three unit tests for Model1, Model2, Model3, and Model4, and then evaluated the ensemble model as Ensemble. Although Ensemble is simpler than actual our challenge submissions, it should be useful for testing the performance of each model and the effectiveness of the ensemble. Table 1 shows the evaluation results. All models significantly outperformed the baseline system, and with these ensembles model achieved the SPIDEr score 20.7. Our model consists of a complex combination of various sub-blocks and cost functions. As a future work, we will conduct ablation studies to determine how each blocks/cost functions has affected.

CONCLUSIONS
This technical report described the system participating to the DCASE 2020 Challenge Task 6 [1]. Our submission focused on solving the indeterminacy problems in word selection and sentence length. We simultaneously solved the main caption generation and sub indeterminacy problems by estimating keywords and sentence length through multi-task learning. The SPIDEr score of our submission on the development-testing dataset was 20.7. Since our model consisted of a complex combination of various sub-blocks and cost functions, as a future work, we will conduct ablation studies for these modules.