{'fraction_non_alphanumeric': 0.029816302040062772, 'fraction_numerical': 0.014308132557924859, 'mean_word_length': 4.7383474576271185, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Named Entity-Factored Transformer for Proper Noun Translation

Subword-based neural machine translation decreases the number of out-of-vocabulary (OOV) words and also keeps the translation quality if input sentences include OOV words. The subword-based NMT decomposes a word into shorter units to solve the OOV problem, but it does not work well for non-compositional proper nouns due to the construction of the shorter unit from words. Furthermore, the lack of translation also occurs in proper noun translation. The proposed method applies the Named Entity (NE) fea-ture vector to Factored Transformer for accurate proper noun translation. The proposed method uses two features which are input sentences in subwords unit and the feature obtained from Named Entity Recognition (NER). The pro-posed method improves the problem of non-compositional proper nouns translation included a low-frequency word. According to the experiments, the proposed method using the best NE feature vector outperformed the baseline sub-word-based transformer model by more than 9.6 points in proper noun accuracy and 2.5 points in the BLEU score.


Introduction
Recent advances in neural machine translation (NMT) have made machine translation (MT) systems useful in practical applications. However, translation of proper nouns still remains difficult in spite of its importance in practice. Proper nouns are sometimes processed as out-of-vocabulary (OOV) words in MT systems due to the limitation of the vocabulary size and data sparseness. Approaches for proper noun translation can be divided roughly into two approaches: the use of hand-crafted bilingual lexicon as the external knowledge and the use of subwords. The former approach uses a bilingual proper noun dictionary to translate proper nouns. Okuma et al (2008) proposed replacement-based proper noun translation. Their method uses a bilingual dictionary whose entries are associated with proper noun classes to replace a proper noun with another surrogate proper noun that frequently appears in the training corpus. Another method called lexically constrained decoder (LCD) (Hokamp et al., 2017) guarantees that proper nouns are translated into the target language sentence constrained by a bilingual dictionary (Chen et al., 2020, Chousa et al., 2021. It extends the beam search algorithm to find the hypothesis that contains all of the proper nouns (Hokamp et al., 2017). The dictionary-based approach works well only if the proper nouns to be translated are included in the bilingual dictionary and requires efforts for developing the dictionary.
In NMT, the subword-based approach is widely used. Sennrich et al (2016) proposed the use of subwords to decompose a word into shorter units. The method decreases the number of OOV words and keeps the translation quality if input sentences include OOV words. However, the subword-based NMT does not always work on a proper noun translation due to wrong compositions of subword translations.
In this paper, we propose a method for NMT focusing on the proper noun translation using Factored Transformer with named entity (NE) features. The proposed method only uses a parallel corpus and an NE recognition (NER) model as external knowledge. With respect to named entities, the use of a class-based language model was proposed to solve the problem of data sparseness in the field of automatic speech recognition (ASR) research field (Yamamoto et al., 1999(Yamamoto et al., , 2004. This idea was extended to MT (Tonoik et al., 2005, Yasuda et al., 2017. This approach improved the translation performance for unknown and low-frequency words by using high-frequency surrogate words in the same category.
The main focus of this paper is proper noun translation in the subword-based NMT using Factored Transformer and NE features from NER without a bilingual dictionary.

Proposed Method
We propose the use of NE features as linguistic factors of Factored Transformer for accurate proper noun translation.

Named Entity Feature Vector
NE features obtained from NER on a source language sentence are injected into the embedding or encoder layer, as a factor. We can use two types of NE features: a one-hot NE vector and an NE probability distribution vector. We extract the NE features from an input source language sentence by the following steps. 1. Apply word segmentation into an input source language sentence. 2. Apply subword segmentation onto the word-segmented input. 3. Apply part-of-speech (POS) tags to the word-segmented input using a POS tagger. 4. Recognize NE in the POS-tagged input sentence to obtain one-hot NE vectors or NE probability distribution vectors. 5. Align those NE feature vectors with the subwords composing corresponding words. Here, a one-hot NE vector represents a 1-best NE category while an NE probability distribution vector represents the ambiguity of NE categories.

Factored Transformer Architecture
We propose a Factored Transformer model that uses two factors: subwords and NE features. We present two types of NE features and the two model variants in factor-injecting layers, as shown in Fig.1. 1-encoder model ( Fig. 1(a)): Each factor has its own embedding layer. The embedding vectors are summed up together with the corresponding positional encoding vector and sent to the following encoder layer. The rest of the model remains unchanged from the vanilla Transformer. Fig. 1 (b)): Each factor has its own encoder in addition to the embedding layer. The outputs from the encoders are summed up and used as encoder outputs. The rest of the model remains unchanged.

Experimental Settings
We conducted Japanese-to-English and English-to-Japanese MT experiments to compare the performance of the proposed method with a standard NMT method.

Named Entity Recognition Model
For the Japanese-to-English experiments, we used an NER model based on a pre-trained BERT model and fine-tuned it using Japanese NER training data generated by using the method presented by Takai et al., (2018).  (Takai et al., 2018) in Japanese-to-English experiments, we used 10 million Japanese and English part sentences of JParaCrawl. We extracted sentence pairs including proper nouns by using tagger and POS. Here, we used Sudachi 2 for Japanese morphological analysis to find proper nouns. We chose 1,000 sentence pairs containing proper nouns for the NER training, based on the sentence pair scores (Morishita et al., 2020).

MT Models
We used Transformer and Factored Transformer models for NMT, with 6-layer encoders and decoders. The configurations of the models and their training were mostly the same as those of the vanilla Transformer, but we used different settings on the hyperparameters as shown in the following Table 2.
We used SentencePiece (Kudo et al., 2018) with a subword unigram model for the subword tokenization. We used Sudachi and Moses 3 as Japanese and English POS taggers.

Training and Dev. Data for MT Models
Details of the corpus for the NMT models are shown in Table 3. For the Japanese-to-English experiments, we used a part of 10 million Japanese-to-English sentence pairs in JParaCrawl (Morishita et al., 2020) for the training of the NMT models. We chose 160,000 sentence pairs that contain proper nouns, have sentence pair scores higher than 0.786, and shorter than 250 subwords. For the English-to-Japanese experiments, we used all the 10 million sentence pairs in JParaCrawl as a training data set due to the effectiveness of the different conditions from the Japanese-to-English one: language pairs and amount of training data.
WMT 2020 development set 4 was used as the development set for all the NMT models.

Evaluation Data
Details of the evaluation data are shown in Table 4.
For the Japanese-to-English, we used an evaluation dataset of 271 sentences containing a single proper noun. It was collected through field experiments with taxis in Japan and was translated manually. The data consisted of conversations between taxi drivers and travelers. For the Englishto-Japanese task, we used WMT 2020 Test set.

Compared Methods
We compared the following NMT models:

Evaluation metrics
We used BLEU (Papineni et al., 2002) as a translation quality metric. We also evaluated proper noun translation accuracy (PRPacc); i.e., the percentage of proper noun words that correctly translated over the entire test set.

directions max token size max epoch
J-E 7,300 60 E-J 7,300 33    Table 5 shows the results. In Japanese-to-English, the proposed 1-encoder models were worse than the baseline, but the 2-encoders models outperformed the baseline. The results by the 2encoder model with NE probability distributions showed the best performance, outperformed the baseline by 9.6 points in PRPacc and 2.5 points in BLEU. In English-to-Japanese, however, the 1encoder models outperformed the baseline. The improvement in BLEU and PRPacc was smaller than that in Japanese-to-English. This may be due to the difference in the training data sizes; the English-to-Japanese MT models were trained using the 60 times larger parallel corpus. Another possible reason is the difference in the degrees of difficulty in these domains; WMT News task would be more difficult than the taxi conversation.

Results
With respect to proper noun translation, the lack of a specific treatment of proper noun translation in the baseline resulted in worse performance than the proposed method. Translation examples are shown in Table 5. The 2-encoders models worked well on two types of proper nouns: the non-compositional proper noun of Table 6 (1), and the combination with proper nouns and general noun of Table 6 (2). This result can be assumed that the factor of NE feature vector directly works on the proper noun translation better in the near decoder.
As shown in Table 6, 2-encoders with NE probability distributions have better performance than 2-encoders with NE one-hot vector performance. Expression of the ambiguity of proper nouns in the NE probability distributions method influent on not only proper noun translation but also the surrounding words of a proper noun.

Conclusions
We proposed a method to enhance accurate proper noun translation using subword-based NMT by Factored-Transformer and NE features. The NE feature vectors are injected into Factored Transformer model as factors together with subwords. In the Japanese-to-English experiments using a small bilingual training corpus, the proposed method using the best NE feature vector outperformed the baseline sub-word-based transformer model by more than 9.6 points in proper noun accuracy and 2.5 points in the BLEU score. It also showed some improvements in the English-to-Japanese experiments using a largescale bilingual corpus.
In future work, we will work on automatic clustering of proper nouns instead of given NE categories.