{'fraction_non_alphanumeric': 0.02817456408235642, 'fraction_numerical': 0.004383804551275737, 'mean_word_length': 4.946983011130639, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Robust Deep Learning Enabled Semantic Communication System for Text

With the advent of the 6G era, the concept of semantic communication has attracted increasing attention. Compared with conventional communication systems, semantic communication systems are not only affected by physical noise existing in the wireless communication environment, e.g., additional white Gaussian noise, but also by semantic noise due to the source and the nature of deep learning-based systems. In this paper, we elaborate on the mechanism of semantic noise. In particular, we categorize semantic noise into two categories: literal semantic noise and adversarial semantic noise. The former is caused by written errors or expression ambiguity, while the latter is caused by perturbations or attacks added to the embedding layer via the semantic channel. To prevent semantic noise from influencing semantic communication systems, we present a robust deep learning enabled semantic communication system (R-DeepSC) that leverages a calibrated self-attention mechanism and adversarial training to tackle semantic noise. Compared with baseline models that only consider physical noise for text transmission, the proposed R-DeepSC achieves remarkable performance in dealing with semantic noise under different signal-to-noise ratios.


I. INTRODUCTION
Distinct from conventional wireless communications, which focus on reducing transmission symbol errors, semantic communication targets to extract and interpret the meaning behind symbols accurately [1]. Therefore, the optimization goal of semantic communication is to narrow the semantic gap between transmitted and received signals rather than lowing the bit error rate. Such a transmission goal determines that semantic communication is mainly applied for communications between agents, such as machine-to-machine communications or human-to-machine communications.
Recently developed semantic communication systems [2]- [7], leverage the substantial power of deep neural networks (DNNs) in semantic extraction to understand the meaning of texts. DeepSC [2] is a pioneer work on semantic communications that presents a novel and effective architecture for text semantic transmission. Works on semantic communications have also been extended to multiple tasks, such as speech transmission [3], [4], image transmission [6], [7], and visual question answering [5]. Most of these works take the impact Xiaoming Tao is the corresponding author. of various kinds of physical channel noise into consideration and use a joint source-channel coding scheme to combat the influence of physical channel noise.
However, besides physical channel noise, semantic noise can also affect the semantic communication system. The key factor that determines the performance of semantic communication is the fidelity of semantic information it extracts and processes, while semantic information may be disturbed by semantic noise.
On the one hand, the original text may contain grammatical errors or slight literal modifications, such as deletions, replacement, order reversion, etc. These literal changes in texts will incur semantic distortion and obstruct the subsequent semantic understanding and interpretation [8]. For example, it is easy to mislead the model by adding a punctuation or character to a word of the text. [9].
On the other hand, due to the limited generalization ability, DNNs-based systems are vulnerable to malicious attacks. A slight perturbation added to input signals can render models misunderstand their semantics [10]. Consequently, the wrong decision will be made. For example, adding unperceived noise to a picture can deceive a classification model [11]. Analogously, adding noise to the embedding representation of a text may also affect the semantic extraction and result in a misunderstanding of the text [12].
Semantic noise could cause semantic ambiguity and make it hard for receivers to convey the underlying meaning of the transmitted text. Conventional communication systems are unable to handle such errors, because they are optimized at the symbol level. However, semantic communication systems are expected to overcome these disturbances and recover the original meaning from the modified text due to their semantic understanding ability.
In this paper, distinct from well-discussed physical noise on wireless channels, we explore different forms of semantic noise and establish a robust semantic communication system named R-DeepSC to effectively eliminate the impact of different kinds of semantic noise in text. To the best of our knowledge, this paper is the first to comprehensively explore semantic noise in text transmission. The detailed contributions of this paper are summarized as follows. • We categorize semantic noise in communications as literal modifications and adversarial noise. To combat the semantic noise, we propose a robust deep learning enabled semantic communication system named R-DeepSC. • For the literal semantic noise, we tailor the transformerbased model and present a calibrated self-attention mechanism for error correction to ensure the semantic fidelity. • For the adversarial semantic noise, we adopt an adversarial training method to train the system. We experimentally verify the effectiveness of the R-DeepSC in resisting different forms of semantic noise.
The remaining parts of this paper are organized as follows. Section II introduces various kinds of semantic noise and our anti-noise methods in detail. The experiment results are shown and discussed in Section III. Section IV concludes the paper.

II. SEMANTIC COMMUNICATION SYSTEM MODEL
In this section, we consider a semantic communication system with physical noise and semantic noise. Fig. 1 is the architecture of a semantic communication system. The transmitter conducts semantic encoding and channel encoding, while the receiver performs the corresponding decoding. Besides physical noise, N P , literal semantic noise, N L , and adversarial semantic noise, N A , could affect the considered semantic communication system.
The source text S can be affected by the literal semantic noise, N L , which is defined as errors in S, such as substitutions, deletions, etc. The literal semantic noise not only makes it difficult for humans to understand the underlying meaning of the text, but also incurs semantic distortions for semantic encoding. The text with literal semantic noise is given by F(S, N L ), where F(·) is a noise-adding function simulating the expression habits of users or vulnerable AI-assisted transmission environment, such as a speech recognition system. Literal semantic noise ratio is defined as the proportion of erroneous words in a sentence.
We denote the input text of the system as S, S = {s 0 , s 1 , · · · , s L }, where s i is the i-the word. After S passes the one-hot encoder and the embedding layer, the embedding vector X embed is represented as where O d (·) is the one-hot encoder according to dictionary d and E γ (·) is the embedding layer with the parameter set γ.
The architecture of the transmitter is illustrated in Fig.2. During this process, the one-hot encoder can hardly be affected by undetectable interference due to its natural sparsity. Conversely, the adversarial semantic noise, N A , which is a slight perturbation added to the embedding vector, X embed , may cause semantic misunderstanding. By considering the adversarial semantic noise, the transmitted signal is given by where C ϕ (·) is the channel encoder with the parameter set ϕ, and S η (·) is the Seq2Seq encoder with the parameter set η. The received signal, Y , can be represented as where H represents the fading channel and N P ∼ CN (0, σ 2 n ). By utilizing the channel decoder and the semantic decoder, the received textŜ can be represented aŝ where C −1 ζ (·) is the channel decoder with the training parameter set ζ, and S −1 δ (·) is the semantic decoder with the training parameter set δ.
The goal of this system is to minimize the semantic gap between transmitted text, S, and reconstructed text,Ŝ. By representing the transmitter and receivers as neural networks, the loss function developed in DeepSC [2] to train the system is given by

III. ANTI-SEMANTIC NOISE METHODS AND PERFORMANCE METRICS
To combat semantic noise and maintain the semantic fidelity of the system, we propose a robust deep learning enabled

A. Calibrated Self-Attention Mechanism
Literal modifications can be operated at the character and word levels. By adopting a spelling-check method, characterlevel errors can be removed effectively [13]. Hence, we mainly focus on word-level literal semantic noise in this paper. Prior efforts have been made to solve the error correction problem from a data or model perspective [14], [15]. A novel detectcorrect framework was established to address the Chinese error-correction problem [14]. [15] handled the grammatical error correction at the data level by leveraging a dynamic mask to generate error-correct examples for training.
For semantic communications, to avoid errors from affecting semantic information, less attention should be paid to erroneous tokens when calculating semantic representation vectors. However, the self-attention mechanism is unable to realize this goal due to the absence of error probability information. To cope with this problem, a detection net is added to infer the error probability of each token.
The architecture of the semantic encoder developed in R-DeepSC is illustrated in Fig. 3. The number of layers in the Transformer enabled Seq2Seq encoder is denoted as N . A detection net, which consists of a GRU and a linear layer, is added to the original semantic encoder of DeepSC. A calibration matrix, C, is obtained based on the output of the detection net. The attention score is calibrated by C to ensure that more attention is devoted to unmistakable tokens.
The calibrated attention score can be represented by where × is element-wise product, Q, K, V , d k is query, key, value and embedding dimension of the semantic encoder. To make the system robust to the literal semantic noise, we propose a new loss function to train the neural network of the developed R-DeepSC, which is given by where L CE is the cross-entropy loss, L M I is the mutual information, L BCE is the binary cross-entropy loss, P is the error probability matrix of tokens that is predicted by the detection net, and label is the ground truth of the error probability. The proportions of L M I and L BCE in the loss function can be controlled by parameters α and β. The loss function is utilized to optimize parameters, including ϕ, η, ζ, δ. In this loss function, L CE aims to make the transmitted text, S, and the received text,Ŝ, as similar as possible, while the L M I maximizes the channel capacity by maximizing the mutual information between the transmitted signal, T X , and the received signal, R X . L BCE loss is applied to train the system for error probability prediction, which is an input of calibrated self-attention.

B. Adversarial Training
For the adversarial semantic noise, adversarial training methods, such as fast gradient sign method (FGSM) [16] and fast gradient method (FGM) [17], were applied to eliminate its inference. [18] has discussed the processing of adversarial noise in semantic communications for images. However, these efforts have not yet been utilized to improve the robustness of semantic communication systems for text transmission.
We take the advantage of adversarial training, which is able to productively improve the robustness of deep learningbased systems, to deal with the adversarial semantic noise. The adversarial training searches for the semantic noise, N A , that can fool deep models by maximizing the loss, while parameters of the system are updated to overcome the impacts of N A . We denote L(·) as the loss function for adversarial training, which could be set as L total or part of L total . An adversarial training process is typically formulated as min ϕ,η,ζ,δ where D, N are the train set and the distribution space of N A . The critical problem of adversarial training is finding the most sensitive semantic noise for the system. The adversarial semantic noise, N A , can be explored by the FGM. The FGM generates N A by where is the normalization factor. The adversarial semantic noise can be obtained using back propagation. The adversarial semantic noise added to the embedding layer could affect the semantic fidelity of semantic communication systems, the adversarial training is able to enhance the robustness of the semantic communication system. In this paper, the FGM method is adopted to improve the robustness of the semantic communication system. The training set is augmented with adversarial examples X embed + N A that are crafted by FGM and the system model is trained to against the adversarial noise.

C. Performance Metrics
Compared with conventional communication systems, metrics, such as bit-error rate and symbol-error rate, are unable to measure the performance of semantic communication systems well. For semantic communications, it is necessary to consider whether there is a semantic gap between the shared text and the received text. Hence, we use the BLEU score [19] and the BERT SCORE [20] to describe the performance of the system comprehensively, which are detailed in the following.
1) BLEU Score: The BLEU utilizes the n-gram matching criterion to evaluate the quality of a received text. We denote C k as the number of the k-th word for the n-gram text, W n as the weight of the n-gram precision, and BP as the penalty index. The BLEU score is obtained as follows.
Particularly, BP is defined as where l R is the length of the received text, and l T is the length of the transmitted text. The value of the BLEU score is between 0 and 1, and the higher score implies greater sentence similarity. The BLEU score is effective but it only evaluates the similarity in the literal variation, rather than the semantic difference. Therefore, we also use BERT SCORE as the metric to depict the semantic similarity between two sentences.
2) BERT SCORE: The BERT SCORE obtains the semantic similarity from a similarity matrix and applies different weights to words according to their corresponding semantic importance. Thus, the semantic similarity evaluated by BERT SCORE correlates well with human judgments.
We assume that the corresponding BERT representation vector of transmitted text S is T 1 , T 2 , . . . , T n , and representation vector of the received textŜ is R 1 , R 2 , . . . , R m . The importance weight function idf (·) can be obtained by where The precision of the BERT SCORE between the transmitted text and the received text can be obtained as Then, the BERT SCORE is scaled to a larger interval using the following transformation to make it more readable bŷ where b is a scale factor. The rescaled BERT SCORE is between -1 and 1, and a higher score implies greater similarity between the compared sentence pair.

IV. NUMERICAL RESULTS
In this section, we conduct experiments to evaluate our developed R-DeepSC under various forms of semantic noise.

A. Corpus and Baseline Models
Europarl [21] has been adopted as our data set, which is based on proceedings of the European Parliament in 11 languages. We have selected Europarl in English, which contains 98, 751 sentences, as the transmitted corpus. 4 kinds of errors have been added to each sentence in this corpus randomly, including replacement, random mask, insertion, and verb errors.
This paper chooses two systems as comparisons. One is the DeepSC based on deep learning, and another one is a conventional communication system that uses Huffman codes for source coding, the Reed-Solomon (RS) codes for channel coding, and 64-QAM for modulation.
We evaluate system performance under different channel environments, including additive white Gaussian noise (AWGN) channels, and Rayleigh fading channels. R-DeepSC is robust to semantic noise by conducting adversarial training with the FGM and utilizing a calibrated self-attention mechanism. that when SNR is below 12 dB, the conventional communication system using Huffman coding and RS coding has a great performance decline in terms of BLEU and BERT SCORE. When SNR increases to 18 dB, although the BLEU score of the conventional system gradually increases to nearly 80%, there is still a non-negligible performance gap between the conventional approach and deep learning-based methods, such as the R-DeepSC and DeepSC.

B. Experimental Results
The conventional system is unable to correct semantic errors due to the lack of semantic perception, so the BLEU score can hardly exceed 80%. While semantic communication systems extract semantic information, they can correct erroneous text to some extent. Among these semantic communication systems, our proposed R-DeepSC achieves superior performance under different SNRs. These results demonstrate that the semantic communication system can mitigate semantic distortion during transmission, while the R-DeepSC outperforms other methods.
In addition, the effectiveness of the FGM is validated. For BLEU score, DeepSC trained with the FGM (labelled as DeepSC+FGM) performs better. Meanwhile, as shown in Fig. 5, if we measure the system performance with the BERT SCORE, which calculates the semantic similarity, the DeepSC trained with the FGM shows the same tendency in Rayleigh fading channels. When SNR is lower than 0 dB, the FGM can hardly promote the system's performance because the distortion is too severe. As SNR increases, the FGM can improve the semantic fidelity of decoded texts effectively.
Moreover, we conducted experiments in scenarios with different levels of literal semantic noise. Fig. 6 shows the results trained under different literal semantic noise ratios. Fig. 6(a) presents that although the semantic fidelity obtained by the semantic communication system decreases when the literal semantic noise ratio increases, our proposed R-DeepSC yields remarkable performance under Rayleigh fading channels. At the same time, Fig. 6(b) shows that the semantic fidelity of R-DeepSC decays more slowly as the proportion of the literal semantic noise in corpus increases to 60%, which indicates that our method is indeed semantic noise-robust.
An example of the decoded text is shown in Table I. About 20% words of the sentence are modified by literal errors that incur semantic distortion. We can see that most errors in texts can be corrected after being transmitted by R-DeepSC and the original semantics of the text are restored. The literal semantic noise, such as verb errors, and insertions, can be eliminated effectively, while some trivial information is filtered. For example, the name "Emma Bonino" is interpreted as "Bonino", but this modification can hardly affect its underlying meaning. In summary, the proposed R-DeepSC, which yields remarkable performance compared with other systems, can effectively correct semantic distortions caused by modifications in texts and adversarial noise. This performance improvement not only comes from the developed architecture and calibrated self-attention mechanism of R-DeepSC, but also from taking advantage of the adversarial training.

V. CONCLUSION
In this paper, we have proposed a robust semantic communication system, which combats different forms of semantic noise and improves the robustness under various wireless environments. In particular, we have elaborated on literal semantic noise and adversarial semantic noise in semantic communication systems. For the literal semantic noise, we have developed a novel semantic encoder architecture and calibrated self-attention scheme that leverages the semantic information extracted by the semantic encoder to correct literal errors. Experiments show the effectiveness of our proposed R-DeepSC when the corpus is erroneous. For the adversarial semantic noise, we have adopted the adversarial training method to find perturbations that disturb the semantic communication system mostly and train our system to resist these perturbations. The experimental results demonstrate that eliminating adversarial semantic noise can improve the performance of semantic communication systems under different SNRs.