{'fraction_non_alphanumeric': 0.03052336044921172, 'fraction_numerical': 0.00763084011230293, 'mean_word_length': 4.881456392887384, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
An Experimental Study of LSTM Encoder-Decoder Model for Text Simplification

Text simplification (TS) aims to reduce the lexical and structural complexity of a text, while still retaining the semantic meaning. Current automatic TS techniques are limited to either lexical-level applications or manually defining a large amount of rules. Since deep neural networks are powerful models that have achieved excellent performance over many difficult tasks, in this paper, we propose to use the Long Short-Term Memory (LSTM) Encoder-Decoder model for sentence level TS, which makes minimal assumptions about word sequence. We conduct preliminary experiments to find that the model is able to learn operation rules such as reversing, sorting and replacing from sequence pairs, which shows that the model may potentially discover and apply rules such as modifying sentence structure, substituting words, and removing words for TS.


Introduction
Text Simplification (TS) aims to simplify the lexical, grammatical, or structural complexity of text while retaining its semantic meaning. It can help various groups of people, including children, non-native speakers, and people with cognitive disabilities, to understand text better. The field of automatic text simplification has been researched for decades. It is generally divided into three categories: lexical simplification (LS), rulebased, and machine translation (MT) [1].
LS is mainly used to simplify text by substituting infrequent and difficult words with frequent and easier words. However, challenges exist for the LS approach. First, a great number of transformation rules are required for reasonable coverage; second, different transformation rules should be applied based on the specific context; third, the syntax and semantic meaning of the sentence is hard to retain. Rule-based approaches use hand-crafted rules for lexical and syntactic simplification, for example, substituting difficult words in a predefined vocabulary. However, such approaches need a lot of humaninvolvement to manually define these rules, and it is impossible to give all possible simplification rules. MT-based approach regards original English and simplified English as two different languages, thus TS is the process to translate ordinary English to simplified English. Neural Machine Translation (NMT) is a newly-proposed deep learning approach and achieves very impressive results [2; 3; 4]. Unlike the traditional phrased-based MT system which operates on small components separately, NMT systems attempt to build a large neural network such that every component is tuned based on the training sentence pairs.
NMT models are types of Encoder-Decoder models, which can represent the input sequence as a vector, and then decode that vector into an output sequence. In this paper, we propose to apply Long Short-Term Memory (LSTM) [5] Encoder-Decoder on TS task. And we show the LSTM Encoder-Decoder model is able to learn operation rules such as reversing, sorting, and replacing from sequence pairs, which are similar to simplification rules that change sentence structure, substitute words, and remove words. Thus this model is potentially able to learn simplification rules. We conduct experiments to show that the trained model has a high accuracy for reversal, sort-ing, and sequence replacement. Also, the word embeddings learned from the model are close to its real meaning.

Related Work
Automatic TS is a complicated natural language processing (NLP) task, it consists of lexical and syntactic simplification levels. Usually, hand-crafted, supervised, and unsupervised methods based on resources like English Wikipedia (EW) and Simple English Wikipedia (SEW) [6] are utilized for extracting simplification rules. It is very easy to mix up the automatic TS task and the automatic summarization task [7; 8]. TS is different from text summarization as the focus of text summarization is to reduce the length and redundant content.
At the lexical level, [9] proposed an lexical simplification system which only requires a large corpus of regular text to obtain word embeddings to get words similar to the complex word. [10] proposed an unsupervised method for learning pairs of complex and simpler synonyms and a context aware method for substituting one for the other. At the sentence level, [11] proposed a sentence simplification model by tree transformation based on Statistical Machine Translation (SMT). [12] presented a data-driven model based on a quasi-synchronous grammar, a formalism that can naturally capture structural mismatches and complex rewrite operations.
The limitation of aforementioned methods requires syntax parsing or hand-crafted rules to simplify sentences. Compared with traditional machine learning [13; 14] and data mining techniques [15; 16; 17], deep learning has shown to produce state-of-the-art results on various difficult tasks, with the help of the development of big data platforms [18; 19]. The RNN Encoder-Decoder is a very popular deep neural network model that performs exceptionally well at the machine translation task [2; 4; 3]. [1] proposed a preliminary work to use RNN Encoder-Decoder model for text simplification task, which is similar to the proposed model in this paper.

The Model
In this section, we first briefly introduce the basic idea of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM), then describe the LSTM Encoder-Decoder model.

Recurrent Neural Network and Long Short-Term Memory
RNN is a class of Neural Network in which internal units may form a directed cycle to demonstrate the state history of previous inputs. The structure of RNN makes it naturally suited for variable-length inputs such as sentences. For a sequence data (x 1 , ..., x T ), where at each t ∈ {1, ..., T }, the hidden state h t of the RNN is then updated via Where f is the activation function. However, the optimization of basic RNN models is difficult because its gradients vanish over long sequences. LSTM is very good at learning long range dependencies through its internal memory cells. Similar to RNN, LSTM updates its hidden state sequentially, but the updates highly depend on memory cells containing three kind of gates: the forget gate c t decides how much remembered information to forget, the update gate i t decides how to update remembered information, the output gate o t decides how much the remembered information to output.
Recent works have proposed many modified LSTM models such as the gated recurrent unit (GRU) [3]. However, [20] showed that none of the LSTM variants can improve upon the standard architecture significantly. In this paper, we use the standard LSTM structure in our model.

LSTM Encoder-Decoder Model
Given a source sentence X = (x 1 , x 2 , ..., x l ) and the target (simplified) sentence Y = (y 1 , y 2 , ..., y l ), where x i and y i are in the same vocabulary, l and l are the length of each sentence. Our goal is to build a neural network to model the conditional probability p(Y |X), then train the model to maximize the probability.
We show our LSTM Encoder-Decoder model in Figure 1. This model uses onehot representation of words in the sequence in the input layer, and converts it to a 300dimensional vector in the following embedding layer. We find that adding an embedding layer can significantly improve performance when the vocabulary becomes large. Then we feed word embeddings through two LSTM layers, and get a vector representation of the input sequence after finishing reading all the words. Finally, we decode this vector to output sequence through two LSTM layers and one output embedding layer.
Let us take the input sentence "Man with high intelligence" as a difficult sentence, the output sentence "a very smart man" as the simplified sentence, and represent the pair of sentences as a pair of word indices (we made up some indices here), we have: We only apply sorting, reversing, and replacement to the indices to simplify a sentence, where sorting and reversing could be highly related to changing the structure of a sentence or simplifying a grammar, replacement could be highly related to lexical simplification or removing redundant words. Motivated by this observation, we conduct experiments to show the LSTM Encoder-Decoder is able to learn these three rules automatically, and thus can potentially perform text simplification.

Experiments
In this section, we conduct experiments to show LSTM Encoder-Decoder can perform basic operations for sequence data. Intuitively, TS should include operations like replacing difficult words with easier words, removing redundant words, simplifying syntax structure by changing word order, etc. In the following experiments, we show that a very basic LSTM Encoder-Decoder model is able to reverse, sort, and replace the elements of a sequence.
We implement the LSTM Encoder-Decoder in Keras [21]. The model contains two LSTM layers for both the encoder and the decoder, the output is fed into a softmax layer. RMSprop [22], which generates its parameter updates using a momentum on the rescaled gradient, was used as the optimizer in out experiment since it achieves the best performance compared to other optimization methods. We utilized early stopping with patience 5 to avoid over-fitting.
We generate sequences of random integer numbers with length 25 as inputs, since sentences are usually less than 25 words.

Reverse
We first conduct experiments to show that the LSTM Encoder-Decoder can reverse a sequence after training on a large set of sequence pairs (X, Y ), where The results are given in Table 1. V, H, E represent the vocabulary size, the number of hidden neurons in the LSTM layer, and the training epoch, respectively. The size of our training data is extremely important for the model. As shown in Table 1, the performance decreases significantly if we reduce the size of our training set from 135k to 9k. The size of the vocabulary also influences the performance. A larger vocabulary requires more training data and more hidden neurons in the LSTM layers. By increasing the number of neurons in the LSTM layers from 128 to 256, we produce a higher capacity model with more neurons that can be trained with fewer epochs and achieve a higher accuracy.
In general, this model can reverse an input sequence with higher than 90 percent accuracy given enough training data. On the other hand, it shows that LSTM is proficient at memorizing long-term dependencies.

Sort
Even though Neural Programmer-Interpreters (NPI) [23], a recent model, can represent and execute programs such as sorting and addition, the LSTM Encoder-Decoder is much simpler and more light-weight compared to NPI. In the following experiment, we show that the LSTM Encoder-Decoder is able to sort a sequence of integers.
The datasets consist of sequence pairs (X, Y ), where We show the results in Table 2. Similarly, the size of the vocabulary, training data, and neurons influence the performance. It is harder to train if we increase the vocabulary to 1000, but the model can still learn the sorting rule with a high accuracy if provided enough training data. We extracted the hidden states of the embedding layer from the trained model of the vocabularies of the size 10 and 100. Since the embedding of each word is a 300-dimensional vector, we use Principle Component Analysis (PCA) for dimension reduction and visualize the word embeddings in Figure 2. Interestingly, we find the learned embedding correctly represents the relationship between each word in 1-dimensional and 2dimensional space. Noted that even though the inputs are integer "numbers", these numbers are actually symbols, or word indices Similarly, if we provide the model with difficult and simple English sentence pairs, the LSTM Encoder-Decoder may be able to learn how to order words to make the sentence simpler.

Replace
We next show that the LSTM Encoder-Decoder can replace words in a sequence. For the sequence pairs (X, Y ), where We let n = 2, 20, 200 when |V | = 10, 100, 1000. We only keep the top 20 percent of words in the vocabulary, and use  Table 3. Compared with the sorting operation, the replacement operation is easier to train when the vocabulary is large, or the size of training data is not large.

Combine Three Operations
We have shown that the LSTM Encoder-Decoder can work well on the reversing, sorting and replacement operations separately, but in reality, a sentence is usually simplified by a complex combination of these three different rules. Therefore, we combine the three operations together to see if this model can still discover the mapping rules between sequences.
So the data is sequence pairs (X, Y ), where Y is obtained by performing modulo for each index in X first, then sorting and reversing. The results are shown in Table 4. The LSTM Encoder-Decoder continue working very well as expected, and even as good as each of the operations alone. Therefore, the LSTM Encoder-Decoder can easily discover mapping patterns of combined operations between sequences, thus it may potentially find complicated simplification rules.

Conclusion and Future Work
In conclusion, we find that the LSTM Encoder-Decoder model is able to learn operation rules such as reversing, sorting, and replacing from sequence pairs, which shows the model may potentially apply rules like modifying sentence structure, substituting words, and removing words for text simplification. This is a preliminary experimental study in solving the text simplification problem using deep neural networks. However, unlike the machine translation task, there are very few text simplification training corpora online. So our future work includes collecting complex and simple sentence pairs from online resources such as English Wikipedia and Simple English Wikipedia, and training our model using natural languages.